MDP 3.0 - MBP and MBOFD Market Recovery
This page describes the market data recovery process for Market by Price (MBP) and Market by Order Full Depth (MBOFD) order books. This method:
should be used for large-scale data recovery (e.g. major outage or late join)
synchronizes client order books with the current state maintained by CME Group
The Market Recovery Feed:
loops the Market Data Snapshot Full Refresh (tag 35-MsgType=W) message.
at the beginning of the week, a Snapshot message is generated each time book/trade activity occurs on an instrument
no Snapshot message is sent for an instrument with no book activity
more than one instrument can be updated in a packet
instruments expired during the trading session will stay on the market recovery feed until the end of the day.
supports primary (A) and backup (B) feeds
Feed A - disseminates market data Snapshot messages for all books having activity since beginning of the week
Feed B - acts as a backup in the event that Feed A becomes inoperable
CME Group strongly recommends that the Market Recovery feeds be used for recovery purposes only. Once the client system has retrieved the recovery data, the client system should stop listening to the Market Recovery feeds.
Market Recovery support is required for client systems that utilize MDP 3.0.
This topic includes:
Market Recovery Feed Messaging
This section describes the messaging and templates for MBP and MBOFD Market Recovery feeds.
MBP Recovery Feed Messaging
The MBP Market Recovery feed:
loops Snapshot messages
implements template SnapshotFullRefresh for recovery processing
Template SnapshotFullRefresh contains:
Top of Book MBP Quotes – Bids and Offers
Top of Book Implied Quotes – Bids and Offers
Last Trade
Opening Price
Session High and Low Trade Prices
Session High Bid and Session Low Offer
Fixing Price
Settlement Price
Cleared Trade Volume
Open Interest
Electronic Volume
Threshold Limits
For MBP books, client systems have the option to concurrently recover instrument books via MBP Natural Refresh logic.
MBOFD Recovery Feed Messaging
The MBOFD Market Recovery feed:
loops Snapshot messages
implements template SnapshotFullRefresh and template SnapshotFullRefreshOrderBook for recovery processing
In addition to template SnapshotFullRefresh, template SnapshotFullRefreshOrderBook contains recovery data in the following order:
MBP information
market state
statistics
MBOFD book information
The MBOFD Market Recovery feed also supports:
recovery chunk tag 37709-NoChunks and tag 37710- CurrentChunk
allows client systems to determine the number of packets required to recover an MBOFD book.
tag 37709-NoChunks - total number of packets that constitute a single instrument order book
tag 37710-CurrentChunk - current chunk in the sequence
The example below shows two different instruments updated in an MBOFD Market Recovery feed.
Market Recovery Price Interleaving (MBOFD)
To optimize recovery time for the highest priority orders, the Snapshot message alternates bid and ask orders from highest to lowest priority (tag 37707-MDOrderPriority).
Market Recovery Processing (MBP & MBOFD)
When packets are missed on both UDP Incremental feeds, packet loss is indicated by a gap in message packet sequence numbers.
the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental feed
tag 60-TransactTime communicates the start transaction time of the last event for the instrument
Market Recovery Feed - Snapshot (tag 35-MsgType=W) Message |
| Incremental Feed |
---|---|---|
369-LastMsgSeqNumProcessed | corresponds to | packet sequence number |
tag 60-TransactTime | tag 60-TransactTime |
When this large-scale market recovery method is used, client systems must subscribe to the Instrument Definition feed to determine if any new instruments have been defined. A client application can determine if recovery is necessary using the tag 779-LastUpdateTime timestamp on the Market Recovery feed to detect if instrument data has changed.
Market Recovery Processing Workflow
This section describes the process to follow for large-scale recovery concurrently processing the Incremental feed and Snapshot messages from the Market Recovery feed. Once a book is recovered, client systems can resume normal processing for that instrument even if other books are still being recovered. Client systems will not recover any missed statistics on the Market Recovery feed.
Recovery processing steps are as follows:
Identify channel(s) in which the client system is out of sync.
Listen to the Incremental feed and queue real-time data for the affected channel(s), and optionally begin MBP natural refresh.
Listen to the Market Recovery feed for the affected channel(s).
For a given instrument, compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental feed Market Data Incremental Refresh (35=X) message packet sequence number.
If the SecurityID appears in both the Incremental feed and Market Recovery feed updates during the comparison, then compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing.
Drop all cached Incremental feed updates with a packet sequence number < 369-LastMsgSeqNumProcessed.
Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery feed.
Market Recovery Feed Processing Examples
The following section provides examples for Market Recovery feed processing.
Example 1 – Standard Recovery
The following example shows a standard recovery scenario using the following workflow:
Identify the channel that is out of sync.
Connect to the Incremental feed and begin queuing data.
Connect to the Market Recovery feed. In this example, the first Market Recovery Snapshot message sequence number = 4.
Compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental feed packet sequence number. In this example, the Incremental feed sequence number compared is 104.
If the instrument (tag 48-SecurityID) appears in both the Incremental and Market Recovery updates during the comparison, then compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime.
In this example, the tag 60-TransactTime comparison is not required due to different instruments between the feeds.
For Market Recovery Snapshot message processing, drop all cached Incremental feed updates with a packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery feed.
Begin normal processing for recovered instrument 13205.
Recover subsequent instruments via the Market Recovery feed (8087, etc).
Additionally, as tag 369-LastMsgSeqNumProcessed increases in subsequent Market Recovery feed updates, continue to drop cached market data where the packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery feed.
Disconnect from the Market Recovery feed once processing is complete for all instruments.
Incremental Feed | Market Recovery Feed | |||||||
Incremental Packet Sequence Number | Tag 35- MsgType | Tag 48- SecurityID(s) | Tag 60-TransactTime(s) | Snapshot Packet Sequence Number | Tag 369- LastMsgSeq NumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911- TotNumReports |
100 - drop | X | 13205 | 20191120-18:36:41.000000000 |
| ||||
101 - drop | X | 5522 | 20191120-18:36:41.000278478 | |||||
904 | 20191120-18:36:41.000293451 | |||||||
102 - drop | X | 904 | 20191120-18:36:41.000320480 | |||||
103 - drop | X | 1741 | 20191120-18:36:41.000353097 | |||||
104 - recover instrument 13205 information via Market Recovery Snapshot message and begin normal processing | X | 7720 | 20191120-18:36:41.000368253
| 4 – Connect to Market Recovery Feed | 104 | 20191120-18:36:41.000000000
| 13205 | 10 |
105 – apply update to 13205 information gathered in the Snapshot | X | 13205 | 20191120-18:36:41.000383610 | 5 – Next instrument to be processed | 108 | 20191120-18:32:22.080423217 | 8087 | 10 |
106 | X | 5522 | 20191120-18:36:41.000412613 | Next Market Recovery Iteration | ||||
107 | X | 12121 | 20191120-18:36:41.000423217 | 1 | 112 |
| 1741 | 10 |
…. |
|
|
| …. |
|
|
|
|
Example 2 - Same Transaction Time for Instrument
In the following example, the instrument (tag 48-SecurityID=15212) appears in both the Incremental and Market Recovery feed updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered.
Incremental Feed | Market Recovery Feed | |||||||
Incremental Packet Sequence Number | Tag 35- MsgType | Tag 48- SecurityID(s) | Tag 60-TransactTime(s) | Snapshot Packet Sequence Number | Tag 369- LastMsgSeq NumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911- TotNumReports |
100 - drop | X | 13205 | 20191120-18:36:41.000000000 |
| ||||
101 - drop | X | 5522 | 20191120-18:36:41.000278478 | |||||
904 | 20191120-18:36:41.000293451 | |||||||
102 - drop | X | 904 | 20191120-18:36:41.000320480 | |||||
103 - drop | X | 15212 | 20191120-18:36:41.000368253 | |||||
104 - recover instrument 15212 information via Market Recovery Snapshot message and begin normal processing
| X | 15212 | 20191120-18:36:41.000368253
TransactTime matches with Snapshot message | 4 – Connect to Market Recovery Feed | 104 | 20191120-18:36:41.000368253
TransactTime matches with Incremental message | 15212 | 10 |
Example 3 - Different Transaction Time for Instrument
In the following example, the instrument (tag 48-SecurityID=13205) appears in both the Incremental and Market Recovery feed updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, client systems must process the next Market Recovery feed iteration for the instrument.
Incremental Feed | Market Recovery Feed | |||||||
Incremental Packet Sequence Number | Tag 35- MsgType | Tag 48- SecurityID(s) | Tag 60-TransactTime(s) | Snapshot Packet Sequence Number | Tag 369- LastMsgSeq NumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911- TotNumReports |
100 - drop | X | 13205 | 20191120-18:36:41.000000000 |
| ||||
101 - drop | X | 5522 | 20191120-18:36:41.000278478 | |||||
904 | 20191120-18:36:41.000293451 | |||||||
102 - drop | X | 904 | 20191120-18:36:41.000320480 | |||||
103 - drop | X | 13205 | 20191120-18:36:41.000353097 | |||||
104 | X | 13205 | 20191120-18:36:41.000353097
| 4 – Connect to Market Recovery Feed | 104 | 20191120-18:36:40.103368841
TransactTime does not match with Incremental message, must process next loop to recover 13205 | 13205 | 10 |
105 | X | 13205 | 20191120-18:36:41.000353097 | 4 – Next instrument to be processed | 107 | 20191120-18:36:22.000423217 | 8087 | 10 |
106 | X | 5522 | 20191120-18:36:41.000412613 | Next Market Recovery Iteration | ||||
107 | X | 12121 | 20191120-18:36:41.000423217 | 1 | 110 |
| 1741 | 10 |
…. |
|
|
| …. |
|
|
|
|
170 | X | 13205 | 20191120-18:36:43.121427232
TransactTime matches with Snapshot message, recover instrument 13205 | 4 | 170 | 20191120-18:36:43.121427232
TransactTime matches with Incremental message, recover instrument 13205 | 13205 | 10 |
Example 4 - New Market Recovery Snapshot Message Added
In the following example there is activity for an instrument (tag 48-SecurityID) that adds a new Market Recovery Snapshot message to the Market Recovery feed. Therefore, to ensure all new instruments are recovered, client systems must process an additional Market Recovery iteration starting from the Snapshot message with a packet sequence number = 1. Client systems can determine if a new Snapshot message has been added by comparing tag 911-TotNumReports with the previous iteration. The order of each Snapshot message iteration is not guaranteed, therefore the newly created Snapshot message may be placed mid-loop in the next iteration.
Incremental Feed | Market Recovery Feed | |||||||
Incremental Packet Sequence Number | Tag 35- MsgType | Tag 48- SecurityID(s) | Tag 60-TransactTime(s) | Snapshot Packet Sequence Number | Tag 369- LastMsgSeq NumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911- TotNumReports |
105 - drop | X | 12171 – New instrument activity generates a Market Recovery Snapshot message | 20191120-18:36:41.000383610 | 4 - Recovery in process | 107 | 20191120-18:36:41.000013217 | 8087 | 10 |
106 - drop | X | 5522 | 20191120-18:36:41.000412613 | Next Market Recovery Iteration | ||||
107 - drop | X | 13205 | 20191120-18:36:41.000423217 | 1 | 110 | 20191120-18:36:41.000353097 | 1741 | 11 – incremented to reflect new Snapshot message for instrument 12171 |
…. |
|
|
| …. |
|
|
|
|
170 – begin normal processing for instrument 12171 from Market Recovery Snapshot due to the Incremental update for packet sequence number < to 369-LastMsgSeqNumProcessed
| X | 904 | 20191120-18:36:41.001223443 | 3 – new Snapshot placed mid-iteration | 170 – recovery instrument via Snapshot | 20191120-18:36:41.000383610 | 12171 | 11 |
171 – apply update to 12171 information gathered in the Snapshot | X | 12171 |
| …. |
|
|
|
|
…. |
|
|
|
How was your Client Systems Wiki Experience? Submit Feedback
Copyright © 2024 CME Group Inc. All rights reserved.