MDP 3.0 - MBP and MBOLD Market Recovery

The Market by Price (MBP) and Market by Order Limited Depth (MBOLD) Market Recovery for the CME MDP Conflated UDP market data group should be used for large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. 

MBP & MBOLD Market Recovery Messaging

The MBP & MBOLD market recovery feed constantly iterates and sends the Market Data Snapshot Full Refresh (tag 35-MsgType=W) message via templates SnapshotFullRefresh and SnapshotFullRefreshOrderBook. Template SnapshotFullRefreshOrderBook53 communicates Market by Order Limited Depth (MBOLD) book information. Template SnapshotFullRefresh52 communicates the following information:

• Market by Price (MBP) 

• Implied book information

• Last trade

• Electronic Volume

• Session Statistics

• Market States

A Market Data Snapshot Full Refresh (35=W) message will be generated only if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the loop until the end of the week, even though the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for the instruments that have only price bands, or thresholds, but have had no book activity since the beginning of the week. For an instrument MBP, market state, and statistics information is sent first then the MBOLD book information is sent. More than one instrument can be updated in a packet. Instruments expired during the trading session will stay on the market recovery feed until the end of the day. The example below shows the message structure of three different instruments updated in an MBP & MBOLD Conflated UDP market recovery feed across two packets.

MBOLD Market Recovery Price Interleaving

To optimize recovery times for the highest priority orders, the MBOLD Market Data Snapshot Full Refresh (tag 35-MsgType=W) message will alternate bid and ask orders from highest to lowest priority (tag 37707-MDOrderPriority).

MBP & MBOLD Market Recovery Processing

When packets are missed on both UDP incremental feeds, a recovery process is required to take place. Packet loss is detected using the message packet sequence numbers. The packet sequence number is incremental; therefore, if a gap is detected between packets, it indicates a packet has been missed. In such a case, it should be assumed that all books maintained in the client system may no longer have the correct, latest state maintained by CME Group. Client systems must resynchronize all books to the latest state maintained by CME Group, and determine whether any new instrument definitions were published.

Tag 369-LastMsgSeqNumProcessed value on the Market Recovery feed Snapshot (tag 35-MsgType=W) message corresponds to the packet sequence number on the Incremental Market Data feed. Additionally, tag 60-TransactTime communicates the start transaction time of the last event the security participated in. Therefore, tag 369-LastMsgSeqNumProcessed in conjunction with tag 60-TransactTime from the Market Recovery feed Snapshot (tag 35-MsgType=W) are used to link to the incremental feed packet sequence number and tag 60-TransactTime respectively. The order of each snapshot iteration is not guaranteed, therefore client systems must process one full iteration of a market recovery snapshot iteration starting at sequence number 1 to ensure full recovery.  

When the Market Recovery method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. A client application can use the tag 779-LastUpdateTime timestamp on the market recovery feed to determine if instrument data has changed, and if recovery is necessary, from the Instrument Replay iteration in addition to the Market Recovery iteration. 

MBP & MBOLD Market Recovery Processing Workflow Overview

This section describes at the process to follow for a large scale outage using Market Recovery while continuing to process the Incremental Market Data feed and obtaining snapshots from the Market Recovery feed. Once books are recovered, client systems can resume normal processing even if other books are still being recovered. Client systems have the option to concurrently recover Market by Price (MBP) instrument books via MBP Natural Refresh logic. Additionally, MBOLD books can be naturally refreshed on the incremental feed due to the overlay full book snapshot updates. Concurrent processing of Market Recovery and natural refresh provide the best book recovery performance. 

Client systems will not recover any missed statistics on the Market Recovery feed.

1. Identify channel(s) in which the client system is out of sync. 

2. Listen to the Incremental Market Data feed and queue real-time data for the affected channel(s), and optionally begin MBP or MBOLD natural refresh. 

3. Listen to the Market Recovery feed for the affected channel(s). 

4. For a given instrument, compare the market recovery snapshot tag 369-LastMsgSeqNumProcessed to the incremental packet sequence number. Next, if the security id appears in both the incremental and market recovery updates during the comparison, then compare the market recovery Tag 60-TransactTime to the incremental Tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing. Additionally, drop cached all incremental updates with a packet sequence number < 369-LastMsgSeqNumProcessed.
   
   

5. Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery feed. 

UDP Market Recovery Processing Examples

The following section provides examples for UDP market recovery feed processing.  

Example 1 – Standard Scenario

The following example shows a standard recovery scenario using the following workflow:

1. Identified channel that is out of sync.  
   
   
   

2. Connect to the incremental feed and begin queuing data.
   
   
   

3. Connect to the market recovery feed. In this example, the first market recovery snapshot has a sequence number is equal to 4. 
   
   
   

4. Compare the market recovery snapshot tag 369-LastMsgSeqNumProcessed to the incremental packet sequence number. In this example, the incremental sequence number compared is 104.

   • If the instrument (tag 48-SecurityID) appears in both the incremental and market recovery updates during the comparison, then compare the market recovery tag 60-TransactTime to the incremental tag 60-TransactTime.

   • In this example, the tag 60-TransactTime comparison is not required due to different instruments between the feeds.

   • For market recovery snapshot processing, drop all cached incremental updates with a packet sequence number < 369-LastMsgSeqNumProcessed.

   • Begin normal processing for recovered instrument 13205.

   • Recover subsequent instruments via the market recovery feed (8087, etc).

   • Additionally, as 369-LastMsgSeqNumProcessed increases in subsequent market recovery updates, continue to drop cached market data where the packet sequence number < to 369-LastMsgSeqNumProcessed.  
     
     
     

5. Disconnect from the market recovery feed once processing is complete for all instruments.

Example 2 - Same Transaction Time Equal For Instrument

In the following example, the instrument (tag 48-SecurityID=15212) appears in both the incremental and market recovery updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the market recovery tag 60-TransactTime to the incremental Tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered. 

Example 3 - Different Transaction Time For Instrument

In the following example, the instrument (tag 48-SecurityID=13205) appears in both the incremental and market recovery updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, for the instrument client systems must process the next market recovery feed iteration.

Example 4 - New Market Recovery Snapshot Added

In the following scenario there’s activity for an instrument (tag 48-SecurityID) which causes a new market recovery snapshot message to be added to the market recovery feed. Therefore, to ensure all new instruments are recovered client systems must process an additional market recovery iteration starting from snapshot packet sequence number equal to one. Client systems can determine if a new snapshot as been added by comparing tag 911-TotNumReports with the previous iteration. The order of each snapshot iteration is not guaranteed, therefore the newly created snapshot may be placed mid-loop in the next iteration.