This page contains information about CME MDP 3.0 SBE market data on CME Smart Stream on Google Cloud Platform (GCP). CME Smart Stream on GCP leverages Google Cloud Pub/Sub technology for market data distribution. For complete details regarding Google Cloud Pub/Sub please refer to Google Pub/Sub documentation.

Within CME Smart Stream, CME MDP refers to the CME Group Market Data Platform, which provides the complete array of CME Group market data. 

...

Market Data Offerings via Google CloudCloud 

CME Smart Stream on GCP includes MDP 3.0 channels (CME, CBOT, COMEX, NYMEX, GME, MGEX), Streamlined SBE channels, Data Insights and BrokerTec U.S. Treasuries as follows:

...

...

The following section outlines testing and certification for GCP.

Certification

Certification via AutoCert+ is required for all client systems that will support CME Smart Stream on GCP. The following market data offerings have separate test suites:

• Globex MDP 3.0
• Data Insights - Settlement and Valuations
• Streamlined - Block Trades
• Streamlined - OTC Daily
• Streamlined - CME CF Cryptocurrency Indices
• Streamlined - Equity Indices
• Streamlined - OTC Daily
• BrokerTec U.S. Treasury Data

CME Globex Test Environments

The New Release test environment allows product and new functionality testing prior to release in production. Use the New Release environment to perform:

...

MDP 3.0

Clients planning to process CME MDP CME Globex market data on Google Cloud Pub/Sub should be familiar with the following CME MDP 3.0 concepts:

...

Statistics

Message Specification

Streamlined SBE

Clients planning to process Streamlined SBE on CME Smart Stream on GCP should be familiar with the following Streamlined SBE functional descriptions:

...

The message specifications for all streamlined systems are available here. 

Data Insights: Settlements and Valuations 

Clients planning to process CME Data Insights Settlements and Valuations market data should be familiar with the functional specification available here. 

BrokerTec U.S. Treasury Data

Clients planning to process process BrokerTec U.S. Treasury market data on CME Smart Stream on GCP should be familiar with the functional specification available here.  

Market Data Support Services

Market data services provide the external data required to process CME Group market data.

Support

...

Services

 There are two MDP services available for markets on CME Globex: Core Globex SBE Schema and CME Reference Data API. An FTP site is used to store the schema files for all environments.

FTP Site Information

CME provides an FTP (ftp://ftp.cmegroup.com) and SFTP (sftpng.cmegroup.com) site to disseminate MDP SBE schemas and market data configuration information. This FTP/SFTP site contains the Schema and Configuration files for all events. 

...

CME MDP on Google Cloud Pub/Sub Message Flow

The following diagram outlines the Publisher (CME Group) relationship with the Subscriber (Client System) for for Google Cloud Pub/Sub technology.   

...

4. The subscriber receives pending messages from its subscription and acknowledges each one to the Cloud Pub/Sub service.  A message contains the CME Group Simple Binary Encoded (SBE) messages.

5. When a message is acknowledged by the subscriber, it is removed from the subscription's message queue.

Message Structure

The following section outlines the CME MDP messaging within a Google Pub/Sub Message structure.  

CME MDP Included Data

The Google Pub/Sub data field provides the full CME Group UDP data packet.  

The following diagram outlines CME data with a Google Pub/Sub message:

CME MDP Included Attributes 

In addition to the full CME Group UDP data packet, Smart Stream on GCP provides the following quick access data in the Google Pub/Sub attributes field:

...

CME Group MDP UDP Packet Headers

The following table outlines CME Group MDP UDP packet header concepts in the diagram above:

Message Ordering

Google Pub/Sub does not guarantee message order. Therefore, the client system must ensure the correct processing of messages.

...

Topic names have the following methodology:  

Topic Name Examples

The following are topic name examples:

...

The following section outlines recovery options for CME Smart Stream on GCP.

Google Cloud Pub/Sub Message Replay

Google Cloud Pub/Sub includes the ability to replay messages. For more information on recovering messages, please refer to the Google quick start guide. (https://cloud.google.com/pubsub/docs/replay-qs).  

Google Cloud Pub/Sub replay is available for all CME all CME MDP channels on Google Cloud Pub/Sub.  Client Client systems that require data recovery from the beginning the beginning of the week on CME Globex can determine the weekly start via the incremental topic when the CME Group sequence number is equal to one.

Large-Scale Market Recovery

This section describes Market by Price (MBP) & Market by Order (MBO) recovery processing for MDP 3.0 channels via Google via Google Cloud Pub/Sub. Market Recovery should be used for large-scale data recovery (i.e. major outage or late join) to synchronize client systems to the latest state maintained by CME Group.  Market by Price (MBP) & Market by Order (MBO) recovery have separate topics.

Once a subscriber requests to recover from a market recovery topic all Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages are looped 3 times for an applicable channel.  

The following is an example of the messaging from the market recovery 3x topic loop:

A Market Data Snapshot Full Refresh (35=W) message will be included if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the recovery topic, even if the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for instruments having no book activity. More than one instrument can be updated in a packet. The following information is available in the Market Data Snapshot Full Refresh (35=W) message:

...

• Last Trade
• Opening Price
• Session High Trade Price
• Session Low Trade Price
• Session High Bid
• Session Low Offer
• Electronic Volume

Large Scale Market Recovery Processing

When recovery is required, client systems must re-synchronize all books to the latest state maintained by CME Group and determine whether any new instrument definitions have been published.

• the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental topic

• tag 60-TransactTime communicates the start transaction time of the last event for the instrument

  Market Recovery Topic - Snapshot (tag 35-MsgType=W) Message		Incremental Topic
  tag 369-LastMsgSeqNumProcessed	corresponds to	packet sequence number
  tag 60-TransactTime		tag 60-TransactTime

...

CME Smart Stream on GCP utilizes the same SBE template processing methodology as CME Globex. 

Market Recovery Processing Workflow

This section describes the process to follow for large-scale recovery concurrently processing the Incremental topic and Snapshot messages from the Market Recovery topic. Once a book is recovered, client systems can resume normal processing for that instrument even if other books are still being recovered. Client systems will not recover any missed statistics on the Market Recovery topic.

...

1. Identify channel(s) in which the client system is out of sync.
2. Listen to the Incremental topic and queue real-time data for the affected channel(s)and optionally begin MBP natural refresh.
3. Listen to the Market Recovery topic for the affected channel(s).

4. For a given instrument, compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental topic Market Data Incremental Refresh (35=X) message packet sequence number.

   • If the Security ID appears in both the Incremental topic and Market Recovery topic updates during the comparison, then compare the Market Recovery topic tag 60-TransactTime to the Incremental topic tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing.

   • Drop all cached Incremental topic updates with a packet sequence number < 369-LastMsgSeqNumProcessed.

5. Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery topic. 

Market Recovery Processing Examples

The following section provides examples for market recovery topic processing.  

Example 1 – Standard Scenario

The following example shows a standard recovery scenario using the following workflow:

...

Example 2 - Same Transaction Time Equal for Instrument

In the following example, the instrument (tag 48-SecurityID=15212) appears in both the Incremental and Market Recovery topic updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the Market Recovery topic tag 60-TransactTime to the Incremental topic tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered. 

Example 3 - Different Transaction Time for Instrument

In the following example, the instrument (tag 48-SecurityID=13205) appears in both the Incremental and Market Recovery topic updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, client systems must process the next Market Recovery topic iteration for the instrument.

Example 4 - New Market Recovery Snapshot Added

In the following example there is activity for an instrument (tag 48-SecurityID) that adds a new Market Recovery Snapshot message to the Market Recovery topic. Therefore, to ensure all new instruments are recovered, client systems must process an additional Market Recovery iteration starting from the Snapshot message with a packet sequence number = 1. Client systems can determine if a new Snapshot message has been added by comparing tag 911-TotNumReports with the previous iteration. The order of each Snapshot message iteration is not guaranteed; therefore the newly created Snapshot message may be placed mid-loop in the next iteration.

Instrument Recovery 

Client systems can use the Instrument Recovery topic on each Globex channel to determine if any new instruments have been defined through the Security Definition (tag 35-MsgType=d) message. The Instrument Recovery topic sends two replays of the current week's Security Definition (tag-MsgType=d) messages and will reflect any additions, modifications, or deletes to available instruments. Tag 911-TotNumReports indicates the total number of messages to process for a complete set of Security Definitions. CME Group does not guarantee the order of the messages being sent.  The Instrument Recovery topic can have multiple messages per packet.

...