This page contains information about CME MDP 3.0 SBE market data on CME Smart Stream on Google Cloud Platform (GCP). CME Smart Stream on GCP leverages Google Cloud Pub/Sub technology for market data distribution. For complete details regarding Google Cloud Pub/Sub please refer to Google Pub/Sub documentation.

Within CME Smart Stream, CME MDP refers to the CME Group Market Data Platform, which provides the complete array of CME Group market data. 

CME MDP 3.0 SBE is the CME Globex event-based bid, ask, trade and statistical market data format.

• Real-Time - Real-time market data via CME Globex

• Delayed (10 minute delay) - Allows customers to use CME Group data under delayed licensing terms. Traditionally, customers licensing delayed data had to self-manage the throttling of real-time feeds from exchanges per licensing policies. This cloud-based delivery mechanism provides customers with low cost, flexible and reliable access to data without the requirement to delay the content so they can focus on its use. Receipt of the data and licensing removes the requirement, complexity and cost of an intermediary service.

Contents

Market Data Offerings via Google Cloud 

CME Smart Stream on GCP includes MDP 3.0 channels (CME, CBOT, COMEX, NYMEX, GME, MGEX), Streamlined SBE channels, Data Insights and BrokerTec U.S. Treasuries as follows:

• Equity indices - S&P and Bloomberg

• Settlements and Valuations

• Block trades – GME

• OTC daily

• BrokerTec U.S. Treasury Market Data

The following table outlines all market data channels supported on CME Smart Stream on GCP.  

Testing and Certification

The following section outlines testing and certification for GCP.

Certification

Certification via AutoCert+ is required for all client systems that will support CME Smart Stream on GCP. The following market data offerings have separate test suites:

• Globex MDP 3.0

• Data Insights - Settlement and Valuations

• Streamlined - Block Trades

• Streamlined - OTC Daily

• Streamlined - CME CF Cryptocurrency Indices

• Streamlined - Equity Indices

• Streamlined - OTC Daily

• BrokerTec U.S. Treasury Data

CME Globex Test Environments

The New Release test environment allows product and new functionality testing prior to release in production. Use the New Release environment to perform:

• New product testing

• Development testing on new functionality

• Certification testing on new functionality
  
  
  

Getting Started with CME MDP Concepts

This section lists current CME MDP 3.0, Streamlined, Data Insight BrokerTec U.S. Treasury market data processing concepts applicable to CME MDP on Google Cloud Pub/Sub. If you are new to CME MDP technology or wish to better understand how your current system aligns with cloud-based technology, CME Group recommends starting here. 

MDP 3.0

Clients planning to process CME MDP CME Globex market data on Google Cloud Pub/Sub should be familiar with the following CME MDP 3.0 concepts:

Simple Binary Encoding (SBE)

Trade Summary

Book Processing

Statistics

Message Specification

Streamlined SBE

Clients planning to process Streamlined SBE on CME Smart Stream on GCP should be familiar with the following Streamlined SBE functional descriptions:

Equity Indices

OTC Daily

Block Trades

CME CF Cryptocurrency Indices

The message specifications for all streamlined systems are available here. 

Data Insights: Settlements and Valuations 

Clients planning to process CME Data Insights Settlements and Valuations market data should be familiar with the functional specification available here. 

BrokerTec U.S. Treasury Data

Clients planning to process BrokerTec U.S. Treasury market data on CME Smart Stream on GCP should be familiar with the functional specification available here.  

Market Data Support Services

Market data services provide the external data required to process CME Group market data.

Support Services

 There are two MDP services available for markets on CME Globex: Core Globex SBE Schema and CME Reference Data API. An FTP site is used to store the schema files for all environments.

FTP Site Information

CME provides an FTP (ftp://ftp.cmegroup.com) and SFTP (sftpng.cmegroup.com) site to disseminate MDP SBE schemas and market data configuration information. This FTP/SFTP site contains the Schema and Configuration files for all events. 

Information applies as follows in the table:

• Environment - specific environment (i.e., Certification, New Release, Production)

• Service - Core Globex SBE Schema

• FTP Site - address of the FTP/SFTP site

• Directory Location - identifies directory

• Client System Update Schedule - client systems should download updates according to schedule specified

Google Cloud Pub/Sub Processing Overview 

This section provides an overview of CME MDP processing for Google Cloud Pub/Sub. For complete details regarding Google Cloud Pub/Sub technology, refer to Google's online Pub/Sub documentation. For Pub/Sub libraries, refer to Google’s Pub/Sub Client Libraries. Common GCP SDK Pub/Sub processing concepts include the following:

• Subscriber methodology in Java 

• Pulling data from a subscription in Python synchronously and asynchronously

• Create a Subscription using JavaScript 

• Listen for messages using JavaScript 

CME MDP on Google Cloud Pub/Sub Message Flow

The following diagram outlines the Publisher (CME Group) relationship with the Subscriber (Client System) for Google Cloud Pub/Sub technology.   

1. CME Group will create a topic in the Cloud Pub/Sub service and send messages to the topic. To process CME MDP on Google Cloud Pub/Sub, client systems must subscribe to a topic to receive data. The Google Cloud message will contain CME Group messages.

2. Message store ensures that published messages are retained on behalf of subscriptions. 

3. Cloud Pub/Sub forwards messages from a topic to each individual subscription. Each subscription receives messages by pulling them from the service. For CME MDP on Cloud Pub/Sub there are two types of topics for a given channel ID that client systems can subscribe to:

• Incremental Topic - CME Group incremental market data using encoded packets containing the following FIX market data message types:

  • Security Definition (tag 35-MsgType=d)

  • Market Data Security Status (tag 35-MsgType=f)

  • Market Data Incremental Refresh (tag 35-MsgType=X)

  • Request for Quote (tag 35-MsgType=R)

• Market by Order (MBO) Market Recovery Snapshot Topic - Market by Order (MBO) Market Recovery method used for real-time and delayed data large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. This topic is only available for CME Globex MDP 3.0 channels.   
  
  

• Market by Price (MBP) Market Recovery Snapshot Topic - Market by Price (MBP) Market Recovery method used for real-time and delayed data large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. This topic is only available for CME Globex MDP 3.0 channels.   

• Instrument Recovery Topic - The Instrument Replay feed constantly replays the current week's Security Definition (tag-MsgType=d) messages will reflect any additions, modifications, or deletes to available instruments. This topic is only available for real-time and delayed data CME Globex MDP 3.0 channels.

Message Structure

The following section outlines the CME MDP messaging within a Google Pub/Sub Message structure.  

CME MDP Included Data

The Google Pub/Sub data field provides the full CME Group UDP data packet.  

The following diagram outlines CME data with a Google Pub/Sub message:

CME MDP Included Attributes 

In addition to the full CME Group UDP data packet, Smart Stream on GCP provides the following quick access data in the Google Pub/Sub attributes field:

CME Group MDP UDP Packet Headers

The following table outlines CME Group MDP UDP packet header concepts in the diagram above:

Message Ordering

Google Pub/Sub does not guarantee message order. Therefore, the client system must ensure the correct processing of messages.

Project Names and Topic Naming Methodology

CME Smart Stream SBE on Google Cloud Platform (GCP) has data per product. Project names have the following methodology:

Topic names have the following methodology:  

Topic Name Examples

The following are topic name examples:

• Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310): PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310

• Production MDP 3.0 Globex Equity Futures Recovery Real-Time Feed (Channel 310): PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.MBORPLY.310

• New Release Streamlined S&P Index Real-Time Feed (Channel 211): NR.SSCL.GCP.MD.RT.SPDJI.STRBIN.v01000.INCR.210

• Production MDP 3.0 BMD Futures Delayed Feed (Channel 430): PROD.SSCL.GCP.MD.DLY.BMD.FIXBIN.v01000.INCR.430

Recovery Processing

The following section outlines recovery options for CME Smart Stream on GCP.

Google Cloud Pub/Sub Message Replay

Google Cloud Pub/Sub includes the ability to replay messages. For more information on recovering messages, please refer to the Google quick start guide. (https://cloud.google.com/pubsub/docs/replay-qs).  

Google Cloud Pub/Sub replay is available for all CME MDP channels on Google Cloud Pub/Sub. Client systems that require data recovery from the beginning of the week on CME Globex can determine the weekly start via the incremental topic when the CME Group sequence number is equal to one.

Large-Scale Market Recovery

This section describes Market by Price (MBP) & Market by Order (MBO) recovery processing for MDP 3.0 channels via Google Cloud Pub/Sub. Market Recovery should be used for large-scale data recovery (i.e. major outage or late join) to synchronize client systems to the latest state maintained by CME Group.  Market by Price (MBP) & Market by Order (MBO) recovery have separate topics.

Once a subscriber requests to recover from a market recovery topic all Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages are looped 3 times for an applicable channel.  

The following is an example of the messaging from the market recovery 3x topic loop: