This page contains information about CME MDP 3.0 SBE market data on CME Smart Stream on Google Cloud Platform (GCP). CME Smart Stream on GCP leverages Google Cloud Pub/Sub technology for market data distribution. For complete details regarding Google Cloud Pub/Sub please refer to Google Pub/Sub documentation.
Within CME Smart Stream, CME MDP refers to the CME Group Market Data Platform, which provides the complete array of CME Group market data.
...
| CME Group Channel ID | Channel Name | Market Data Platform Source | DCM/Venue/Market |
|---|---|---|---|
| 200 | CME Credit OTC | Streamlined - OTC Daily | CME |
| 201 | CME Cleared OTC | Streamlined - OTC Daily | CME |
| 210 | S&P Indexes | Streamlined - Equity Indices | S&P |
| 211 | Dow Jones Indexes | Streamlined - Equity Indices | DJIA |
| 213 | CME CF Rates and Indexes | Streamlined - CME CF Cryptocurrency Indices | CME |
| 215 | BrokerTec US Treasury Data | BrokerTec US Treasury Data | BrokerTec/NEX Data |
| 230 | CME Block Trades | Streamlined - Block Trades | CME |
| 231 | CBOT Block Trades | Streamlined - Block Trades | CBOT |
| 232 | NYMEX Block Trades | Streamlined - Block Trades | NYMEX |
| 233 | COMEX Block Trades | Streamlined - Block Trades | COMEX |
| 234 | GME Block Trades | Streamlined - Block Trades | GME |
| 251 | CME Settlement and Valuations | Data Insights - Settlement and Valuations | CME |
| 252 | CBOT Settlement and Valuations | Data Insights - Settlement and Valuations | CBOT |
| 253 | NYMEX Settlement and Valuations | Data Insights - Settlement and Valuations | NYMEX |
| 254 | COMEX Settlement and Valuations | Data Insights - Settlement and Valuations | COMEX |
| 261 | CME Benchmark Administration Premium | Data Insights - CME Benchmark Administration Premium | CME |
| 310 | CME Globex Equity Futures | Globex MDP 3.0 | CME |
| 311 | CME Globex Equity Options | Globex MDP 3.0 | CME |
| 312 | CME Globex Interest Rate Futures | Globex MDP 3.0 | CME |
| 313 | CME Globex Interest Rate Options | Globex MDP 3.0 | CME |
| 314 | CME Globex FX Futures | Globex MDP 3.0 | CME |
| 315 | CME Globex FX Options | Globex MDP 3.0 | CME |
| 316 | CME Globex Commodity Futures | Globex MDP 3.0 | CME |
| 317 | CME Globex Commodity Options | Globex MDP 3.0 | CME |
| 318 | CME Globex Equity Futures - excludes E-mini S&P 500 | Globex MDP 3.0 | CME |
| 319 | CME Globex Equity Options - excludes E-mini S&P 500 | Globex MDP 3.0 | CME |
| 320 | CME Globex FX Futures II | Globex MDP 3.0 | CME |
| 321 | CME Globex FX Options II | Globex MDP 3.0 | CME |
| 323 | CME Globex Micro Equity Options | Globex MDP 3.0 | CME |
| 326 | Cryptocurrency - Futures | Globex MDP 3.0 | CME |
| 327 | Cryptocurrency - Options | Globex MDP 3.0 | CME |
| 330 | SpotFX+ | Globex MDP 3.0 | CME |
| 340 | CBOT Globex Commodity Futures | Globex MDP 3.0 | CBOT |
| 341 | CBOT Globex Commodity Options | Globex MDP 3.0 | CBOT |
| 342 | CBOT Globex Equity Index Futures | Globex MDP 3.0 | CBOT |
| 343 | CBOT Globex Equity Index Options | Globex MDP 3.0 | CBOT |
| 344 | CBOT Globex Interest Rate Futures | Globex MDP 3.0 | CBOT |
| 345 | CBOT Globex Interest Rate Options | Globex MDP 3.0 | CBOT |
| 346 | CBOT Globex Commodity Futures II | Globex MDP 3.0 | CBOT |
| 348 | CBOT Globex Interest Rate Futures II | Globex MDP 3.0 | CBOT |
| 360 | COMEX Globex Futures | Globex MDP 3.0 | COMEX |
| 361 | COMEX Globex Options | Globex MDP 3.0 | COMEX |
| 380 | NYMEX Globex Emissions Futures | Globex MDP 3.0 | NYMEX |
| 381 | NYMEX Globex Emissions Options | Globex MDP 3.0 | NYMEX |
| 382 | NYMEX Globex Crude & Crude Refined Futures | Globex MDP 3.0 | NYMEX |
| 383 | NYMEX Globex Crude & Crude Refined Options | Globex MDP 3.0 | NYMEX |
| 384 | NYMEX Globex Metals, Softs, & Alternative Market Futures | Globex MDP 3.0 | NYMEX |
| 385 | NYMEX Globex Metals, Softs, & Alternative Market Options | Globex MDP 3.0 | NYMEX |
| 386 | NYMEX Globex Nat Gas & other Non-Crude Energy Futures | Globex MDP 3.0 | NYMEX |
| 387 | NYMEX Globex Nat Gas & other Non-Crude Energy Options | Globex MDP 3.0 | NYMEX |
| 440 | GME Globex Futures | Globex MDP 3.0 | GME |
| 441 | GME Globex Options | Globex MDP 3.0 | GME |
| 460 | MGEX Globex Futures | Globex MDP 3.0 | MGEX |
| 461 | MGEX Globex Options | Globex MDP 3.0 | MGEX |
...
Certification via AutoCert+ is required for all client systems that will support CME Smart Stream on GCP. The following market data offerings have separate test suites:
- Globex MDP 3.0
- Data Insights - Settlement and Valuations
- Streamlined - Block Trades
- Streamlined - OTC Daily
- Streamlined - CME CF Cryptocurrency Indices
- Streamlined - Equity Indices
- Streamlined - OTC Daily
- BrokerTec U.S. Treasury Data
...
Clients planning to process BrokerTec U.S. Treasury market data on CME Smart Stream on GCP should be familiar with the functional specification available here.
Market Data Support Services
Market data services provide the external data required to process CME Group market data.
| Gliffy | ||||||||
|---|---|---|---|---|---|---|---|---|
|
Support Services
There are two MDP services available for markets on CME Globex: Core Globex SBE Schema and CME Reference Data API. An FTP site is used to store the schema files for all environments.
Service | Description |
|---|---|
Core Globex SBE Schema | MDP is a template-based SBE protocol wherein a given message is interpreted by means of its corresponding template. Each message contains a unique Schema ID that references the template to use to interpret the message. The core CME Globex SBE schema is used across all market data groups. |
| CME Reference Data API | Product and instrument referential data for CME Smart Stream on GCP can be obtained via CME Reference Data API.
|
...
The following diagram outlines the Publisher (CME Group) relationship with the Subscriber (Client System) for Google Cloud Pub/Sub technology.
| Gliffy | ||||||
|---|---|---|---|---|---|---|
|
...
4. The subscriber receives pending messages from its subscription and acknowledges each one to the Cloud Pub/Sub service. A message contains the CME Group Simple Binary Encoded (SBE) messages.
5. When a message is acknowledged by the subscriber, it is removed from the subscription's message queue.
...
The following section outlines the CME MDP messaging within a Google Pub/Sub Message structure.
CME MDP Included Data
The Google Pub/Sub data field provides the full CME Group UDP data packet.
The following diagram outlines CME data with a Google Pub/Sub message:
| Gliffy | ||||
|---|---|---|---|---|
|
...
| Project Name | Type | Environment |
|---|---|---|
| cmegroup-marketdata-cr-nr | Cryptocurrency Only | New Release |
| cmegroup-marketdata-newrel | All Other Data | New Release |
| cmegroup-marketdata-cr | Cryptocurrency Only | Production |
| cmegroup-marketdata | All Other Data | Production |
Topic names have the following methodology:
| Name | Valid Values | Description |
|---|---|---|
| EnvType | PROD = Production NR = New Release CERT = Certification | Environment |
| ServiceName | SSCL = Smart Stream Cloud Link | Cloud service provider |
| Provider | GCP = Google Cloud Products | Name of the cloud provider |
| ContentType | MD = Market Data | Type of content |
| Content Timing | RT = Real Time DLY = Delayed | Time data is sent |
| ContentOwner | CMEG = CME Group Owned Market Data SPDJI = S&P DowJones USTBTEC = BrokerTec US Treasury Data MGEX = Minneapolis Grain Exchange GME = Gulf Mercantile Exchange | Reference to exchange |
| Format | FIXBIN = MDP 3.0 Globex Data Channels STRBIN = Streamlined SVBIN = Settlements & Valuations | Feed Type |
| Version | v01000 | CME Group version of cloud solution |
| Topic Type | INCR = Incremental RPLY = MBP Recovery MBORPLY = MBO Recovery INST = Instrument Recovery | Defines the type of topic |
| Channel Number | Examples: 230 = CME Block Trades 340 = CBOT Globex Commodity Futures | CME Group assigned channel ID See Market Data Offerings via Google Cloud for channel IDs. Product and instrument referential data for CME Smart Stream on GCP can be obtained via CME Reference Data API. |
Topic Name Examples
The following are topic name examples:
...
Google Cloud Pub/Sub Message Replay
Google Cloud Pub/Sub includes the ability to replay messages. For more information on recovering messages, please refer to the Google quick start guide. (https://cloud.google.com/pubsub/docs/replay-qs).
Google Cloud Pub/Sub replay is available for all CME MDP channels on Google Cloud Pub/Sub. Client systems that require data recovery from the beginning of the week on CME Globex can determine the weekly start via the incremental topic when the CME Group sequence number is equal to one.
Large-Scale Market Recovery
This section describes Market by Price (MBP) & Market by Order (MBO) recovery processing for MDP 3.0 channels via Google Cloud Pub/Sub. Market Recovery should be used for large-scale data recovery (i.e. major outage or late join) to synchronize client systems to the latest state maintained by CME Group. Market by Price (MBP) & Market by Order (MBO) recovery have separate topics.
Once a subscriber requests to recover from a market recovery topic all Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages are looped 3 times for an applicable channel.
| Info |
|---|
| The Pub/Sub market recovery topic is different the from the MDP 3.0 implementation where the market feed continually iterates. |
The following is an example of the messaging from the market recovery 3x topic loop:
Market Recovery Topic | ||||
CME Snapshot Sequence Number | Tag 369-LastMsgSeqNumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911-TotNumReports |
| First Market Recovery Iteration (1) | ||||
1 - First message from market recovery topic | 107 | 20191010-18:36:41.000013217 | 8087 | 3 |
| 2 | 120 | 20191010-18:36:41.000033041 | 773 | 3 |
| 3 | 131 | 20191010-18:36:41.000054910 | 12171 | 3 |
Next Market Recovery Iteration (2) | ||||
1 | 140 | 20191120-18:36:41.000071886 | 8087 | 3 |
2 | 152 | 20191120-18:36:41.000093224 | 773 | 3 |
3 | 157 | 20191120-18:36:41.000111773 | 12171 | 3 |
Final Market Recovery Iteration (3) | ||||
1 | 166 | 20191120-18:36:41.000129878 | 8087 | 3 |
2 | 174 | 20191120-18:36:41.000150215 | 773 | 3 |
3 - Last message from market recovery topic | 179 | 20191120-18:36:41.000172319 | 12171 | 3 |
A Market Data Snapshot Full Refresh (35=W) message will be included if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the recovery topic, even if the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for instruments having no book activity. More than one instrument can be updated in a packet. The following information is available in the Market Data Snapshot Full Refresh (35=W) message:
...
- the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental topic
tag 60-TransactTime communicates the start transaction time of the last event for the instrument
Market Recovery Topic - Snapshot (tag 35-MsgType=W) Message Incremental Topic tag 369-LastMsgSeqNumProcessed corresponds to packet sequence number tag 60-TransactTime tag 60-TransactTime
...
Incremental Topic | Market Recovery Topic | |||||||
Incremental Packet Sequence Number | Tag 35- MsgType | Tag 48- SecurityID(s) | Tag 60-TransactTime(s) | Snapshot Packet Sequence Number | Tag 369- LastMsgSeq NumProcessed | Tag 60-TransactTime
| Tag 48-SecurityID | Tag 911- TotNumReports |
100 - drop | X | 13205 | 20191120-18:36:41.000000000 | |||||
101 - drop | X | 5522 | 20191120-18:36:41.000278478 | |||||
904 | 20191120-18:36:41.000293451 | |||||||
102 - drop | X | 904 | 20191120-18:36:41.000320480 | |||||
103 - drop | X | 15212 | 20191120-18:36:41.000368253 | |||||
104- recover instrument 15212 information via market recovery snapshot message and begin normal processing | X | 15212 | 20191120-18:36:41.000368253 TransactTime matches with snapshot message | 1 – Connect to Market Recovery topic | 104 | 20191120-18:36:41.000368253
TransactTime matches with incremental message | 15212 | 10 |
...