Versions Compared

Old Version 6

changes.mady.by.user CME Group

Saved on

compared with

New Version Current

changes.mady.by.user CME Group

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This page contains information about CME MDP 3.0 SBE market data on CME Smart Stream on Google Cloud Platform (GCP). CME Smart Stream on GCP leverages Google Cloud Pub/Sub technology for market data distribution. For complete details regarding Google Cloud Pub/Sub please refer to Google Pub/Sub documentation.

Within CME Smart Stream, CME MDP refers to the CME Group Market Data Platform, which provides the complete array of CME Group market data. 

...

Market Data Offerings via Google CloudCloud 

CME Smart Stream on GCP includes MDP 3.0 channels (CME, CBOT, COMEX, NYMEX, GME, MGEX), Streamlined SBE channels, Data Insights and BrokerTec U.S. Treasuries as follows:

...

CME Group Channel IDChannel NameMarket Data Platform SourceDCM/Venue/Market
200 CME Credit OTCStreamlined - OTC DailyCME
201 CME Cleared OTCStreamlined - OTC DailyCME
210S&P IndexesStreamlined - Equity IndicesS&P
211Dow Jones IndexesStreamlined - Equity IndicesDJIA
213CME CF Rates and IndexesStreamlined - CME CF Cryptocurrency IndicesCME
215BrokerTec US Treasury DataBrokerTec US Treasury DataBrokerTec/NEX Data
230CME Block TradesStreamlined - Block TradesCME
231CBOT Block TradesStreamlined - Block TradesCBOT
232NYMEX Block TradesStreamlined - Block TradesNYMEX
233 COMEX Block TradesStreamlined - Block TradesCOMEX
234 GME Block TradesStreamlined - Block TradesGME
251CME Settlement and ValuationsData Insights - Settlement and ValuationsCME
252CBOT Settlement and ValuationsData Insights - Settlement and ValuationsCBOT
253NYMEX Settlement and ValuationsData Insights - Settlement and ValuationsNYMEX
254COMEX Settlement and ValuationsData Insights - Settlement and ValuationsCOMEX
261CME Benchmark Administration PremiumData Insights - CME Benchmark Administration PremiumCME
310CME Globex Equity FuturesGlobex MDP 3.0CME
311CME Globex Equity OptionsGlobex MDP 3.0CME
312CME Globex Interest Rate FuturesGlobex MDP 3.0CME
313CME Globex Interest Rate OptionsGlobex MDP 3.0CME
314CME Globex FX FuturesGlobex MDP 3.0CME
315CME Globex FX OptionsGlobex MDP 3.0CME
316CME Globex Commodity FuturesGlobex MDP 3.0CME
317CME Globex Commodity OptionsGlobex MDP 3.0CME
318CME Globex Equity Futures - excludes E-mini S&P 500Globex MDP 3.0CME
319CME Globex Equity Options - excludes E-mini S&P 500Globex MDP 3.0CME
320CME Globex FX Futures IIGlobex MDP 3.0CME
321CME Globex FX Options IIGlobex MDP 3.0CME
323CME Globex Micro Equity OptionsGlobex MDP 3.0CME
326Cryptocurrency - FuturesGlobex MDP 3.0CME
327Cryptocurrency - OptionsGlobex MDP 3.0CME
330SpotFX+Globex MDP 3.0CME
340CBOT Globex Commodity FuturesGlobex MDP 3.0CBOT
341CBOT Globex Commodity OptionsGlobex MDP 3.0CBOT
342CBOT Globex Equity Index FuturesGlobex MDP 3.0CBOT
343CBOT Globex Equity Index OptionsGlobex MDP 3.0CBOT
344CBOT Globex Interest Rate FuturesGlobex MDP 3.0CBOT
345CBOT Globex Interest Rate OptionsGlobex MDP 3.0CBOT
346CBOT Globex Commodity Futures IIGlobex MDP 3.0CBOT
348CBOT Globex Interest Rate Futures IIGlobex MDP 3.0CBOT
360COMEX Globex FuturesGlobex MDP 3.0COMEX
361COMEX Globex OptionsGlobex MDP 3.0COMEX
380NYMEX Globex Emissions FuturesGlobex MDP 3.0NYMEX
381NYMEX Globex Emissions OptionsGlobex MDP 3.0NYMEX
382NYMEX Globex Crude & Crude Refined FuturesGlobex MDP 3.0NYMEX
383NYMEX Globex Crude & Crude Refined OptionsGlobex MDP 3.0NYMEX
384NYMEX Globex Metals, Softs, & Alternative Market FuturesGlobex MDP 3.0NYMEX
385NYMEX Globex Metals, Softs, & Alternative Market OptionsGlobex MDP 3.0NYMEX
386NYMEX Globex Nat Gas & other Non-Crude Energy FuturesGlobex MDP 3.0NYMEX
387NYMEX Globex Nat Gas & other Non-Crude Energy OptionsGlobex MDP 3.0NYMEX
440GME Globex FuturesGlobex MDP 3.0GME
441GME Globex OptionsGlobex MDP 3.0GME
460MGEX Globex FuturesGlobex MDP 3.0MGEX
461MGEX Globex OptionsGlobex MDP 3.0MGEX

...

Certification via AutoCert+ is required for all client systems that will support CME Smart Stream on GCP. The following market data offerings have separate test suites:

  • Globex MDP 3.0
  • Data Insights - Settlement and Valuations
  • Streamlined - Block Trades
  • Streamlined - OTC Daily
  • Streamlined - CME CF Cryptocurrency Indices
  • Streamlined - Equity Indices
  • Streamlined - OTC Daily
  • BrokerTec U.S. Treasury Data

...

Clients planning to process BrokerTec U.S. Treasury market data on CME Smart Stream on GCP should be familiar with the functional specification available here.  

Market Data Support Services

Market data services provide the external data required to process CME Group market data.

Gliffy
macroId1ca81590-fdaf-46ab-8afc-c8d04b152c92
displayNameGCP SBE MD Support
nameGCP SBE MD Support
pagePin2

Support Services

 There are two MDP services available for markets on CME Globex: Core Globex SBE Schema and CME Reference Data API. An FTP site is used to store the schema files for all environments.

Service

Description

Core Globex SBE Schema 

MDP is a template-based SBE protocol wherein a given message is interpreted by means of its corresponding template. Each message contains a unique Schema ID that references the template to use to interpret the message. The core CME Globex SBE schema is used across all market data groups.

CME Reference Data API 

Product and instrument referential data for CME Smart Stream on GCP can be obtained via CME Reference Data API.

  • MDP 3.0 tag 48-SecurityID maps to the Reference data API instrument field globexSecurityId.
  • Additionally, client systems can map Settlements and Valuations tag 7178-ProductGUID to CME Reference Data API fields and map Settlements and Valuations tag 37513-InstrumentGUID to CME Reference Data API GUID fields.

...

The following diagram outlines the Publisher (CME Group) relationship with the Subscriber (Client System) for Google Cloud Pub/Sub technology.   

Gliffy
displayNamePub-Sub
namePub-Sub
pagePin3

...

4. The subscriber receives pending messages from its subscription and acknowledges each one to the Cloud Pub/Sub service.  A message contains the CME Group Simple Binary Encoded (SBE) messages.

5. When a message is acknowledged by the subscriber, it is removed from the subscription's message queue.

...

The following section outlines the CME MDP messaging within a Google Pub/Sub Message structure.  

CME MDP Included Data

The Google Pub/Sub data field provides the full CME Group UDP data packet.  

The following diagram outlines CME data with a Google Pub/Sub message:

Gliffy
namePubSub Message
pagePin2

...

Project NameTypeEnvironment
cmegroup-marketdata-cr-nrCryptocurrency OnlyNew Release
cmegroup-marketdata-newrelAll Other DataNew Release
cmegroup-marketdata-crCryptocurrency OnlyProduction
cmegroup-marketdataAll Other DataProduction

Topic names have the following methodology:  

NameValid ValuesDescription
EnvType

PROD = Production

NR = New Release

CERT = Certification 

Environment
ServiceNameSSCL = Smart Stream Cloud LinkCloud service provider
Provider

GCP = Google Cloud Products

Name of the cloud provider

ContentTypeMD = Market DataType of content
Content Timing

RT = Real Time

DLY = Delayed 

Time data is sent
ContentOwner

CMEG = CME Group Owned Market Data

SPDJI = S&P DowJones

USTBTEC = BrokerTec US Treasury Data

MGEX = Minneapolis Grain Exchange

GME = Gulf Mercantile Exchange

Reference to exchange
Format

FIXBIN = MDP 3.0 Globex Data Channels

STRBIN = Streamlined

SVBIN = Settlements & Valuations

Feed Type
Version

v01000

CME Group version of cloud solution
Topic Type

INCR = Incremental

RPLY = MBP Recovery

MBORPLY = MBO Recovery

INST = Instrument Recovery

Defines the type of topic
Channel Number

Examples:

230 = CME Block Trades

340 = CBOT Globex Commodity Futures

CME Group assigned channel ID

See Market Data Offerings via Google Cloud for channel IDs.

Product and instrument referential data for CME Smart Stream on GCP can be obtained via CME Reference Data API.

Topic Name Examples

The following are topic name examples:

...

Google Cloud Pub/Sub Message Replay

Google Cloud Pub/Sub includes the ability to replay messages. For more information on recovering messages, please refer to the Google quick start guide. (https://cloud.google.com/pubsub/docs/replay-qs).  

Google Cloud Pub/Sub replay is available for all CME MDP channels on Google Cloud Pub/Sub. Client systems that require data recovery from the beginning of the week on CME Globex can determine the weekly start via the incremental topic when the CME Group sequence number is equal to one.

Large-Scale Market Recovery

This section describes Market by Price (MBP) & Market by Order (MBO) recovery processing for MDP 3.0 channels via Google Cloud Pub/Sub. Market Recovery should be used for large-scale data recovery (i.e. major outage or late join) to synchronize client systems to the latest state maintained by CME Group.  Market by Price (MBP) & Market by Order (MBO) recovery have separate topics.

Once a subscriber requests to recover from a market recovery topic all Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages are looped 3 times for an applicable channel.  

Info
The Pub/Sub market recovery topic is different the from the MDP 3.0 implementation where the market feed continually iterates.

The following is an example of the messaging from the market recovery 3x topic loop:

Market Recovery Topic

CME Snapshot Sequence Number

Tag 369-LastMsgSeqNumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-TotNumReports

First Market Recovery Iteration (1)

1  - First message from market recovery topic

107 

20191010-18:36:41.000013217

8087

3

2 12020191010-18:36:41.0000330417733
313120191010-18:36:41.000054910121713

Next Market Recovery Iteration (2)

1 

140

20191120-18:36:41.000071886

8087

3

2

152

20191120-18:36:41.000093224

773

3

3

157 

20191120-18:36:41.000111773

12171

3

Final Market Recovery Iteration (3)

1 

166

20191120-18:36:41.000129878

8087

3

2

174

20191120-18:36:41.000150215

773

3

3 - Last message from market recovery topic

179

20191120-18:36:41.000172319

12171

3

A Market Data Snapshot Full Refresh (35=W) message will be included if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the recovery topic, even if the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for instruments having no book activity. More than one instrument can be updated in a packet. The following information is available in the Market Data Snapshot Full Refresh (35=W) message:

...

  • the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental topic

  • tag 60-TransactTime communicates the start transaction time of the last event for the instrument

    Market Recovery Topic - Snapshot (tag 35-MsgType=W) Message
    		Incremental Topic
    tag 369-LastMsgSeqNumProcessed corresponds topacket sequence number
    tag 60-TransactTime tag 60-TransactTime

...

Incremental Topic

Market Recovery Topic

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

15212

20191120-18:36:41.000368253

104- recover instrument 15212 information via market recovery snapshot message and begin normal processing


X

15212

20191120-18:36:41.000368253


TransactTime matches with snapshot message

1 – Connect to Market Recovery topic

104

20191120-18:36:41.000368253

 

TransactTime matches with incremental message

15212

10

...