CME Group offers industry-leading real-time Futures, Options, and Trade Blocks data through Google Pub/Sub, an asynchronous and scalable messaging service. Pub/Sub allows consumers to subscribe to CME Group’s Pub/Sub topics, where market events are published in real-time for client consumption. This topic includes the following content, including a Frequently Asked Questions (FAQ) and answer section:  

Contents

Google Setup

Prior to using CME Group’s real-time data via Google Pub/Sub, ensure that you have a Google Cloud account created, along with a service / member account. An account admin can create and manage your service account(s) along with a member account from the Google Cloud Console’s IAM tab.

Use this service account to create and manage subscription(s) to CME Group’s market data. Typically, service accounts will follow a naming convention like “service_account_name@project-id.iam.gserviceaccount.com.” See additional resources about service accounts and member accounts.

CME Group Setup

To begin receiving CME Group data via Google Pub/Sub, visit the CME Group Data Services page: 

Open

1. From the top menu, navigate to Featured Data > CME Smart Stream.

2. Either create an account or sign-in to your existing CME Group account to proceed.

If receiving SBE data, it is required that a certification process be complete before receiving data. The onboarding process will ask for information about:  

• The company 

• Key contacts 

• The data application receiving and using data 

• Google Identity and Access Management (IAM) info

• Expected data usage (for derived products)

• Interested data products 

At the end of this process, a click-through certification agreement is available to sign for Cloud Connectivity. Once signed and approved by your CME Group Global Account Manager (GAM), you will receive access to CME Group Certification (CERT) and New Release (NR) data in Google Pub/Sub. 

Production Onboarding

Following the certification process, an Information License Agreement (ILA) and Production Cloud Connectivity Agreement must be signed and approved by both parties. For more information, reach out to the data Sales team along with your Global Account Manager.  

Once all agreements are signed, data access can be provided within an hour.

Accessing Data in Google Pub/Sub

To use Pub/Sub, ensure that the Pub/Sub API is enabled. This can be done by an account admin. Additionally, ensure that the member/service account can create subscriptions, otherwise an account admin needs to grant permissions.

See details on using this service in CME Smart Stream on GCP.

Pub/Sub Permissions

The following Google Cloud IAM role levels are available for Pub/Sub: 

• Pub/Sub Admin 

• Pub/Sub Editor 

• Pub/Sub Publisher (not needed to subscribe to CME Group’s messages)  

• Pub/Sub Subscriber 

• Pub/Sub Viewer 

Open

Creating a Pub/Sub Subscription

There are several ways to create a subscription to CME Group messages, including: 

• Google Cloud Console interface 

• gCloud Shell 

• REST API 

• Coding language of choice (C++, Java, Python, etc.) 

Subscriptions to CME data should be setup to “pull” the data, although there is also an option to directly stream the data into BigQuery. These parameters are defined while setting up the subscription.

If using the Google Cloud console - on the Subscriptions tab, ensure that the CME Group topic is Entered Manually.

For more documentation and sample codes, please refer to the Google Cloud “Create and manage Pub/Sub subscriptions” page. 

Pulling Pub/Sub Messages From a Subscription 

There are several ways to pull messages from subscriptions to CME Group data, including: 

• Google Cloud Console interface

• gCloud Shell

• BigQuery

• Coding language of choice (Python, C++, C#, Go, Java, Node.js, PHP, Ruby, etc.)

For more documentation and sample codes, please refer to the Google Cloud Publish and subscribe with client libraries documentation.  

Customer Support

For Pub/Sub inquiries or support, please contact the CME Group Global Command Center (GCC). 

CME Group Data via Google Pub Sub FAQ

Q: Can I use my regular Google email to connect to CME Group data?

A: No, generally you will need to use a service account. This comes in the format: service_account_name@project-id.iam.gserviceaccount.com but can vary.

Q: How come I cannot create a subscription?

A: There are a few possible reasons why you were unable to create a subscription. Below are possible reasons along with ways to fix them.

1. Incorrect project name, topic name, or format

   • Verify that the topic and project name are correct when creating the subscription.

2. Subscription created in the wrong organization or project

   • Consumers must create a subscription in their own organization’s project and not within CME’s organization or project.

3. Logged in or authorized with the incorrect account (or incorrect permissions on account)

   • Verify that you are authenticated/logged into the correct account to create your subscription. This is the same service account licensed with CME Group and will likely be in the format, “service_account_name@project-id.iam.gserviceaccount.com,” but may vary. You can also check if your account has permissions to create a subscription from your Google IAM page.

Q: What data topics are available for consumption? Why can't I see them?

A: There are over 8000+ JSON topics and 200+ SBE topics available. Either all JSON topics and/or all SBE topics will be made available based on licensing. This does not mean that the topics appear in your Google account. Rather, the licensing enables subscriptions, but the data topics continue to remain in CME Group's account.

For access to additional topics, ensure that you have the correct licensing in place.

Q: What is the naming format for JSON or SBE Pub/Sub topics?

A: Topics follow a set naming format for JSON or for SBE. Examples for each respectively:

• PROD.SSCL.GCP.MD.RT.CMEG.JSON.v01000.TOB.XCME.BTC

  • Production, JSON, Top of Book, Bitcoin Futures

• PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310

  • Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310)

Additionally, a complete list of the available designated contract markets and instruments offered through Pub/Sub is available from the CME Globex Product Reference Sheet or use Reference Data API to automate market understanding for all CME Group and hosted partner markets.

For more information on the topic naming format, please reference the JSON topic and SBE topic.

Q: What is the difference between real-time (RT) and delayed (DLY) topics? What are the real-time latencies?

A: Real-time data provides slightly different latencies depending on the format you take. Additionally, latencies vary depending on market conditions and other external factors. The below latencies are taken on an average market data message day.

SBE latency:

• 30-80ms in the US

• 80-120ms in EMEA

• 180-200ms in APAC

JSON latency:

• 80-100 in the US

• 100-200 in EMEA

• 200-300 in APAC

Data is hosted in Google Cloud US Central – Iowa. Delayed data adds a 10-minute delay to the delivery to support different use cases.

Q: How many price levels are provided for SBE and JSON Futures & Options order book data?

A: SBE provides either Market by Order (MBO) or Market Depth. MBO includes all market orders while Market Depth provides 10 levels deep for futures, and 5 levels deep for options.

JSON provides either the top of book or up to 2 price levels of the order book, along with trade information (TRD) or trade statistics (STAT).

Q: Where can I find Google Cloud SDKs for working with Google Pub/Sub?

A: Google Cloud SDK documentation

Common SDK's and Key Routines

• Java: Subscriber Method

• Python: Creating a Subscription

• Pulling Data from Subscription: synchronously or asynchronously

• Node.js Create Subscription

• Listen for Messages

Q: Are there any network setting changes required such as opening ports for my firewall?

A: Most will not be impacted but if so, all interactions are over port 443, only over HTTPS. All Google Cloud APIs are HTTPS only.

Google Cloud / CME Group does not have service specific CIDR ranges. There are large netblocks that cover all of Cloud APIs. Please see this Google content if you want to extract *.googleapis.com blocks.

If trying to whitelist ranges with more specificity, use L7 firewall capabilities if edge equipment can filter egress by subdomains: e.g., pubsub.googleapis.com, oauth2.googleapis.com.

If additional support is needed, please contact the CME Group Global Command Center (GCC). 

Q: Why am I seeing low volumes of data in Certification (CERT) and New Release (NR) data?

A: CERT and NR data typically publishes data at significantly lower rates. Please keep this in mind prior to shifting to Production data as it may require some refactoring to manage the Production level data velocity.

Q: What is required to receive Production data?

A: For SBE data, both an Information License Agreement and certification testing must be completed.

For JSON data, only an Information License Agreement is needed.

Q: How do I handle messages from Saturday testing?

A: CME Group can cut off the connection on Saturday for usual testing, and CME Group also supports client testing on Saturday via the channels.

CME Group normally produces email notices about testing and thus clients should have a programmatic solution to handle any Saturday messages as to not rely on email notices. There are a couple of options that CME Group recommends:

• Throw out messages based upon timestamp which will incur Pub/Sub metering as a data pull from Pub/Sub.

• Set your subscription Cache to less than 12 hours. Elegant if you do not need long term Caching in your processing.

• Update your subscription Cache to 1 minute and then update to 7 days on Sunday mornings – This will cause Pub/Sub to drop all messages greater than 1 minute immediately and then begin to cache up to 7 days again. This will incur no added Pub/Sub charges.

• Leverage the purge button in the GUI. 

• Use Gutils to purge a subscription as of a certain time:

  • For example: gcloud alpha pubsub subscriptions seek SUBSCRIPTION_NAME --time=2021-02-03 12:00:00Z

  • Python

  • RestAPI