CME Group offers industry-leading real-time Futures, Options, and Trade Blocks Blocks data through Google Pub/SubSub, an asynchronous and scalable messaging service. Pub/Sub allows consumers to subscribe to CME Group’s Pub/Sub topics, where market events are published in real-time for client consumption. This topic includes the following content, including a Frequently Asked Questions (FAQ) and answer section:  

Contents

Google Setup

Prior to using CME Group’s real-time data via Google Pub/Sub, ensure that you have a Google Cloud account created, along with a service / member account. An account admin can create and manage your service account(s) along with a member account from the Google Cloud Console’s IAM tab.

Use this service account to create and manage subscription(s) to CME Group’s market data. Typically, service accounts will follow a naming convention like “service_account_name@project-id.iam.gserviceaccount.com.” See additional resources about service accounts and and member accounts.

CME Group Setup

To begin receiving CME Group data via Google Pub/Sub, visit the CME Group Data Services page: Image Removed

...

1. From the top menu, navigate to Featured Data > CME Smart Stream.

2. Either create an account or sign-in to your existing CME Group account to proceed.

If receiving SBE data, it is required that a certification process be complete before receiving data. The onboarding process will ask for information about:    

• The

  company 

  company 

• Key

  contacts 

  contacts 

• The data application receiving and using

  data 

  data 

• Google Identity and Access Management (IAM) info

• Expected data usage (for derived products)

• Interested data

  products 

  products 

At the end of this process, a click-through certification agreement is available to sign for Cloud Connectivity. Once signed and approved by your CME Group Global Account Manager (GAM), you will receive access to CME Group Certification (CERT) and New Release (NR) data in Google Pub/Sub. 

Image Modified 

Production Onboarding

Following the certification process, an Information License Agreement (ILA) and Production Cloud Connectivity Agreement must be signed and approved by both parties. For more information, reach out to the data Sales team along with your Global Account Manager.    

Once all agreements are signed, data access can be provided within an hour.

Accessing Data in Google Pub/Sub

To use Pub/Sub, ensure that the Pub/Sub API is enabled. This can be done by an account admin. Additionally, ensure that the member/service account can create subscriptions, otherwiseotherwise an account admin needs needs to grant permissions.

See details on using this service in CME Smart Stream on GCP.

Pub/Sub Permissions

The following Google Cloud IAM role levels are available for Pub/Sub: 

• Pub/Sub

  Admin 

  Admin 

• Pub/Sub

  Editor 

  Editor 

• Pub/Sub Publisher (not needed to subscribe to CME Group’s messages)

    

    

• Pub/Sub

  Subscriber 

  Subscriber 

• Pub/Sub

  Viewer 

  Viewer 

...

More information on Pub/Sub permissions is available on the Google site. 

Creating a Pub/Sub Subscription

There are several ways to create a subscription to CME Group messages, including: 

• Google Cloud Console

  interface 

  interface 

• gCloud

  Shell 

  Shell 

• REST

  API 

  API 

• Coding language of choice (C++, Java, Python, etc.) 

Subscriptions to CME data should be setup to “pull” the data, although there is also an option to directly stream the data into BigQuery. These parameters are defined while setting up the subscription.

If using the Google Cloud console - on the Subscriptions tab, ensure that the CME Group topic is Entered Manually.Image Removed

...

For more documentation and sample codes, please refer to the Google Cloud “Create and manage Pub/Sub subscriptions” page. 

Pulling Pub/Sub Messages From a Subscription 

There are several ways to pull messages from subscriptions to CME Group data, including: 

• Google Cloud Console interface

• gCloud Shell

• BigQuery

• Coding language of choice (Python, C++, C#, Go, Java, Node.js, PHP, Ruby, etc.)

For more documentation and sample codes, please refer to the Google Cloud Publish and subscribe with client libraries documentation.    

Customer Support

For Pub/Sub inquiries or support, please contact the CME Group Global Command Center (GCC). 

CME Group Data via Google Pub Sub FAQ

Q: Can  Can I use my regular Google email to connect to CME Group data?

...

A: There are a few possible reasons why you were unable to create a subscription. Below are possible reasons along with ways to fix them.

1. Incorrect project name, topic name, or format

   • Verify that the topic and project name are correct when creating the subscription.

2. Subscription created in the wrong organization or project

   • Consumers must create a subscription in their own organization’s project and not within CME’s organization or project.

3. Logged in or authorized with the incorrect account (or incorrect permissions on account)

   • Verify that you are authenticated/logged into the correct account to create your subscription. This is the same service account licensed with CME Group and will likely be in the format, “service_account_name@project-id.iam.gserviceaccount.com,” but may vary. You can also check if your account has permissions to create a subscription from your Google IAM page.

Q: What data topics are available for consumption? Why can't I see them?

A: There are over 8000+ JSON topics and 200+ SBE topics available. Either all JSON topics and/or all SBE topics will be made available based on licensing. This does not mean that the topics appear in your Google account. Rather, the licensing enables subscriptions, but the data topics continue to remain in CME Group's account.

...

A: Topics follow a set naming format for JSON or for SBE. Examples for each respectively:

• PROD.SSCL.GCP.MD.RT.CMEG.JSON.v01000.TOB.XCME.BTC

  • Production, JSON, Top of Book, Bitcoin Futures

• PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310

  • Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310)

Additionally, a complete list of the available designated contract markets and instruments offered through Pub/Sub is available from the CME Globex Product Reference Sheet or use Reference Data API to automate market understanding for all CME Group and hosted partner markets.

For more information on the topic naming format, please reference the JSON topic and the JSON topic and SBE topic.

Q: What is the difference between real-time (RT) and delayed (DLY) topics? What are the real-time latencies?

A: Real-time data provides slightly different latencies depending on the format you take. Additionally, latencies vary depending on market conditions and other external factors. The below latencies are taken on an average market data message day.

SBE latency:

• 30-80ms in the US

• 80-120ms in EMEA

• 180-200ms in APAC

JSON latency:

• 80-100 in the US

• 100-200 in EMEA

• 200-300 in APAC

Data is hosted in Google Cloud US Central – Iowa. Delayed data adds a 10-minute delay to the delivery to support different use cases.

...

Common SDK's and Key Routines

• Java: Subscriber Method

• Python: Creating a Subscription

• Pulling Data from Subscription: synchronously or asynchronously

• Node.js Create Subscription

• Listen for Messages

Q: Are there any network setting changes required such as opening ports for my firewall?

...

CME Group normally produces email notices about testing and thus clients should have a programmatic solution to handle any Saturday messages as to not rely on email notices. There are a couple of options that CME Group recommends:

• Throw out messages based upon timestamp which will incur Pub/Sub metering as a data pull from Pub/Sub.

• Set your subscription Cache to less than 12 hours. Elegant if you do not need long term Caching in your processing.

• Update your subscription Cache to 1 minute and then update to 7 days on Sunday mornings – This will cause Pub/Sub to drop all messages greater than 1 minute immediately and then begin to cache up to 7 days again. This will incur no added Pub/Sub charges.

• Leverage the purge button in the GUI. 

• Use Gutils to purge a subscription as of a certain time:

  • For example: gcloud alpha pubsub subscriptions seek SUBSCRIPTION_NAME --time=2021-02-03 12:00:00Z

  • Python

  • RestAPI