CME Group offers industry-leading real-time Futures, Options, and Trade Blocks data through Google Pub/Sub, an asynchronous and scalable messaging service. Pub/Sub allows consumers to subscribe to CME Group’s Pub/Sub topics, where market events are published in real-time for client consumption. This topic includes the following content, including a Frequently Asked Questions (FAQ) and answer section:

Contents

Google Setup

Prior to using CME Group’s real-time data via Google Pub/Sub, ensure that you have a Google Cloud account created, along with a service / member account. An account admin can create and manage your service account(s) along with a member account from the Google Cloud Console’s IAM tab.

Use this service account to create and manage subscription(s) to CME Group’s market data. Typically, service accounts will follow a naming convention like “service_account_name@project-id.iam.gserviceaccount.com.” See additional resources about service accounts and member accounts.

CME Group Setup

To begin receiving CME Group data via Google Pub/Sub, visit the CME Group Data Services page: 

Open

1. From the top menu, navigate to Featured Data > CME Smart Stream.

2. Either create an account or sign-in to your existing CME Group account to proceed.

If receiving SBE data, it is required that a certification process be complete before receiving data. The onboarding process will ask for information about:  

• The company 

• Key contacts 

• The data application receiving and using data 

• Google Identity and Access Management (IAM) info

• Expected data usage (for derived products)

• Interested data products 

At the end of this process, a click-through certification agreement is available to sign for Cloud Connectivity. Once signed and approved by your CME Group Global Account Manager (GAM), you will receive access to CME Group Certification (CERT) and New Release (NR) data in Google Pub/Sub. 

Production Onboarding

Following the certification process, an Information License Agreement (ILA) and Production Cloud Connectivity Agreement must be signed and approved by both parties. For more information, reach out to the data Sales team along with your Global Account Manager.  

Once all agreements are signed, data access can be provided within an hour.

Accessing Data in Google Pub/Sub

To use Pub/Sub, ensure that the Pub/Sub API is enabled. This can be done by an account admin. Additionally, ensure that the member/service account can create subscriptions, otherwise an account admin needs to grant permissions.

See details on using this service in CME Smart Stream on GCP.

Pub/Sub Permissions

The following Google Cloud IAM role levels are available for Pub/Sub: 

• Pub/Sub Admin 

• Pub/Sub Editor 

• Pub/Sub Publisher (not needed to subscribe to CME Group’s messages)  

• Pub/Sub Subscriber 

• Pub/Sub Viewer 

Open

Creating a Pub/Sub Subscription

There are several ways to create a subscription to CME Group messages, including: 

• Google Cloud Console interface 

• gCloud Shell 

• REST API 

• Coding language of choice (C++, Java, Python, etc.) 

Subscriptions to CME data should be setup to “pull” the data, although there is also an option to directly stream the data into BigQuery. These parameters are defined while setting up the subscription.

If using the Google Cloud console - on the Subscriptions tab, ensure that the CME Group topic is Entered Manually.

For more documentation and sample codes, please refer to the Google Cloud “Create and manage Pub/Sub subscriptions” page. 

Pulling Pub/Sub Messages From a Subscription 

There are several ways to pull messages from subscriptions to CME Group data, including: 

• Google Cloud Console interface

• gCloud Shell

• BigQuery

• Coding language of choice (Python, C++, C#, Go, Java, Node.js, PHP, Ruby, etc.)

For more documentation and sample codes, please refer to the Google Cloud Publish and subscribe with client libraries documentation.  

Customer Support

For Pub/Sub inquiries or support, please contact the CME Group Global Command Center (GCC). 

CME Group Data via Google Pub Sub FAQ

Q: Can I use my regular Google email to connect to CME Group data?

Q: How come I cannot create a subscription?

1. Incorrect project name, topic name, or format

   1. Verify that the topic and project name are correct when creating the subscription.

2. Subscription created in the wrong organization or project

   1. Consumers must create a subscription in their own organization’s project and not within CME’s organization or project.

3. Logged in or authorized with the incorrect account (or incorrect permissions on account)

   1. Verify that you are authenticated/logged into the correct account to create your subscription. This is the same service account licensed with CME Group and will likely be in the format, “service_account_name@project-id.iam.gserviceaccount.com,” but may vary. You can also check if your account has permissions to create a subscription from your Google IAM page.

Q: What data topics are available for consumption? Why can't I see them?

Q: What is the naming format for JSON or SBE Pub/Sub topics?

•  

  • PROD.SSCL.GCP.MD.RT.CMEG.JSON.v01000.TOB.XCME.BTC

    • Production, JSON, Top of Book, Bitcoin Futures

  • PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310

    • Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310)

Q: What is the difference between real-time (RT) and delayed (DLY) topics? What are the real-time latencies?

•  

  • 30-80ms in the US

  • 80-120ms in EMEA

  • 180-200ms in APAC

•  

  • 80-100 in the US

  • 100-200 in EMEA

  • 200-300 in APAC

Q: How many price levels are provided for SBE and JSON Futures & Options order book data?

Q: Where can I find Google Cloud SDKs for working with Google Pub/Sub?

•  

  • Java: Subscriber Method

  • Python: Creating a Subscription

  • Pulling Data from Subscription: synchronously or asynchronously

  • Node.js Create Subscription

  • Listen for Messages

Q: Are there any network setting changes required such as opening ports for my firewall?

Q: Why am I seeing low volumes of data in Certification (CERT) and New Release (NR) data?

Q: What is required to receive Production data?

Q: How do I handle messages from Saturday testing?

•  

  • Throw out messages based upon timestamp which will incur Pub/Sub metering as a data pull from Pub/Sub.

  • Set your subscription Cache to less than 12 hours. Elegant if you do not need long term Caching in your processing.

  • Update your subscription Cache to 1 minute and then update to 7 days on Sunday mornings – This will cause Pub/Sub to drop all messages greater than 1 minute immediately and then begin to cache up to 7 days again. This will incur no added Pub/Sub charges.

  • Leverage the purge button in the GUI. 

  • Use Gutils to purge a subscription as of a certain time:

    • For example: gcloud alpha pubsub subscriptions seek SUBSCRIPTION_NAME --time=2021-02-03 12:00:00Z

    • Python

    • RestAPI