CME Group offers industry-leading real-time Futures, Options, and Trade Blocks data through Google Pub/Sub, an asynchronous and scalable messaging service. Pub/Sub allows consumers to subscribe to CME Group’s Pub/Sub topics, where market events are published in real-time for client consumption. This topic includes the following content, including a Frequently Asked Questions (FAQ) and answer section:
Contents
Table of Contents
...
Use this service account to create and manage subscription(s) to CME Group’s market data. Typically, service accounts will follow a naming convention like “service_account_name@project-id.iam.gserviceaccount.com.”See additional resources about service accounts andmember accounts.
CME Group Setup
To begin receiving CME Group data via Google Pub/Sub, visit the CME Group Data Services page:
...
| Info |
|---|
While creating the subscription, enter the Cloud Pub/Sub topic manually and include the full CME Topic Name. This will follow the format "projects/<project_name>/topics/<topic_name>" Example: projects/cmegroup-marketdata-newrel/topics/CERT.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.MBORPLY.311 |
If using the Google Cloud console - on the Subscriptions tab, ensure that the CME Group topic is Entered Manually.
For more documentation and sample codes, please refer to the Google Cloud “Create and manage Pub/Sub subscriptions” page.
...
CME Group Data via Google Pub Sub FAQ
Q: Can I use my regular Google email to connect to CME Group data?
A: No, generally you will need to use a service account. This comes in the format: service_account_name@project-id.iam.gserviceaccount.com but can vary.
Q: How come I cannot create a subscription?
A: There are a few possible reasons why you were unable to create a subscription. Below are possible reasons along with ways to fix them.
- Incorrect project name, topic name, or format
- Verify that the topic and project name are correct when creating the subscription.
- Subscription created in the wrong organization or project
- Consumers must create a subscription in their own organization’s project and not within CME’s organization or project.
- Logged in or authorized with the incorrect account (or incorrect permissions on account)
- Verify that you are authenticated/logged into the correct account to create your subscription. This is the same service account licensed with CME Group and will likely be in the format, “service_account_name@project-id.iam.gserviceaccount.com,” but may vary. You can also check if your account has permissions to create a subscription from your Google IAM page.
Q: What data topics are available for consumption? Why can't I see them?
A: There are over 8000+ JSON topics and 200+ SBE topics available. Either all JSON topics and/or all SBE topics will be made available based on licensing. This does not mean that the topics appear in your Google account. Rather, the licensing enables subscriptions, but the data topics continue to remain in CME Group's account.
For access to additional topics, ensure that you have the correct licensing in place.
Q: What is the naming format for JSON or SBE Pub/Sub topics?
A: Topics follow a set naming format for JSON or for SBE. Examples for each respectively:
- PROD.SSCL.GCP.MD.RT.CMEG.JSON.v01000.TOB.XCME.BTC
- Production, JSON, Top of Book, Bitcoin Futures
- PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310
- Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310)
Additionally, a complete list of the available designated contract markets and instruments offered through Pub/Sub is available from the CME Globex Product Reference Sheet or use Reference Data API to automate market understanding for all CME Group and hosted partner markets.
For more information on the topic naming format, please reference the JSON topic and SBE topic.
Q: What is the difference between real-time (RT) and delayed (DLY) topics? What are the real-time latencies?
A: Real-time data provides slightly different latencies depending on the format you take. Additionally, latencies vary depending on market conditions and other external factors. The below latencies are taken on an average market data message day.
SBE latency:
- 30-80ms in the US
- 80-120ms in EMEA
- 180-200ms in APAC
JSON latency:
- 80-100 in the US
- 100-200 in EMEA
- 200-300 in APAC
Data is hosted in Google Cloud US Central – Iowa. Delayed data adds a 10-minute delay to the delivery to support different use cases.
Q: How many price levels are provided for SBE and JSON Futures & Options order book data?
A: SBE provides either Market by Order (MBO) or Market Depth. MBO includes all market orders while Market Depth provides 10 levels deep for futures, and 5 levels deep for options.
JSON provides either the top of book or up to 2 price levels of the order book, along with trade information (TRD) or trade statistics (STAT).
Q: Where can I find Google Cloud SDKs for working with Google Pub/Sub?
A: Google Cloud SDK documentation
Common SDK's and Key Routines
- Java: Subscriber Method
- Python: Creating a Subscription
- Pulling Data from Subscription: synchronously or asynchronously
- Node.js Create Subscription
- Listen for Messages
Q: Are there any network setting changes required such as opening ports for my firewall?
A: Most will not be impacted but if so, all interactions are over port 443, only over HTTPS. All Google Cloud APIs are HTTPS only.
Google Cloud / CME Group does not have service specific CIDR ranges. There are large netblocks that cover all of Cloud APIs. Please see this Google content if you want to extract *.googleapis.com blocks.
If trying to whitelist ranges with more specificity, use L7 firewall capabilities if edge equipment can filter egress by subdomains: e.g., pubsub.googleapis.com, oauth2.googleapis.com.
If additional support is needed, please contact the CME Group Global Command Center (GCC).
Q: Why am I seeing low volumes of data in Certification (CERT) and New Release (NR) data?
A: CERT and NR data typically publishes data at significantly lower rates. Please keep this in mind prior to shifting to Production data as it may require some refactoring to manage the Production level data velocity.
Q: What is required to receive Production data?
A:
...
For SBE data, both an Information License Agreement and certification testing must be completed.
For JSON data, only an Information License Agreement is needed.
Q: How do I handle messages from Saturday testing?
A: CME Group can cut off the connection on Saturday for usual testing, and CME Group also supports client testing on Saturday via the channels.
CME Group normally produces email notices about testing and thus clients should have a programmatic solution to handle any Saturday messages as to not rely on email notices. There are a couple of options that CME Group recommends:
- Throw out messages based upon timestamp which will incur Pub/Sub metering as a data pull from Pub/Sub.
- Set your subscription Cache to less than 12 hours. Elegant if you do not need long term Caching in your processing.
- Update your subscription Cache to 1 minute and then update to 7 days on Sunday mornings – This will cause Pub/Sub to drop all messages greater than 1 minute immediately and then begin to cache up to 7 days again. This will incur no added Pub/Sub charges.
- Leverage the purge button in the GUI.
- Use Gutils to purge a subscription as of a certain time: