Packet Capture Dataset

Owned by CME Group
Last updated: Nov 01, 2024
5 min read

Market Data Packet Captures (PCAP) are the complete historical raw data feeds on all channels for both futures and options from the channels' MBO release. The Settlements and Valuations channel is a new channel available in PCAP format.

Contents

Other Datasets

Resources

Dates Available

Packet Capture data is available on four exchanges from January 2017 to present and all files are in FIX format. For the availability of each Channel please reference the FAQs below.

Please note that CME Term SOFR will not be available on Channel 312 after January 8th.


By Exchange


Exchange

Open Date

Close Date

CME (Electronic)

5/13/2017

Present

CBOT (Electronic)

4/1/2017

Present

NYMEX (Electronic)

1/7/2017

Present

COMEX (Electronic)

1/7/2017

Present



[Top]



Message Specifications

 
 Specifications

Field contains the following data:

  • Message Length: 2 bytes
  • Binary Packet header: 12 bytes
    • MsgSeqNum: 4 bytes
    • SendingTime: 8 bytes
  • Message Header: 10 bytes
    • MsgSize: 2 bytes (repeats here)
  • SBE Message Payload
    • SBE Header: 8 bytes
      • Block Length: 2 bytes
      • Template ID: 2 bytes
      • Schema ID: 2 bytes
      • Version: 2 bytes
  • SBE Message: MsgSize - 22 bytes

[Top]

Security Definition Files


 The market data Security Definition (tag 35-MsgType=d) message identifies the instrument and provides all instrument attributes such as expiration, strike price, etc.

Tag Usage by Security Type

TagFIX NameFuture OutrightNYMEX Decay/Variable FutureFuture SpreadOption OutrightOption SpreadUDS ComboUDS Covered
Application Sequence Control

5799

MatchEventIndicator

X

X

X

X

X

X

X

911

TotNumReports

c

c

c

c

c

c

c

980

SecurityUpdateAction

X

X

X

X

X

X

X

779

LastUpdateTime

X

X

X

X

X

X

X

1180

ApplID

X

X

X

X

X

X

X

Instrument Data

1300

MarketSegmentID

X

X

X

X

X

X

X

55

Symbol

X

X

X

X

X

X

X

48

SecurityID

X

X

X

X

X

X

X

22

SecurityIDSource

X

X

X

X

X

X

X

200

MaturityMonthYear

X

X

X

X

X

X

X

1151

SecurityGroup

X

X

X

X

X

X

X

6937

Asset

X

X

X

X

X

X

X

167

SecurityType

FUT

FUT

FUT

OOF

OOF

OOF

MLEG

762

SecuritySubType

–

–

X

–

X

X

X

461

CFI Code

X

X

X

X

X

X

X

201

PutOrCall

–

–

–

X

–

–

–

462

UnderlyingProduct

X

X

X

X

–

–

–

207

SecurityExchange

X

X

X

X

X

X

X

1682

MDSecurityTradingStatus

c

c

c

c

c

c

c

202

StrikePrice

–

–

–

X

–

–

–

947

StrikeCurrency

–

–

–

X

–

–

–

15

Currency

X

X

X

X

X

X

X

120

SettlCurrency


 –

 -

c

–

–

–

9850

MinCabPrice

–

–

–

c

–

–

–

9779

UserDefinedInstrument

–

–

–

–

–

Y

Y

Underlying (options only)

711

NoUnderlyings

-

-

-

X

-

-

-

→311

UnderlyingSymbol

-

-

-

X

-

-

-

→309

UnderlyingSecurityID

-

-

-

X

-

-

-

→305

UnderlyingSecurityIDSource

-

-

-

X

-

-

-

Leg Group (spreads only)

555

NoLegs

-

-

X

-

X

X

X

→602

LegSecurityID

-

-

X

-

X

X

X

→624

LegSide

-

-

X

-

X

X

X

→623

LegRatioQty

-

-

X

-

X

X

X

→566

LegPrice

-

-

-

-

-

-

X

→1017

LegOptionDelta

-

-

-

-

-

-

X

Trading Rules

1141

NoMdFeedTypes

X

X

X

X

X

X

X

→1022

MDFeedType

X

X

X

X

X

X

X

→264

MarketDepth

X

X

X

X

X

X

X

864

NoEvents

2

2

2

2

2

2

2

→865

EventType

X

X

X

X

X

X

X

→1145

EventTime

X

X

X

X

X

X

X

1142

MatchAlgorithm

X

X

X

X

X

X

X

1234

NoLotTypeRules

X

X

X

X

X

X

X

→1093

LotType

X

X

X

X

X

X

X

→1231

MinLotSize

X

X

X

X

X

X

X

562

MinTradeVol

X

X

X

X

X

X

X

1140

MaxTradeVol

X

X

X

X

X

X

X

969

MinPriceIncrement

X

X

X

c

X

X

X

1146

MinPriceIncrementAmount

X

X

-

X

-

-

-

9787

DisplayFactor

X

X

X

X

X

X

X

5770

PriceRatio

(Implied Intercommodity Ratio Spreads only)

–

–

c

–

–

–

–

6350

TickRule

–

–

–

c

c

–

–

37702

MainFraction

c

–

–

c

–

–

–

37703

SubFraction

c

–

–

c

–

–

–

9800

PriceDisplayFormat

c

_

 – 

c

–

–

–

Instrument Attribute Group

870

NoInstAttrib

X

X

X

X

X

X

X

→871=24

InstAttribType

X

X

X

X

X

X

X

→872

InstAttribValue

X

X

X

X

X

X

X

Contract Lot Size/Measure/Unit

1435

ContractMultiplierUnit

-

X

-

-

-

-

-

1439

FlowScheduleType

-

X

-

-

-

-

-

231

ContractMultiplier

-

X

-

-

-

-

-

996

UnitOfMeasure

X

X

X

X

X

X

X

1147

UnitOfMeasureQty

X

X

-

X

-

-

-

5818

DecayQty

-

c

-

-

-

-

-

5819

DecayStartDate

-

c

-

-

-

-

-

5849

OriginalContractSize

-

c

-

-

-

-

-

Statistics and Limits

1150

TradingReferencePrice

c

c

c

c

c

c

c

731

SettlPriceType

c

c

c

c

c

c

c

5796

TradingReferenceDate

c

c

c

c

c

c

c

5792

OpenInterestQty

c

c

c

c

c

c

c

5791

ClearedVolume

c

c

c

c

c

c

c

1149

HighLimitPrice

c

c

c

c

c

c

c

1148

LowLimitPrice

c

c

c

c

c

c

c

1143

MaxPriceVariation

c

c

c

–

–

–

–

[Top]


Sample Files

In order to obtain a sample PCAP file you must set up an s3 account and then contact CME Group's Data Sales team @CMEDataSales@cmegroup.com.  You can find instruction on how to entitle your s3 account to DataMine data here. 

Layout Guides

View layout guides and message specifications for the following market data formats.

CME MDP 3.0 (Current Production Format)

The CME Group Market Data Platform (MDP) disseminates event-based bid, ask, trade, and statistical data for CME Group markets and also provides recovery and support services for market data processing. Click below to access layout guide.

[Top]



FAQ

 PCAP Format

What format is the file delivered in?

PCAP files are SBE raw data grouped by channel. The file contains channel number, length of message and actual payload.  For example: 312 (channel), 96 (length of message), <actual payload>.

How do I best understand the timestamp(s)?

All messages will have FIX tag 60 (within payload), this represents the Globex started processing time at nanosecond granularity with microsecond accuracy.

What is the precision level for timestamping in PCAP?

PCAP is delivered in nanosecond granularity with microsecond accuracy.

Does PCAP come from the same source as Top-of-Book and Time & Sales data?

No, our PCAP files are produced from CME Globex Market Data Platform FIX/FAST feeds, while our Top-of-Book and Time & Sales data come from an internal post-trade processing system. The granularity of the timestamps for the data offerings is different due to the sources.

 PCAP Availability

What is the Settlements and Valuations Channel?

With the launch of Settlements and Valuations, CME Group introduced theoretical valuations. Theoretical valuations is new type of data produced by CME Group for futures and options products without volume, open interest, or underlying pricing required for Clearing products. For more information, see here.

How far back do you maintain MBO records?

ChannelChannel DescriptionStart DateEnd Date
310CME Globex Equity Futures5/20/2017Present
311CME Globex Equity Options5/20/2017Present
312CME Globex Interest Rate Futures5/20/2017Present
313CME Globex Interest Rate Options5/20/2017Present
314CME Globex FX Futures5/13/2017Present
315CME Globex FX Options5/13/2017Present
316CME Globex Commodity Futures5/13/2017Present
317CME Globex Commodity Options5/13/2017Present
318CME Globex Equity Futures- excludes E-mini S&P 5005/13/2017Present
319CME Globex Equity Options- excludes E-mini S&P 5005/20/2017Present
320CME Globex FX Futures II5/13/2017Present
321CME Globex FX Options II5/13/2017Present
340CBOT Globex Commodity Futures4/1/2017Present
341CBOT Globex Commodity Options4/1/2017Present
342CBOT Globex Equity Index Futures4/22/2017Present
343CBOT Globex Equity Index Options4/22/2017Present
344CBOT Globex Interest Rate Futures4/22/2017Present
345CBOT Globex Interest Rate Options4/22/2017Present
346CBOT Globex Commodity Futures II4/1/2017Present
360COMEX Globex Futures1/7/2017Present
361COMEX Globex Options1/7/2017Present
380NYMEX Globex Emissions Futures1/7/2017Present
381NYMEX Globex Emissions Options1/7/2017Present
382NYMEX Globex Crude & Crude Refined Futures1/21/2017Present
383NYMEX Globex Crude & Crude Refined Options1/7/2017Present
384NYMEX Globex Metals, Softs, & Alternative Market Futures1/7/2017Present
385NYMEX Globex Metals, Softs, & Alternative Market Options1/7/2017Present
386NYMEX Globex Nat Gas & other Non-Crude Energy Futures1/7/2017Present
387NYMEX Globex Nat Gas & other Non-Crude Energy Options1/7/2017Present

What product is a channel on?

Please reference the CME Globex Product Reference Sheet to see the MDP channel for each product.  Please note the channel is listed in column AC.

How many levels of depth are available? Has this depth changed over time?

PCAP has the Full depth of book. This has not changed over time.

Are spreads included in the PCAP files?

 Exchange-listed spread products are included for both futures and options spreads.

Do you have sample files available?

In order to obtain a sample PCAP file you must set up an s3 account and then contact CME Group's Data Sales team CMEDataSales@CMEGroup.com. You can find instruction on how to entitle your s3 account to DataMine data here. 

Are there any anomolies with this data?

For Channel 381, NYMEX Globex Emissions Options, there is a data gap from 6/27/2021 - 7/12/2021

 PCAP Delivery

When are these files delivered?

CME Group’s process to distribute PCAP files begins at 5:00 p.m. Central Time (CT) and is typically complete by midnight.

How many files do I receive per day for a PCAP subscription?

You will receive 2 files per day for each channel you are subscribed to. The first will have the -e in the file name. This is data that occurred from 4:30 PM CT to Midnight CT. The second file will have the -f in the file name and will be from Midnight CT to 4:00 PM CT. On Sundays, you will receive all data from the start of the feed. The earliest heartbeat messages that are captured have a timestamp of 5:30 AM CT.

If I purchase daily updates of PCAP data, will I get historical data as well?

No. When an order is placed for daily updates of PCAP data, the first file included will be generated for the start date of the subscription. However, files remain accessible for 30 days after purchase, enabling the customer to reference previous day’s data.

 PCAP Interpretation

Where can I obtain Security Definitions for the data in the PCAP data?

A security definition file is published daily and sent to you with the PCAP file.

How do I interpret PCAP data?

PCAP files include all bids and offers, full depth of the book.

How large are the PCAP files?

The size of the file varies upon the channel and date.  On our busiest days in 2018, channel 311 (Equity options) were averaging between 7.5 GB and 10 GB a day. 

Will a channel be missing packet level sequence numbers?

No you should receive all packet level sequence numbers.

Should I be missing instrument level sequence numbers (Tag 83)?

No, all instrument level sequence numbers should be in each file.

Will I see heartbeat messages?

Yes, you will see these messages.

Can trade summaries be split into different files?

Yes, if a strategy is traded and the instruments are on two different channels, the summary will be split between the two different channels.

Should an Order ID (Tag 37) change during the life of an order?

No, tag 37 should stay the same for each order until it has been cancelled or filled.  If there is any modification other than a cancel or fill the Order ID stays the same.

Should orders be tied to implied prices?

No, orders are not tied to implied prices.

 



[Top]



How was your Client Systems Wiki Experience? Submit Feedback

Copyright © 2024 CME Group Inc. All rights reserved.