Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device.
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
This topic provides CME Group clients and vendors with information on accessing and utilizing Google Cloud data sharing platforms to exchange data with CME Group, either by publishing or consuming data. Clients and vendors can take advantage of this capability with out having any presence in Google Cloud.
CME Group leverages Google Cloud Storage (GCS) to collaborate on resources and data.
Google Cloud Storage: facilitates file uploads and downloads between CME Group and onboarded and permissioned clients and vendors. This is a scalable way to upload and download files from CME Group.
Create and Entitle an OAuth API ID: Access the CME Customer Center to create an OAuth API ID via My Profile > APIs > Create an API ID > Type OAuth.
Create API ID in full lowercase to avoid any Google entitlement conflict and suffix it with the environment you want to access. api_example_dev api_example_qa api_example_uat api_example_prod
Fill Out an Intake Form: Submit anIntake Form to initiate the process of entitling your API ID with the correct GCS Bucket access.
Role - Whether you want to read/download files from CME Group or publish files to CME Group
Environment - Select which bucket environment to access (DEV, QA, UAT, PROD)
CME Contact - The email address of a CME Group employee who is the liaison for the data sharing initiative
Once you submit the form, you will get a confirmation email that your API ID is successfully entitled with the correct permissions. Additionally, your CME Group contact will give you details on the buckets as well as file paths to access. After receiving confirmation, proceed with step 4 and accessing the data from GCS.
Generate Google Access Token using CME OAuth: Use the newly created API ID, along with the corresponding API passwordto generate a short lived Google access token to send or receive data from CME provided Google Cloud Storage bucket for your organization. SeeCloud Data Exchange for step by step details.
Access Shared Data in Google Cloud Storage
This section provides methods to obtain a Google access token to access and operate Google Cloud Storage buckets.
Python Example
To run this example code, you may need to install few libraries using below command:
You may need to substitute proper values from line 8 - 18
CME_TOKEN_URL should be replaced with below values:
import jwt
import time
import json
import requests
from google.cloud import storage
from google.auth import exceptions
from google.auth import identity_pool
TOKEN_URL = 'https://auth.cmegroup.com/as/token.oauth2'
API_ID="YOUR_API_ID"
API_PASSWORD="YOUR_API_PASSWORD"
# Replace with your bucket name and file details
#for upload
BUCKET_NAME = "your-bucket-name"
FILE_PATH = "path/to/your/local/file.txt" # Path to the file you want to upload
DESTINATION_BLOB_NAME = "path/in/bucket/uploaded_file.txt" # Desired path in GCS
#for download
BLOB_NAME_TO_DOWNLOAD = "path/in/bucket/file_to_download.txt" # Path to the file in GCS
DESTINATION_FILE_PATH = "/path/to/local/downloaded_file.txt" # Where to save the downloaded file
#For non prod environments, please check with CME Data Platform team for audience value
AUDIENCE = "//iam.googleapis.com/projects/282603793014/locations/global/workloadIdentityPools/iamwip-cmegroup/providers/customer-federation" # Set GCP Audience.
class CMETokenSupplier(identity_pool.SubjectTokenSupplier):
def __init__(self,token_url,api_id ,api_password):
self.token_url=token_url
self.api_id=api_id
self.api_password=api_password
def get_subject_token(self, context, request):
audience=self.api_id
algorithms = ["RS256"]
response = requests.post(
self.token_url,
data={"grant_type": "client_credentials"},
auth=(self.api_id, self.api_password)
)
if(response.status_code==200):
access_token = response.json()["access_token"]
return access_token
else:
print(response.text)
return None
def upload_file_with_oidc(bucket_name, file_path, destination_blob_name,credentials):
"""Uploads a file to GCS"""
storage_client = storage.Client(credentials=credentials) # Don't use default credentials
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
try:
# Upload the file using the authenticated session
blob.upload_from_filename(file_path)
except Exception as e:
print(f"Error uploading file: {e}")
return False # Indicate failure
return True # Indicate success
def list_bucket_contents_with_oidc(bucket_name, credentials):
"""Lists the contents of a GCS bucket"""
storage_client = storage.Client(credentials=credentials) # Don't use default credentials
bucket = storage_client.bucket(bucket_name)
try:
# List blobs (files) in the bucket using the authenticated session
blobs = bucket.list_blobs()
print(f"Contents of gs://{bucket_name}:")
for blob in blobs:
print(blob.name)
except Exception as e:
print(f"Error listing bucket contents: {e}")
return False # Indicate failure
return True # Indicate success
def download_file_with_oidc(bucket_name, blob_name, destination_file_path, credentials):
"""Downloads a file from GCS"""
storage_client = storage.Client(credentials=credentials) # Don't use default credentials
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
try:
# Download the file using the authenticated session
blob.download_to_filename(destination_file_path)
print(f"File gs://{bucket_name}/{blob_name} downloaded to {destination_file_path} using OIDC token.")
except Exception as e:
print(f"Error downloading file: {e}")
return False # Indicate failure
return True # Indicate success
if __name__ == "__main__":
supplier = CMETokenSupplier(TOKEN_URL,API_ID,API_PASSWORD)
credentials = identity_pool.Credentials(
AUDIENCE, # Set GCP Audience.
"urn:ietf:params:oauth:token-type:jwt", # Set subject token type.
subject_token_supplier=supplier, # Set supplier.
)
#below lines work based on permissions attached to identity
upload_file_with_oidc(BUCKET_NAME,FILE_PATH,DESTINATION_BLOB_NAME,credentials) # can be used to upload files to bucket if enough permissions are there
list_bucket_contents_with_oidc(BUCKET_NAME,credentials) # can be used to list bucket contents if enough permissions are there
download_file_with_oidc(BUCKET_NAME,BLOB_NAME_TO_DOWNLOAD,DESTINATION_FILE_PATH,credentials) # can be used to download file from bucket if enough permissions are there
Java Example
The following example demonstrates how to get a token and perform operations on a bucket with a Java Maven project:
Java Main class (Main.java)
Substitute proper values from line 5 to 11.
package org.example;
import java.io.IOException;
public class Main {
private static final String TOKEN_ENDPOINT="https://auth.cmegroup.com/as/token.oauth2";
private static final String API_ID="YOUR_API_ID";
private static final String API_PASSWORD="YOUR_API_PASSWORD";
private static final String BUCKET_NAME="YOUR_BUCKET_NAME";
private static final String LOCAL_FILE_PATH="LOCAL_PATH_OF_FILE_TO_UPLOAD";
private static final String FILE_NAME_IN_BUCKET="FILE_NAME_TO_STORE_IN_BUCKET";
private static final String LOCAL_PATH_TO_STORE_FILE="LOCAL_PATH_TO_STORE_FILE"; //where to download in local
public static void main(String[] args) throws IOException {
GcsOIDC gcs= new GcsOIDC(TOKEN_ENDPOINT,API_ID,API_PASSWORD);
gcs.uploadFile(BUCKET_NAME,LOCAL_FILE_PATH,FILE_NAME_IN_BUCKET);
gcs.listFiles(BUCKET_NAME);
}
}
Java Class for bucket operations (GcsOIDC.java)
package org.example;
import com.google.api.gax.paging.Page;
import com.google.auth.oauth2.*;
import com.google.cloud.storage.*;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Base64;
import java.util.Collections;
public class GcsOIDC {
private final Storage storage;
//audience should not be changed
//For non prod environments, please check with CME Data Platform team for audience value
private static final String AUDIENCE =
"//iam.googleapis.com/projects/282603793014/locations/global/workloadIdentityPools/iamwip-cmegroup/providers/customer-federation";
public GcsOIDC(String tokenURL, String apiID, String apiPassword) throws IOException {
this.storage = createStorageClient(tokenURL, apiID, apiPassword);
}
public void uploadFile(
String bucketName, String filePath, String destinationBlobName)
throws IOException {
BlobId blobId = BlobId.of(bucketName, destinationBlobName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
storage.create(blobInfo, Files.readAllBytes(Paths.get(filePath)));
System.out.println(
"File " + filePath + " uploaded to bucket " + bucketName + " as " + destinationBlobName);
}
public void listFiles(String bucketName) {
Page<Blob> blobs = storage.list(bucketName);
System.out.println("Files in " + bucketName + ":");
for (Blob blob : blobs.iterateAll()) {
System.out.println(blob.getName());
}
}
public void downloadFile(String bucketName, String blobName, String destFilePath) {
Blob blob = storage.get(BlobId.of(bucketName, blobName));
blob.downloadTo(Paths.get(destFilePath));
System.out.println("Downloaded gs://" + bucketName + "/" + blobName + " to " + destFilePath);
}
private static Storage createStorageClient(String tokenURL, String apiID, String apiPassword) throws IOException {
IdentityPoolSubjectTokenSupplier subjectTokenSupplier = new CmeOidcTokenSupplier(tokenURL, apiID, apiPassword);
// The key fix: Use IdentityPoolCredentials for Workload Identity Federation.
// This builder has the .setSubjectTokenSupplier() method and is the correct
// one for this authentication flow.
IdentityPoolCredentials credentials =
IdentityPoolCredentials.newBuilder()
.setAudience(AUDIENCE)
.setSubjectTokenType("urn:ietf:params:oauth:token-type:jwt")
.setTokenUrl("https://sts.googleapis.com/v1/token")
.setScopes(Collections.singletonList("https://www.googleapis.com/auth/devstorage.read_write"))
.setSubjectTokenSupplier(subjectTokenSupplier)
.build();
return StorageOptions.newBuilder().setCredentials(credentials).build().getService();
}
private static class CmeOidcTokenSupplier implements IdentityPoolSubjectTokenSupplier {
// Reuse HttpClient and Gson across all token requests for better performance.
private static final HttpClient HTTP_CLIENT = HttpClient.newHttpClient();
private static final Gson GSON = new Gson();
private final String tokenUrl;
private final String clientId;
private final String clientSecret;
public CmeOidcTokenSupplier(String tokenUrl, String clientId, String clientSecret) {
this.tokenUrl = tokenUrl;
this.clientId = clientId;
this.clientSecret = clientSecret;
}
@Override
public String getSubjectToken(ExternalAccountSupplierContext externalAccountSupplierContext) throws IOException {
try {
// This assumes a standard OAuth 2.0 client credentials flow to get the token.
String requestBody = "grant_type=client_credentials";
String authHeader = "Basic " + Base64.getEncoder().encodeToString((clientId + ":" + clientSecret).getBytes(StandardCharsets.UTF_8));
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(tokenUrl))
.header("Content-Type", "application/x-www-form-urlencoded")
.header("Authorization", authHeader)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
HttpResponse<String> response = HTTP_CLIENT.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
throw new IOException("Failed to fetch subject token. Status: " + response.statusCode() + ", Body: " + response.body());
}
JsonObject jsonResponse = GSON.fromJson(response.body(), JsonObject.class);
// The OIDC token from a client credentials flow is often in the 'access_token' field and is a JWT.
// If your provider returns an ID Token, you might need to look for an 'id_token' field instead. Please verify with your IdP.
return jsonResponse.get("access_token").getAsString();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted while fetching subject token.", e);
}
}
}
}
A chosen filename must be applied uniformly throughout these steps.
3
Run “python generate_token.py >/tmp/token”
This operation generates a token and saves it to the /tmp/token file path, which can be modified to any desired location.
4
To create your Google credentials file, use the template below and replace UNO_TOKEN_FILE_PATH with the file name where you saved the token in the previous step. (Use "/tmp/token" if you did not change the default path.)
Save this file to a folder (e.g., "/opt/creds/credentials.json").
For credential_source, the file field specifies the token file path, such as /tmp/token. You do not need to make any changes if you are using the exact Python code provided in the previous step.
5
To log in to Google Cloud using a Google credentials file, use the gcloud command-line tool with the following command: