Cloud

As enterprises increasingly adopt model context protocol (MCP) to extend capabilities of AI models to better integrate with external tools, databases, and APIs, it becomes even more important to ensure secure MCP deployment. 

MCP unlocks new capabilities for AI systems; it can also introduce new risks, such as tool poisoning, prompt injection, and dynamic tool manipulation. These can lead to data exfiltration, identity subversion and misuse of AI systems. 

Securing an MCP deployment begins with a strong security foundation. Here are five key MCP deployment risks you should be aware of, and how using a centralized proxy architecture on Google Cloud can help mitigate them. 

Top five MCP deployment risks you should know

While there are some broader risks unique to AI, these five are especially important to be aware of when designing MCP deployments:

1. Unauthorized tool exposure: A misconfigured MCP manifest can create a vulnerability that allows unauthorized individuals or agents to access sensitive tools, such as internal administration functions.  

2. Session hijacking: An attacker may steal a legitimate user’s session ID to impersonate them. This allows the attacker to either make unauthorized API calls directly or, in stateful systems, inject malicious payloads into a shared data queue for delivery to the victim’s active session.

3. Tool shadowing and shadow MCP: Rogue MCP tools, mimicking legitimate services, can be deployed by malicious actors. This deceptive practice can trick both AI agents and human employees into interacting with harmful tools under the belief they are genuine.

4. Sensitive data exposure and token theft: Improperly configured environments or inadequate data-handling practices can lead to the accidental exposure of sensitive information, such as API keys, credentials, tokens, and personally identifiable information (PII). Attackers can exploit exposed credentials to gain unauthorized access to corporate resources, which could lead to significant data breaches.

5. Authentication bypass: Weak, misconfigured, or inadequately-enforced authentication mechanisms can be exploited by attackers to circumvent security controls, allowing them to gain unauthorized access by successfully impersonating legitimate users and trusted entities.

How a centralized MCP proxy architecture can help

To address these core risks in a scalable manner, we recommend a pattern built around a centralized MCP proxy that acts as a secure intermediary for all communication between clients and MCP servers.

Built on Cloud Run, Apigee, or Google Kubernetes Engine, this intelligent proxy intercepts all tool calls, enforcing your organization’s security policies and providing extensive monitoring. By serving as a centralized security enforcement point, the MCP proxy enables important risk-managing capabilities such as consistent access controls (acts as an authorization server), advanced traffic management, audit logging, secret scanning, resource limits, and real-time threat detection, all without requiring modifications to individual remote MCP servers.

Without centralization, organizations deploying MCP servers face challenges:

• Fragmented authentication: Each MCP server enforces its own policies, creating inconsistencies.

• Operational overhead: Updating agents with new endpoints is extensive and error-prone.

• Security blind spot: Limited visibility into health, usage, and anomalies across distributed MCP servers.

• Expended attack surface: Vulnerabilities such as prompt injection, tool poisoning, and unmonitored traffic flows.

A unified MCP proxy on Google Cloud addresses these challenges by:

• Enforcing organizational security policies at a single, managed entry point for all MCP requests and responses.

• Authenticating and authorizing consistently using the Google Identity Platform.

• Centralizing observability through Cloud Logging and Security Command Center.

• Standardizing protections against threats, aligned with Google secure AI agent guide and  OWASP guidance for agentic AI applications.

Architecture overview

The following reference architecture shows how Google Cloud services help secure MCP client requests end-to-end.

Key components:

• Global Load Balancing provides a single entry point for all MCP client traffic.

• Certificate Manager manages TLS certificates for secure communication.

• Cloud Armor provides DDoS and WAF protections against common OWASP web threats.

• MCP Proxy acts as the central enforcement layer for authentication, authorization, request inspection, and routing.

• Google Identity Platform validates identities and issues OAuth 2.0 tokens.

• Model Armor screens model prompts and responses for prompt injection and jailbreak detection, sensitive data, and can help ensure responsible AI practices.

• Secret Manager stores API keys, database credentials, and sensitive configuration values.

• Artifact Registry ensures MCP server images are scanned, verified and encrypted at rest with customer managed encryption keys.

• Internal Load Balancing routes sanitized traffic to the appropriate MCP servers.

• Cloud Logging and Security Command Center deliver visibility, anomaly detection, and compliance reporting.

Layered security controls

Google Cloud applies security at every layer of the architecture. Let’s dig into these capabilities below.

Network-level security control

• Network segmentation can isolate MCP tools servers, load balancers, and databases using virtual private networks (VPC), Cloud Run ingress, and Cloud Run egress options to reduce lateral movement risk. 

• Web attack and DDOS protection can help block malicious traffic, enforce IP allow and deny lists, and provide resilience against DDoS attacks with Global Load Balancer with Cloud Armor.

• Advanced traffic management can help provide custom domain path matching for MCP tool servers, using Internal Application Load Balancer (ALB) with advanced traffic management capabilities. By inspecting the incoming request host and path, the ALB intelligently directs requests to the appropriate backend service or instance of your MCP server. 

The backend service supports various back end types which can be MCP servers deployed in Google Cloud (such as VMs, Cloud Run, and Google Kubernetes Engine), in a remote network (using hybrid NEG) or on the internet (using internet NEG).

Authentication and authorization

The MCP proxy functions as the MCP Authorization Service (MAS) and integrates with an external Identity provider (such as Google Identity Platform, Okta, and Entra ID). This allows for secure access to your MCP server without the need to build and maintain a complex identity system. 

This highly scalable and reliable authentication solution can help users securely authenticate through their existing identity platform. Additionally, Cloud Identity and Access Management (IAM) can enable enforcement of role-based access, and restrict access to cloud resources.

Protection in-line MCP client prompt and response with Model Armor

Proxy leverages Model Armor to provide robust, real-time protection against runtime threats like prompt injection, jailbreaking, tool poisoning, dynamic tool manipulation, and sensitive data leakage. 

Using Model Armor templates, you can configure the MCP client prompts to be scrutinized and sanitized by Model Armor before being forwarded to the application load balancer. This process ensures that only sanitized prompts reach your MCP tools servers, thereby preventing unintended actions.

Logging and auditing

All actions performed by MCP clients will be logged by the proxy to Cloud Logging, such as accessing specific tools, modifying or writing to memory locations, initiating or receiving communication. The log will include a timestamp, agent ID, session ID, payload, and input/output signature fingerprints. 

The comprehensive nature of these logs, combined with the detailed metadata, can help administrators and auditors collect the necessary information to reconstruct client activities, identify anomalies, and enforce security policies effectively.

Additional security controls 

In addition to layered security controls, a robust defense-in-depth strategy requires continuous monitoring and proactive measures to identify and mitigate threats.

Secrets scanning with Sensitive Data Protection

Sensitive Data Protection (SDP) discovery periodically scans the MCP Tools server on Cloud Run, specifically targeting the detection of hardcoded secrets. This proactive measure involves regular scans of the server’s build and runtime environment variables. 

These scans detect insecure coding-practice risks, such as instances where sensitive information (including API keys, database credentials, and access tokens) has been inadvertently embedded directly in the code or configuration.

Vulnerability scanning

MCP server images stored in Artifact Registry are scanned for vulnerabilities before they are deployed. When a new image is pushed to Artifact Registry, it is automatically scanned for known vulnerabilities, providing actionable insights into potential weaknesses.

You can enforce policies that block the deployment of images with critical vulnerabilities as part of securing your environment from the ground up.

Threat detection

Security Command Center (SCC) provides AI protection that helps manage the security posture of your AI workloads by detecting threats and helping with  mitigating risks to AI asset inventories. This includes managing threats to the MCP deployment through detection, investigation, and response. 

SCC can also identify unauthorized access and data exfiltration attempts, delivering real-time alerts and remediation recommendations.

AI Protection helps you manage the security posture of your AI workloads by detecting threats and helping you to mitigate risks to your AI asset inventory.

Bring it all together

By strategically implementing these Google Cloud security services, you can establish a secure and resilient environment for your Model Context Protocol remote servers. Prioritizing authentication, network security, data protection, and continuous monitoring will ensure the integrity and confidentiality of your AI models and the sensitive information they process. 

This unified, Zero Trust approach can help mitigate emerging AI-specific risks, and also provide a scalable and future-proof foundation for the evolution of your AI-driven applications.

To learn more about securing your AI workload, please refer to our documentation.

MCP Toolbox for Databases (Toolbox) is an open-source MCP server that makes it easy for developers to connect gen AI agents to enterprise data, with initial support for databases like BigQuery, AlloyDB, Cloud SQL, and Spanner. Since launching earlier this year, Toolbox has made it easier for millions of developers to access enterprise data in databases.

Today, we’re expanding Toolbox with a comprehensive new set of tools for Firestore. This will help developers build more modern web and mobile applications. Let’s explore how these new capabilities can improve your development process.

What MCP is, and how it unlocks AI-assisted workflows

MCP is an emerging open standard for connecting AI systems with tools and data sources through a standardized protocol, replacing fragmented, custom integrations. Think of MCP as a universal adapter for AI, allowing any compatible assistant to plug into any tool or database without needing a custom-built connector each time. Now with the MCP Toolbox, these assistants (such as those in an IDE or a CLI like the Gemini CLI) can connect directly to your Firestore database.

This is a massive step for AI-assisted workflows, from debugging data and testing security rules to managing your collections—all using natural language. For instance, a developer building a retail app can now ask their assistant, ‘Find all users whose wishlists contain discontinued product IDs,’ to perform data cleanup, without writing a single line of code.

Let’s explore how these new capabilities can improve your development process.

AI-assisted development meets the NoSQL world

As you carry out AI-assisted tasks, you’re probably looking for the most efficient way to interact with your data. Our new pre-built tools for Firestore enable you to do just that, directly from your Gemini CLI or other AI-powered development environment.

Firestore’s flexible document structure and powerful security rules offer unique capabilities for building modern mobile and web applications. These tools are crafted to empower the Firestore developer, helping them master both the flexibility of the document model and the creation of robust access controls that protect their app. You can now use your AI assistant to perform queries, carry out targeted document updates, and even validate your security rules before you deploy them, saving you time and preventing errors.

From QA bug to resilient feature: A developer’s story

Let’s take a hypothetical example. Alex is a full-stack developer on a team building a new e-commerce application using Firestore. She uses  Gemini CLI to help her code, debug, and test. This morning, a high-priority bug was filed by the QA team: an issue in the staging environment is causing items to reappear in a user’s “wishlist” after being removed. Because of the blog, her release was blocked.

The bug hunt begins

Until now, investigating a bug meant Alex would have to manually click through the Cloud Console to inspect test documents or write a custom script just to query the staging database—a slow and cumbersome process. Now, she can simply have a conversation with Gemini CLI.

The bug report contains the test user accounts. Alex opens her terminal and asks:

“Hey, show me the Firestore data for the test users qa_user_123 and qa_user_456 from the users-staging collection.”

Gemini CLI understands this, calls the firestore-get-documents tool, and instantly displays the JSON for both user documents. Alex confirms the bug—the wishlist array contains stale data. She continues the conversation to understand the scope:

“Okay, that’s the bug. Find all users in the users-staging collection whose wishlist contains product-glasses(inactive).”

CLI uses the firestore-query-collection tool and reports back that 20 test accounts are affected. After developing a code fix, she needs to clean the test environment to verify it.

“For all 20 test users you just found, please remove product-glasses(inactive) from their wishlist.”

Gemini CLI confirms the plan and uses the firestore-update-document tool to perform the cleanup, clearing the way for a successful re-test.

From reactive fix to proactive hardening

The immediate bug is fixed, but Alex wants to ensure this class of error can never happen again. She decides to enforce the correct data structure with Firestore Security Rules.

Until now, validating security rules meant a disruptive context switch. Alex would have to copy her rules, navigate away from her terminal to the Firebase Console’s Rules Playground, or set up a local emulator just to check for syntax errors. This friction often discourages thorough, iterative testing within the main development loop.

Now, Alex drafts her new, stricter security rule. Before deploying, she asks Gemini CLI for a pre-flight check right from her terminal:

“new_rules.txt is a new Firestore Security Rule I’m working on for staging. Can you validate it for me?” 

CLI uses the firestore-validate-rules tool and replies, “The issue is a missing semicolon at the end of the return statement” Alex fixes the typo instantly. For a final check, she asks:

“Show me the active Firestore security rules for this project.”

Using the firestore-get-rules tool,  CLI displays the current ruleset, allowing Alex to do a final comparison. Confident and assured, she deploys her changes.

A task that could have taken hours of manual investigation, scripting, and context switching was completed in minutes. Alex didn’t just fix a bug; she used her AI assistant to make the entire application more resilient.

Getting started

These new Firestore tools within the MCP Toolbox represent our commitment to providing you with powerful, intuitive tools that accelerate the entire development lifecycle. By connecting your Firestore database directly to your AI assistant, you can spend less time on tedious tasks and more time building incredible applications.

Learn more about MCP Toolbox for Databases, connect it to your favorite AI-assisted coding platform, and experience the future of AI-accelerated, database-connected software development today.

• Docs: Connecting your AI Assistant to Toolbox
• Quick Start Guide: Using Firestore Tools w/ MCP
• Contribute to MCP Toolbox
• Join our Discord community

People often think of BigQuery in the context of data warehousing and analytics, but it is a crucial part of the AI ecosystem as well. And today, we’re excited to share significant performance improvements to BigQuery that make it even easier to extract insights from your data with generative AI.

 In addition to native model inference where computation takes place entirely in BigQuery, we offer several batch-oriented generative AI capabilities that combine distributed execution in BigQuery with distributed execution with remote LLM inference on Vertex AI, with functions such as: 

1. ML.GENERATE_TEXT to generate text via Gemini, other Google-hosted partner LLMs (e.g., Anthropic Claude, Llama) or any open-source LLMs
2. ML.GENERATE_EMBEDDING to generate text or multimodal embeddings
3. AI.GENERATE_TABLE to generate structured tabular data via LLMs and their constrained decoding capabilities. 

In addition to the above table-valued functions, you can use our row-wise functions such as AI.GENERATE for more convenient SQL query composition. All these functions are compatible with text data in managed tables and unstructured object files, such as images and documents. Thanks to the performance improvements we are unveiling today, users can expect dramatic gains in scalability, reliability, and usability across BigQuery and BigQuery ML:

1. Scalability: Over 100x gain for first-party LLM models (tens of millions of rows per six-hour job), over 30x gain for first-party embedding models (tens to hundreds of millions rows per six-hour job), and added support for Provisioned Throughput quotas. 

2. Reliability: Over 99.99% LLM inference query completion without any row failures; and over 99.99% row-level success rate across all jobs, with the rare per-row failures being easily retriable without failing the query. 

3. Usability: Enhanced user experience by supporting default connections, enabling global endpoints. We also enabled a single place for quota management (in Vertex AI) by automatically retrieving quota from Vertex AI to BigQuery. 

Let’s take a closer look at these improvements.

1. Scalability 

We’ve dramatically improved throughput for our users on pay-as-you-go (PayGo) pricing model: ML.GENERATE_TEXT function throughput on first-party LLMs has increased by over 100x, and ML.GENERATE_EMBEDDING throughput by over 30x. For users requiring even higher performance, we’ve also added support for Vertex AI’s Provisioned Throughput (PT).

Google embeddings LLMs
In BigQuery, each project has a fixed default quota for first-party text embedding models, including the most popular text-embedding-004/005 models. To enhance utilization and scalability, we introduced dynamic token-based batching to pack as many rows as possible into a single request, under the token constraint. Combined with other optimizations, this boosts scalability from 2.7 million to approximately 80 million rows (30x) per six-hour job (based on a default quota of 1500 QPM and 50 tokens per row dataset). You can further increase this capacity by raising your Vertex AI quota to 10,000 QPM without manual approval, which enables embedding over 500 million rows in a six-hour job.

Early access customers such as Faraday are excited for this scalability boost:

“I just did 12,678,259 embeddings in 45 min with BigQuery’s built-in Gemini. That’s about 5000 per second. Try doing that with an HTTP API!” – Seamus Abshere, Co-founder and CTO, Faraday

Google Gemini: PayGo users via dynamic shared quota
Gemini models served via Vertex AI now use dynamic shared quota (DSQ) for pay-as-you-go requests. DSQ provides access to a large, shared pool of resources, with throughput dynamically allocated based on real-time availability and demand across all customers. We rebuilt our remote inference system with adaptive traffic and producer-consumer-based error-retry mechanism. This enabled us to effectively leverage the higher amount of, but less-guaranteed, quota from DSQ. Based on internal benchmarking results, now we can process roughly 10.2 million rows in a six-hour job with gemini-2.0-flash-001, or 9.3 million rows with gemini-2.5-flash, where each row has an average of 500 input and 50 output tokens. Specific numbers depend on factors such as token count and model. See the chart below for more details.

Google Gemini: Dedicated quota via provisioned throughput
While our generative AI inference with Dynamic Shared Quota offers high throughput, it has an inherent upper bound and the potential for quota errors due to its non-guaranteed nature. To overcome these limitations, we’ve added support for Provisioned Throughput from Vertex AI. By purchasing dedicated capacity, you can help ensure consistently high throughput for demanding workloads, and get a reliable and predictable user experience. After purchasing Vertex AI provisioned throughput, you can easily leverage it in BigQuery gen AI queries by setting the “request_type” argument to “dedicated”. 

2. Reliability 

Facing limited and non-guaranteed generative AI inference quotas, we implemented a partial failure mode to allow queries to succeed, even if some rows fail. Via adaptive traffic control and a robust retry mechanism, across all users’ query traffic, we’ve now achieved 1) over 99.99% generative AI queries can finish without a single row failure, and 2) a row success rate of over 99.99%. This greatly enhanced BigQuery gen AI reliability. 

Note that the above row-level success rate is achieved based on independent queries that invoke our generative AI functions, including both table-valued functions and row-wise scalar functions, where the error retry takes place implicitly. If you have even large workloads that see occasional errors you can use this simple SQL script or the Dataform package, which allow you to iteratively retry failed rows to get to a 100% row success rate in almost all cases.

3. Usability 

We have also made Gen AI experience on BigQuery more user-friendly. We’ve streamlined complex workflows like quota management and BQ connection/permission setup, and enabled global endpoint more accessible experiences for users in all regions.

3.1 Default connection for remote model creation
Previously, setting up gen AI remote inference required a series of manual steps for users to configure a BigQuery connection and grant permissions to its service account. To solve this, we launched a new default connection. This feature automatically creates the connection and grants the necessary permissions, eliminating all manual config steps. 

3.2 Global endpoint for first-party models
BigQuery gen AI inference now supports the global endpoint from Vertex AI, in addition to dozens of regional endpoints. This global endpoint provides higher availability across the world, which is particularly useful for smaller regions that may not have immediate access to the latest first-party gen AI models. Users without a data residency requirement for ML processing can use the global endpoint.

The SQL below illustrates how to create a Gemini model with the global endpoint and the BigQuery default connection.

3.3 Automatic quota sync from Vertex AI
BigQuery now automatically syncs with Vertex AI to fetch your quota. Thus, if you have a higher than default quota, BigQuery uses it automatically, without any manual configuration on your part. 

Get started today

Ready to build your next AI application? Start generating text, embedding, or structured tables using gen AI directly in BigQuery today. Dive into our gen AI overview page and its linked documentation for more details. If you have any questions or feedback, reach out to us at bqml-feedback@google.com.

It’s hard to believe it’s been over 10 years since Kubernetes first set sail, fundamentally changing how we build, deploy, and manage applications. Google Cloud was at the forefront of the Kubernetes revolution with Google Kubernetes Engine (GKE), providing a robust, scalable, and cutting-edge platform for your containerized workloads. Since then, Kubernetes has emerged as the preferred platform for workloads such as AI/ML.  

Kubernetes is all about sharing machine resources among the applications and pod networking is essential for the connectivity between workloads and services using the Container Network Interface (CNI).

As we celebrate the 10th year anniversary of GKE, let’s take a look at how we’ve built out network interfaces to provide you with the performant, secure, and flexible pod networking and how we’ve evolved our networking model to support AI workloads with the Kubernetes Network Driver.

Let’s take a look back in time and see how we got there.

2015-2017: Laying the CNI foundation with kubenet

In Kubernetes’s early days, we needed to establish reliable communication between containers. For GKE, we adopted a flat model of IP addressing so that the pods and the node could communicate freely with other resources in the Virtual Private Cloud (VPC) without going through tunnels and gateways. During these formative years, GKE’s early networking models often used kubenet, a basic network plugin. Kubenet provided a straightforward way to get clusters up and running, by creating a bridge on each node and allocating IP addresses to pods from a CIDR range dedicated to that node. While Google Cloud’s network handled routing between nodes, Kubenet was responsible for connecting pods to the node’s network and enabling basic pod-to-pod communication within the node.

During this time, we also introduced route-based clusters, which were based on Google Cloud routes, part of the Andromeda engine that powers all of cloud networking. The routes feature in Andromeda played a crucial role in IP address allocation and routing within the cluster network using VPC routing rules. This required advertising the pod IP ranges between the nodes. 

However, as Kubernetes adoption exploded and applications grew in scale and complexity, we faced challenges around managing IP addresses and achieving high-performance communication directly between pods across different parts of a VPC. This pushed us to develop a networking solution that was more deeply integrated with the underlying cloud network.

2018-2019: Embracing VPC-native networking

To address these evolving needs and integrate with Google Cloud’s powerful networking capabilities, we introduced VPC-native networking for GKE. This marked a significant leap forward for how CNI operates in GKE, with alias IP ranges (the IP ranges that pods use in a node) becoming a cornerstone of the solution. VPC-native networking became the default and recommended approach, helping to increase the scale of the GKE clusters up to 15K nodes. With VPC-native clusters, the GKE CNI plugin ensures that pods receive IP addresses directly from your VPC network — they become first-class citizens on your network.

This shift brought a multitude of benefits:

• Simplified IP management: GKE CNI plugin works with GKE to allocate pod IPs directly from the VPC, making them directly routable and easier to manage alongside your other cloud resources.

• Enhanced security through VPC integration: Because pod IPs are VPC-native, you can apply VPC firewall rules directly to pods. 

• Improved performance and scalability: GKE CNI plugin facilitates direct routing within the VPC, reducing overhead and improving network throughput for pod traffic.

• A foundation for advanced CNI features: VPC-native networking laid the groundwork for more sophisticated CNI functionalities that followed.

We referred to GKE’s implementation of CNI’s with VPC-native networking as GKE standard networking with dataplane v1 (DPv1). During this time, we also announced GA support for network policies with Calico. Network policies allow you to specify rules for traffic flow within your cluster, and also between pods and the outside world.

2020 and beyond: The eBPF revolution

The next major evolution in GKE’s CNI strategy arrived with the power of extended Berkeley Packet Filter or eBPF, which lets you run sandboxed programs in a privileged context. eBPF makes it safe to program the Linux kernel dynamically, opening up new possibilities for networking, security, and observability at the CNI level without having to recompile the kernel.

Recognizing this potential, Google Cloud embraced Cilium, a leading open-source CNI project built on eBPF, to create GKE Dataplane V2 (DPv2). Reaching general availability in May 2021, GKE Dataplane V2 represented a significant leap in GKE’s CNI capabilities: 

• Enhanced performance and scalability: By leveraging eBPF, CNI can bypass traditional kernel networking paths (like kube-proxy’s iptables-based service routing) for highly efficient packet processing for services and network policy.

• Built-in network policy enforcement: GKE Dataplane V2 comes with Kubernetes network policy enforcement out-of-the-box, meaning you don’t need to install or manage a separate CNI like Calico solely for policy enforcement when using DPv2.

• Enhanced observability at the data plane layer: eBPF enables deep insights into network flows directly from the kernel. GKE Dataplane V2 provides the foundation for features like network policy logging, offering visibility into CNI-level decisions.

• Integrated security in the dataplane: eBPF enforces network policies efficiently and with context-awareness directly within CNI’s dataplane.

• Simplified operations: As it’s a Google-managed CNI component, GKE Dataplane V2 simplifies operations for Customer workloads.

• Advanced networking capabilities: Dataplane V2 unlocks a suite of powerful features that were not available or harder to achieve with Data Plane V1. These include:

• IPv6 and dual-stack support: Enabling pods and services to operate with both IPv4 and IPv6 addresses.

• Multi-networking: Allowing pods to have multiple network interfaces, connecting to different VPC networks or specialized network attachments, crucial for use cases like cloud native network functions (CNFs) and traffic isolation.

• Service steering: Providing fine-grained control over traffic flow by directing specific traffic through a chain of service functions (like virtual firewalls or inspection points) within the cluster.

• Persistent IP addresses for pods: In conjunction with the Gateway API, GKE Dataplane V2 allows pods to retain the same IP address across restarts or rescheduling, which is vital for certain stateful applications or network functions.

GKE Dataplane V2 is now the default CNI for new clusters in GKE Autopilot mode and our recommended choice for GKE Standard clusters, underscoring our commitment to providing a cutting-edge, eBPF-powered network interface.

2024: Scaling new heights for AI Training and Inference

In 2024, we marked a monumental achievement in GKE’s scalability, with the announcement that GKE supports clusters of up to 65,000 nodes. This incredible feat, a significant jump from previous limits, was made possible in large part by GKE Dataplane V2’s robust, highly optimized architecture. Powering such massive clusters, especially for demanding AI/ML training and inference workloads, requires a dataplane that is not only performant but also incredibly efficient at scale. The version of GKE Dataplane V2 underpinning these 65,000-node clusters is specifically enhanced for extreme scale and the unique performance characteristics of large-scale AI/ML applications — a testament to CNI’s ability to push the boundaries of what’s possible in cloud-native computing.

For AI/ML workloads, GKE Dataplane v2 also supports ever-increasing bandwidth requirements such as in our recently announced A4 instance. GKE Dataplane v2 also supports a variety of compute and AI/ML accelerators such the latest GB200 GPUs and Ironwood, Trillium TPUs.

For today’s AI/ML workloads networking plays critical role: AI and machine learning workloads are pushing the boundaries of computing as well as networking, presenting unique challenges for GKE networking interfaces:

• Extreme throughput: Training large models requires processing massive datasets that demand upwards of terabit networking orchestrated by GKE networking interfaces.

• Ultra-low latency: Distributed training relies on near-instantaneous communication between processing units.

• Multi-NIC capabilities: Providing pods with multiple network interfaces, managed by GKE Dataplane V2’s multi-networking capability, can significantly boost bandwidth and allow for network segmentation.

2025 – Beyond CNI: addressing next gen Pod Networking challenges

Dynamic resource allocation (DRA) for networking 

A promising Kubernetes innovation is dynamic resource allocation (DRA). Introduced to provide a more flexible and extensible way for workloads to request and consume resources beyond CPU and memory, DRA is poised to significantly impact how CNIs manage and expose network resources. While initially focused on resources like GPUs, its framework is designed for broader applicability.

In GKE, DRA (available in preview from GKE version 1.32.1-gke.1489001+) opens up possibilities for more efficient and tailored network resource management, helping demanding applications get the network performance they need using the Kubernetes Network Drivers (KNDs).

KNDs use DRA to expose Network resources at the Node level that can be referenced by all the Pod (or all containers). This is particularly relevant for AI/ML workloads, which often require very specific networking capabilities. 

Looking ahead: Innovations shaping the future

The journey doesn’t stop here. With the increased adoption of accelerated workloads driving new architectures on GKE, the demands on Kubernetes networking will continue. One of the reference implementations for the Kubernetes Network Driver is the DRANET project. We look forward to continued discussions with the community and contributions to the DRANET project. We are committed to working with the community to deliver innovative customer centric solutions addressing these new challenges.

Trading in capital markets demands peak compute performance, with every microsecond impacting critical decisions and market outcomes. At Google Cloud, we’re committed to providing global markets with the cutting-edge infrastructure they need to create and participate in digital exchange ecosystems. Our industry investments enable a purpose-built, cloud-native market infrastructure solution leveraging a global network that was built for security and scale, data, and AI capabilities. At the same time, we’re building industry-specific innovations that offer performant, scalable, and resilient environments for exchanges and trading participants, ultimately transforming how they access markets, utilize data, and manage risk. 

Following our investments in optimized infrastructure for digital exchanges, we’ve continued to push the boundaries of what’s possible in cloud-based trading.

The general availability of our C3 and C4 machine series, powered by the latest Intel Xeon Scalable processors and our custom Titanium offload network investments, represents a significant leap forward for latency-sensitive trading applications. In addition, Citadel Securities recently joined us as part of our Cloud WAN layer 2 solution that enables point-to-point connections over Google’s proprietary global network and complements our existing Network Connectivity Center layer 3 solution. Together, these offerings help trading participants achieve low-latency compute and connectivity for their globally distributed trading infrastructure. 

Building on these foundational investments, we’re excited to announce new benchmarks specifically tailored for trading participants who require minimal latency and jitter with maximal throughput to handle the increasing market velocity of their trading infrastructures. In collaboration with 28Stone, a consultancy with expertise in capital markets technology and electronic trading solutions, we’ve tested and validated the performance of our C3 machine types to meet the needs of trading participants.  

28Stone’s published report highlights that Google Cloud can achieve a round-trip trading decision in less than 50 microseconds at P99 across a range of compute profiles.

“As a 24×7 digital exchange focused on delivering institutional exchange level consistency in the cloud, we are excited to see that Bullish’s participants can immediately leverage what 28Stone and Google Cloud have publicly demonstrated: the technical capabilities to rapidly spot opportunity and engage with confidence.” – Alan Fraser, VP of Platform & Operations, Bullish

The role of latency, jitter, and throughput

In today’s hyper-competitive electronic trading landscape, the performance of underlying technology infrastructure is not just a contributing factor to success — it’s fundamental. Three key performance metrics stand out for their impact on trading outcomes: latency, jitter, and throughput. 

Latency: the race 

In the context of trading, latency refers to the time it takes to receive, understand, and decide an action from a single datum. For trading systems, this means the time it takes for market data to reach the trading algorithm, for that algorithm to make a decision, and finally send a response back to the exchange. In a world of high-frequency trading (HFT) and algorithmic execution, single-microsecond delays can mean the difference between a profitable trade and a missed opportunity, a less favorable execution price, or getting filled on a resting order.

28Stone demonstrated latency for participants to make a straightforward trade decision — from receiving the ticks to processing a trade decision — of between 1.5µs and 3.5µs (P50 and P99, respectively) for normal replay speed of CME Group Equity market pcap files using open source Data Plane Development Kit (DPDK) network acceleration. Additionally, they demonstrated similar latency profiles with data rates increasing by up to 100x.

Jitter: the quest for consistency 

Jitter refers to the variation in latency over time. While low latency is critical, high jitter — meaning unpredictable and inconsistent delays — can negatively impact trading performance. If the time it takes for an order to reach the exchange varies significantly, it becomes incredibly difficult to predict execution outcomes, manage risk, or implement trading strategies that rely on time certainty. If you were to use UberEats to order lunch but were told it would be delivered between 12 and 5pm, you would be unlikely to choose that option.  

28Stone demonstrated that participants can expect their experience in Google Cloud to be uniform, regardless of volatile market conditions. Google Cloud infrastructure delivers low jitter, allowing trading algorithms to operate with a higher degree of certainty, leading to more stable and reliable market mechanics – as well as profitability from good trading signals. 

As illustrated below for various C3 instance sizes using C++ with or without DPDK, the percentiles demonstrate strong consistent message performance. The increase in latency between each percentile demonstrates network and compute consistency measured in single and low tens of microseconds. The 28Stone report contains complete histograms for various configurations, allowing customers to see how to balance their specific latency and jitter requirements against the configurations’ cost profiles.

Throughput: handling the pressure of information

Throughput measures the amount of data that can be processed or transmitted within a given window of time (typically one second). In trading, throughput is a system’s capacity to handle large volumes of market data updates, process numerous events simultaneously, and aggregate trade events efficiently — especially during periods of high market volatility or peak trading hours. Insufficient throughput can lead to data queues, order rejections, increased risk exposure, and an inability to keep pace with market activity.

The C3 machine series leverages Google Cloud’s high-bandwidth networking capabilities, which include up to 200 Gbps per VM Tier 1 networking and services such as Cloud WAN. These machines are designed to provide the high throughput traders need to ingest vast streams of market data and execute on a large number of orders per second. The result is a trading system that performs optimally under strenuous load.

As highlighted, 28Stone tested increasingly higher replay speeds, resulting in higher bit rates that various trading systems may see in a variety of markets.

In summary, minimizing latency, reducing jitter, and maximizing throughput aren’t abstract technical pursuits but about consistency and certainty, akin to your lunch order arriving near your lunch break and not in the evening. Modern capital markets trading participants and digital exchanges demand these capabilities to enable market quality, fairness, and operational resilience. 

Embracing new market dynamics with cloud

Financial markets are in a perpetual state of flux, characterized by evolving regulations, a proliferation of new asset classes (including those that trade 24/7 like FX and digital assets), and sudden, sharp spikes in trading volumes. In this dynamic environment, the ability to scale infrastructure rapidly, operate with flexibility, and manage costs effectively is not just an advantage — it’s a prerequisite for survival and growth. 

With Google Cloud, exchanges and participants enjoy several benefits:  

• Elastic scalability: Automatically scale resources up or down based on real-time demand. This means that during unexpected volume spikes — driven by market news, geopolitical events, or algorithmic trading activity — trading infrastructure can dynamically access additional compute power to maintain optimal performance. When volumes normalize, resources can be scaled back down, so that firms only pay for what they use. 

• Flexibility for continuous trading: Cloud infrastructure provides the resilience and availability that continuous 24/7 trading requires, as demonstrated daily by critical workloads for global 24/7 industry platforms like air travel, retail banking, media, and retail. Google Cloud’s global network and multiple regions help ensure high uptime and fault tolerance, both critical for markets that never sleep. As seen with Deutsche Börse’s development of a digital asset trading platform on Google Cloud, the architecture is designed for 24/7 availability and can be rolled out quickly to new markets. Google Cloud’s continuous operations allow firms to capitalize on opportunities around the clock without the massive investment and operational overhead of maintaining private data centers with equivalent N+1 redundancy globally.

• Rapid engagement in new markets: Historically, expanding into new geographical markets or launching new asset classes involved lengthy and costly infrastructure build-outs. With Google Cloud’s global regions, firms can deploy trading infrastructure in new regions in a fraction of the time. This agility allows for rapid market entry, thereby enabling businesses to seize new revenue streams and diversify their business with lower upfront investment costs and risk. Quickly provisioning and de-provisioning resources also means firms can experiment with new market strategies more freely, knowing they are not locked into long-term hardware commitments.

Want to see it for yourself?

By harnessing the scale, flexibility, and compelling cost-to-performance ratio of Google Cloud, including the powerful C3 and C4 family instances, trading participants can transform market volatility from a threat into an opportunity. They can confidently handle volume surges, support round-the-clock trading, and swiftly enter new markets, all while maintaining tight control over their operational expenditure and maximizing their competitive edge.

Want to apply these capabilities to the announced CME Group market migration to Google Cloud? Register with Google Cloud for notifications and engagements.