Tibor Kiss

We’re excited to announce the general availability of our next-generation C4D virtual machine family. Powered by 5th Gen AMD EPYC processors (Turin) paired with Google Titanium’s latest advancements, C4D provides customers with meaningful performance improvements — up to 80% higher throughput for web serving and 30% better performance for general computing workloads compared to the previous generation. This improvement in performance enables you to maximize your cloud investment and achieve more with fewer resources. 

Beyond raw performance, C4D supports key enterprise capabilities including our first AMD-based Bare Metal instances offering direct access to all the resources on the server for maximum control and performance (bare metal will be available in the coming weeks); and the next-gen Titanium Local SSD, which enhances I/O-intensive operations with up to 35% lower latency vs. the prior generation. These hardware advancements are paired with enterprise-grade security and reliability, featuring a 30-day uptime window between planned maintenance events. With this combination of peak performance, expanded capabilities and enterprise-grade controls, C4D is suited for a wide range of general-purpose computing workloads, from databases, AI inference, web, application and game servers, to mission-critical business applications.

Supercharge your workloads while optimizing costs

General computing and web serving workloads

For general-purpose computing workloads, C4D VMs deliver up to 25% better performance per dollar than C3D, based on estimated SPECrate®2017_int_base benchmark results. C4D also delivers up to 20% better performance per dollar than comparable, generally available offerings from other cloud providers on the same benchmark. This helps you meet demanding performance requirements for a wide range of workloads — including web servers, game, ad and application servers, and containerized microservices — while optimizing resource usage and reducing costs. 

For web-serving workloads, C4D leverages AMD Turin’s improved L3 cache efficiency and next-generation branch prediction to deliver higher throughput/vcpu, resulting in up to 75% increase in price-performance compared to the previous-generation C3D. This drives faster page rendering and a smoother end-user experience, with up to 45% higher price-performance compared to competitive offerings. 

“AppLovin, a global leader in mobile advertising, is constantly looking for cutting-edge infrastructure innovations to deliver exceptional performance for our clients. Google Cloud’s C4D VMs enable us to do just that — driving up to 40% improvement over the prior generation, which leads to significant efficiency gains and latency reduction.” – Basil Shikin, Chief Technology Officer, AppLovin

“On C4D, our ad servers perform 191% faster than N2D, and 81% faster than C3D to serve the same amount of ad-requests. This improved performance comes at a lower overall cost because we can run a smaller number of more efficient nodes, but not only that, for us more performance/less latency means not only savings but more revenue since the fill-rate for ads (successful bid/ask matching) grows exponentially.” – Pablo Loschi, Principal Systems Engineer, Verve Group

“C4D Performance was impressive — most workloads, including page rendering & video processing, were 40+% faster than the previous generation. This kind of improvement makes a real difference to users of platforms like SpareRoom.” – Dimitrios Kechagias, Principal Developer, Cloud Infrastructure Optimisation Lead, SpareRoom

Databases and data-intensive applications

The C4D family is purpose-built for data-intensive applications such as databases and data analytics, by offering the latest generation compute and advanced storage capabilities. C4D’s high core frequency of up to 4.1Ghz and improved Instructions Per Clock (IPC) accelerate transactional workloads such as MySQL by up to 55% versus the prior generation with faster, more efficient query processing. For applications that require large datasets in memory, C4D provides VM sizes scaling up to 384vcpu and 3TB of high-bandwidth DDR5 memory. To scale database I/O performance, customers benefit from the integrated Hyperdisk Extreme storage with up to 500k IOPS and the new Titanium Local SSD that reduces read latency by 35% compared to the prior generation. Together, these capabilities increase performance and responsiveness for your mission-critical databases, delivering up to 35% better price-performance for Redis and MySQL workloads than comparable generally available VMs from other hyperscalers.

“We are constantly looking for advanced computing options to improve experience for the players. With Google’s new C4D VMs, we see drastically improved performance for our observability stack which handles 50k inserts/sec concurrently. Compared to C3D, we were able to cut our resource footprint by half, while reducing CPU load by 35% and seeing a 30% improvement in indexing latency. We look forward to adopting C4D at scale.” – Grzegorz Dlugolecki, Principal Cloud & Kubernetes Engineer, Chess.com

“Across over 100 benchmarks, going from the C3D to C4D yielded 1.7x the performance! This is a heck of a generational improvement for Google Cloud or any public cloud provider for that matter. C4D performance is extremely compelling and opens up a lot of new compute possibilities in the public cloud.” – Michael Larabel, Founder and Principal Author, Phoronix (Read the full study here)

AI inference and complex computations
C4D’s processor offers full support for AVX-512 with a 512 bit datapath, a 50% increase in memory channels, and higher IPC compared to the prior generation. This provides significant improvements for compute-heavy tasks such as CPU-based inference, matrix operations, financial modeling and simulations, and analytics. For recommendation inference, C4D VMs demonstrate an up to 75% price-performance uplift compared to C3D and up to 35% better price-performance versus the comparable competitive offerings from leading hyperscalers, to accelerate the time to results and reduce TCO.

“Silk has tested C4D, and found it to deliver a dramatic increase of up to 40% in performance compared to the previous generation, C3D, enabling our customers to enjoy significant gains in efficiency and agility of their mission critical workloads, over the transactional, analytics and AI use cases.” – Adik Sokolovski, Chief R&D Officer, Silk

Security, maintenance controls and shapes   

With Titanium, C4D offers improved infrastructure performance, lifecycle management, reliability, and security. Storage and network management is offloaded to the Titanium adapter, reserving the host resources for running customer workloads. 

Titanium also enables our first AMD-based Bare Metal instances, which provide direct access to server resources. Bare metal instances are ideal for workloads that require low-level system access — like custom hypervisors, container platforms, or applications with specialized performance or licensing needs. Sectors such as financial services, security, and private cloud platforms will particularly benefit from C4D Bare Metal offerings.

C4D VMs support Hyperdisk, Google Cloud’s workload-optimized block storage. Designed for high performance and scalability, Hyperdisk is cost-efficient, easy to manage at scale, and enterprise-ready. C4D VMs are compatible with Hyperdisk Balanced and Extreme, supporting up to 512 TiB of capacity per instance. With up to 320K IOPS per instance, Hyperdisk Balanced offers an optimal mix of performance and cost-efficiency, for a broad range of workloads. Hyperdisk Extreme delivers ultra-low latency and supports up to 500K IOPS and 10,000 MiB/s throughput per instance — making it well-suited for demanding workloads like databases and caching layers. With real-time tuning of IOPS and bandwidth, Hyperdisk helps ensure your applications always have the storage resources they need.

To enhance security, C4D VMs support confidential computing with AMD Secure Encrypted Virtualization (AMD SEV), utilizing hardware-based memory encryption to help protect your data and applications while in use. This makes C4D an excellent choice for sensitive data, privileged information, PII, and workloads subject to data privacy regulations and compliance requirements.

Experience C4D today

C4D delivers exceptional performance, scalability, and efficiency for today’s most demanding workloads. Powered by next-gen AMD processors, Titanium infrastructure, and Hyperdisk storage, C4D delivers the performance and capabilities needed to make the most of your cloud resources. Whether you’re just getting started with Compute Engine or planning to upgrade from previous generations, C4D offers a clear path to greater efficiency and performance. C4D is now available in 12 regions and 28 zones — check regional availability on our Regions and Zones page and deploy your first instance in the Google Cloud console or with Google Kubernetes Engine.

As cloud environments expand beyond traditional architectures to include multiple clouds, managing your infrastructure effectively becomes more complex. Imagine easily accessing consistent and up-to-date location information across different cloud providers, so your multi-cloud applications are designed and optimized with performance, security, and regulatory compliance in mind.

Today, we are making this a reality with Cloud Location Finder, a new Google Cloud service which provides up-to-date location data across Google Cloud, Amazon Web Services (AWS), Microsoft Azure, and Oracle Cloud Infrastructure (OCI). Now, you can strategically deploy workloads across different cloud providers with confidence and control.

Why use Cloud Location Finder?

• Unified location data: Cloud Location Finder makes it easy for you to access the latest location information, providing a single source of truth for cloud location data for Google Cloud, AWS, Azure, and OCI.   

• Rich location attributes: Data includes public cloud region and zone information, and location metadata like proximity¹, territory code, and carbon footprint. 

• Up-to-date information: Cloud Location Finder provides 24-hour data freshness for active regions, removing outdated information promptly when locations are turned down. This eliminates inconsistencies from hard-coded lists and ensures your information is current without manual monitoring.   

• Programmatically accessible: Especially for organizations with diverse cloud and location requirements, Cloud Location Finder eliminates the need to manually manage and choose new locations. And because it is an API, it is easy to integrate into your applications.

How can Cloud Location Finder help you?

Whether you’re a partner or customer, a Cloud Architect or application developer designing a hybrid setup or a platform admin ensuring governance, Cloud Location Finder offers valuable insights:

• Optimize deployments: Easily identify the nearest Google Cloud region or zone to an existing AWS/Azure/OCI deployment, to help you optimize your multi-cloud application for performance and latency.

• Meet sustainability goals: As your business grows, choose nearby cloud locations that are also sustainable.

• Help ensure compliance: Find a list of regions and zones in a specified territory to help you ensure compliant log storage or data processing across multiple clouds.

• Improve reliability: Rely on a consistent source of truth for location data, which can be integrated directly into your applications.   

Getting started

Cloud Location Finder is accessible via REST APIs and gcloud CLI, and available at no cost. You can easily list locations, get specific location details, find nearby locations, and filter these based on criteria such as cloud provider, location type, territory, or carbon footprint.   

Ready to streamline your multi-cloud location strategy? Explore the Cloud Location Finder documentation to learn more and start building with consistent, accurate location data today!

^{1. Currently, for GCP regions only}

Know Your Customer (KYC) processes are foundational to any Financial Services Institution’s (FSI) regulatory compliance practices and risk mitigation strategies. KYC is how financial institutions verify the identity of their customers and assess associated risks. But as customers expect instant approvals, FSIs face pressure to streamline their manual, time-consuming and error-prone KYC processes. 

The good news: As LLMs get more capable and gain access to more tools to perform useful actions, employing a robust ‘agentic’ architecture to bolster the KYC process is just what FSIs need. The challenge? Building robust AI agents is complex. Google’s Agent Development Kit (ADK) gives you essential tooling to build multi-agent workflows. Plus, combining ADK with Search Grounding via Gemini can help give you higher fidelity and trustworthiness for tasks requiring external knowledge. Together, this can give FSIs:

1. Improved efficiency: Automate large portions of the KYC workflow, reducing manual effort and turnaround times.

2. Enhanced accuracy: Leverage AI for consistent document analysis and comprehensive external checks.

3. Strengthened compliance: Improve auditability through clear reporting and source attribution (via grounding).

To that end, this post illustrates how Google Cloud’s cutting-edge AI technologies – the Agent Development Kit (ADK), Vertex AI Gemini models, Search Grounding, and BigQuery – can be leveraged to build such a multi-agent KYC solution.

Tech stack from Google Cloud

This multi-agent architecture we’ll show you today effectively utilizes several key Google Cloud services:

• Agent Development Kit (ADK): Simplifies the creation and orchestration of agents. ADK handles agent definition, tool integration, state management, and inter-agent communication. It’s a platform and model-agnostic agentic framework which provides the scaffolding upon which complex agentic workflows can be built.

• Vertex AI & Gemini models: The agents are powered by Gemini models (like gemini-2.0-flash) hosted on Vertex AI. These models provide the core reasoning, instruction-following, and language understanding capabilities. Gemini’s potential for multimodal analysis (processing images in IDs or documents) and multilingual support further enhances the KYC process for diverse customer bases.

• Search Grounding: The google_search tool, used by the Resume_Crosschecker and External_Search agents, leverages Gemini’s Google Search grounding capabilities. This connects the Gemini model’s responses to real-time information from Google Search, significantly reducing hallucinations and ensuring that external checks are based on up-to-date, verifiable public data. The agents are instructed to cite sources (URIs) provided by the grounding mechanism, enhancing transparency and auditability.

• BigQuery: The search_internal_database custom tool demonstrates direct integration with BigQuery. The KYC_Agent uses this tool early in the workflow to check if a customer profile already exists within the institution’s internal data warehouse, preventing duplicate entries and leveraging existing information. This showcases how agents can securely interact with internal, structured datasets.

Deep dive: How to build a KYC agent in three steps 

Our example KYC solution utilizes a root agent (KYC Agent) that orchestrates several specialized sub-agents:

• Document Checker: Analyzes uploaded documents (ID, proof of address, bank statements, etc.) for consistency, validity, and potential discrepancies across documents.

• Resume Crosschecker: Verifies information on a customer’s resume against public sources like LinkedIn and company websites using grounded web searches.

• External Search: Conducts external due diligence, searching for adverse media, Politically Exposed Person (PEP) status, and sanctions list appearances using grounded web searches.

• Wealth Calculator: Assesses the client’s financial position by analyzing financial documents, calculating net worth, and verifying the source of wealth legitimacy.

The root KYC_Agent manages the overall workflow, calling these child agents sequentially and handling tasks like checking if the customer is already present in internal databases and generating unique case IDs to track KYC requests.

Step 1: Define your root agent (which receives the initial request from the user) and the child agents which handle the specialised tasks involved in the KYC process.

Step 2: Define the tools needed by your agents in order to perform their respective tasks

Step 3: Run your agent locally using the command “adk web”. ADK provides a built-in UI for developers to visualise and debug the agent during the development process:

Start building now 

This multi-agent KYC architecture demonstrates the power of combining ADK, Gemini, Search Grounding, and BigQuery. It provides a blueprint for building intelligent, automated solutions for complex business processes.

• Learn more: Dive deeper into the technologies used:

• Google Agent Development Kit (ADK)

• Vertex AI Gemini Models

• Grounding on Vertex AI

• BigQuery

By embracing a multi-agent approach powered by Google Cloud’s AI stack, FSIs can transform their KYC processes, achieving greater efficiency, accuracy, and compliance in an increasingly digital world.

Credentials are an essential part of modern software development and deployment, granting bearers privileged access to systems, applications, and data. However, credential-related vulnerabilities remain the predominant entry point exploited by threat actors in the cloud.

Stolen credentials “are now the second-highest initial infection vector, making up 16% of our investigations,” said Jurgen Kutscher, vice-president, Mandiant Consulting, in his summary of our M-Trends 2025 report. 

Ensuring the safe management of these credentials is a vital task. Developers may accidentally include credentials in artifacts like source code, built software packages, or Docker images. If these credentials fall into the wrong hands, they can be used by malicious actors for data exfiltration, cryptojacking, ransomware attacks, and general resource abuse. 

Safeguarding credentials is particularly acute for open-source developers because when a credential is accidentally included in an artifact that is pushed to a public repository (like GitHub, PyPI or DockerHub), that credential becomes available to anyone on the Internet. 

To address this critical issue, we’ve developed a powerful tool to scan open-source package and image files by default for leaked Google Cloud credentials to help protect Google Cloud customers who publish open-source artifacts. Created by Google’s deps.dev team in collaboration with Google Cloud’s credential protection team, we’ve seen significant results in identifying and reporting exposed credentials like API keys, service account keys, and OAuth client secrets in historical artifacts. 

While this effort has initially focused on Google Cloud credentials, we plan to expand scanning to include third-party credentials later this year.

Beyond retrospective reporting, the tool also scans newly published open-source artifacts for leaked credentials. This pivotal advance can help drive remediation for immediate security breach threats, significantly reducing the risk of developer compromise. 

The tool can also cultivate a culture of improved security by effectively shifting security to earlier in the development lifecycle when problems are easier to solve. By shifting left and encouraging earlier security awareness, the tool can help foster improved credential management practices in the open-source community, ultimately strengthening the resilience and security of the entire software supply chain.

Understanding the dangers of exposed cloud credentials

Exposed credentials present a serious security risk to cloud users because they allow an individual to gain access to a user’s cloud environment, including their resources, applications and managed user data. A malicious actor can exploit this access for nefarious purposes such as data theft, cryptojacking, ransomware attacks, and general resource abuse which can result in severe financial, reputational, and operational damage. 

Once a credential is obtained by malicious actors it should be considered permanently compromised because compromised credentials are easily copied and shared.

Open source developers, while contributing to the collaborative ecosystem, face the risk of inadvertently exposing sensitive credentials. While source code repository hosts like GitHub and GitLab already scan public source code (and, in some cases, package repositories) for exposed credentials, the challenge extends significantly beyond source code. 

Built packages and Docker images often include configuration, compiled binaries, and build scripts, all potential sources of leaked credentials. Publishing these artifacts on open-source repositories like Maven Central, PyPI, or DockerHub can expose leaked credentials to exploitation by any individual on the internet. The ease and speed with which open-source artifacts are shared and distributed magnifies the potential damage, making strong credential management and proactive leak detection and remediation critical.

How to scan open source code for credentials

The deps.dev team provides services to help developers better understand the structure, construction, and security of open-source software. The team maintains and analyzes a continuously updated corpus of over 5 billion unique files, across hundreds of millions of open-source software artifacts like source code repositories, software packages and Docker containers. 

The pipeline to support this corpus automatically ingests hundreds of millions of public artifacts from a variety of open source repositories. These include package managers (such as npm, Maven Central, PyPI,) source code repository hosts (such as GitHub and GitLab) and Docker images.

Once artifacts are ingested, they undergo a comprehensive decomposition process, which extracts all constituent parts: every file at every commit in a Git repository, every unarchived or unzipped file in a software package, and every file in every individual layer of a Docker image — not just the files in the final image filesystem. These files are then analyzed which includes scanning them for exposed Google Cloud credentials. 

When a suspected Google Cloud credential is detected, the credential reporting backend immediately alerts the credential protection program. Since its creation, we’ve observed this system detect and remediate leaked credentials in minutes of their publication, matching or exceeding the speed with which malicious actors have been demonstrated to exploit them.

Credential containment and recovery

We’ve set up a web endpoint so vetted Google Cloud users and security researchers can submit suspected exposed credentials for review.Once a submitter’s identity is validated, the Google Cloud credential protection system proceeds to confirm the validity of the reported credentials. If the credential is confirmed to be active, Google Cloud provides immediate customer notification through multiple channels, including email, telemetry logs, and in-product alerts. 

Google Cloud may take automated remediation steps to mitigate potential damage in accordance with customer configurable policy, such as disabling affected service account keys.

What’s next?

We are actively working to further secure open source communities and protect Google Cloud customers alike by taking a proactive approach to credential exposure. Our efforts in this area include several key initiatives:

• Broadening the scope of credential scanning: We’re expanding the range of credential types the tool can scan for, which can help protect more organizations and developers.

• Increasing open-source coverage: We’re scanning more open-source platforms and repositories to discover exposed credentials, which can help mitigate risks across more of the ecosystem.

• Empowering open-source communities with preventative measures: We’re developing and offering tools that allow open-source communities to integrate credential exposure checks directly into their publish workflow, which can help prevent credential leaks before they happen.

By focusing on both detection and prevention, we aim to foster a more secure and resilient open source environment. To report exposed Google Cloud credentials, please contact gcp-credentials-reports@google.com. If you are a credential provider and would like to talk about partnering with us to scan for your credentials, please contact depsdev@google.com.

As foundation model training infrastructure scales to tens of thousands of accelerators, efficient utilization of those high-value resources becomes paramount. In particular, as the cluster gets larger, hardware failures become more frequent (~ few hours) and recovery from previously saved checkpoints becomes slower (up to 30 minutes), significantly slowing down training progress. A checkpoint represents the saved state of a model’s training progress at any given time and consists of a set of intermediary model weights and other parameters. 

We recently introduced multi-tier checkpointing in AI Hypercomputer, our integrated supercomputing system that incorporates lessons from more than a decade of Google’s expertise in AI. This solution increases the ML Goodput of large training jobs (e.g. by 6.59% in a 35K-chip workload on TPU v5p) by utilizing multiple tiers of storage, including in-cluster memory (RAM) and replication, and Google Cloud Storage, thereby minimizing lost progress during a training job and improving mean-time-to-recovery (MTTR). This solution is compatible with JAX using MaxText as a reference architecture as well as NeMo with PyTorch / GPUs.

What this means is that you can take a checkpoint at the most optimal frequency (the checkpoint save scales sub-linearly to < 5 minutes) for the biggest models and across a very large node cluster and restore in under a minute across a cluster with thousands of nodes.  

Increases in Goodput can translate directly to decreases in infrastructure costs. For example, consider the case where you are using accelerator chips to train a model that takes one month to complete. Even with a somewhat smaller training workload, the cost savings with optimal checkpointing can be significant. If you have a week-long training job spanning 1K VMs that cost $88/hour (a3-highgpu-8g), a 6.5% increase in Goodput on this training task could result in almost $1M in infrastructure savings.

More failures require more checkpointing

Probabilistically, the mean time between failure (MTBF) of a training job decreases — failures happen more frequently — as the size of the cluster increases. Therefore, it is important that foundation model producers take checkpoints more frequently so they don’t lose too much progress on their training job. In the past, Google Kubernetes Engine (GKE) customers could only write a checkpoint every 30 minutes (saving it to Cloud Storage) and had to wait up to 30 minutes to read the last saved checkpoint and distribute it to all the nodes in the cluster. 

Multi-tier checkpointing allows for much faster checkpoint writes and more frequent saves by writing data asynchronously to memory (RAM) on the node and then periodically replicating this data inside the cluster, and backing that data up to Cloud Storage. In the event of a failure, a job’s progress can be recovered quickly by using data from a nearby neighbor’s in-memory checkpoint. If the checkpoint data isn’t available in a nearby node’s RAM, checkpoints are downloaded from Cloud Storage bucket backups. With this solution, checkpoint write latency does not increase with the number of nodes in a cluster — it remains constant. Reads are also constant and scale independently, enabling faster checkpoint loading and reducing MTTR. 

Architectural details

Conceptually, the multi-tier checkpointing solution provides a single “magic” local filesystem volume for ML training jobs to use for saving checkpoints and from which to restore. It’s “magic” because while it provides ramdisk-level read/write speeds, it also provides data durability associated with Cloud Storage.

When enabled, local volume (Node storage) is the only storage tier visible to ML jobs. The checkpoints written there are automatically replicated in-cluster to one/two/or more peer nodes and are regularly backed up to Cloud Storage.

When the job restarts, the checkpoint data specific for the new portion of the training job running on the node (i.e., NodeRank) automatically appears on the local volume for ML jobs to use. Behind the scenes, the necessary data may be fetched from another node in the cluster, or from Cloud Storage. Finding the most recent fully written checkpoint (no matter where it is) also happens transparently for ML jobs.

The component responsible for data movement across tiers is called Replicator and is running on every Node as a part of a CSI driver that provides local volume mount.

Delving deeper, the Replicator performs the following critical functions:

• Centralized intelligence: It analyzes Cloud Storage backups and the collective in-cluster data to determine the most recent, complete checkpoint with which to restore a job upon restart. Furthermore, it detects successful checkpoint saves by all nodes, signaling when older data can be safely garbage-collected, and strategically decides which checkpoints to back up to Cloud Storage.

• Smart peer selection: Because it’s aware of the underlying network topology used by both TPUs and GPUs, the Replicator employs smart criteria to select replication peers for each node. This involves prioritizing a “near” peer with high bandwidth and low latency. This “near” peer may have a potentially higher risk of correlated failure (e.g., within the same TPU Slice or GPU Superblock) and as such, it also selects a “far” peer — one with slightly increased networking overhead but enhanced resilience to independent failures (e.g., that resides in a different GPU Superblock). In data parallelism scenarios, preference is given to any peers that possess identical data.

• Automatic data deduplication: When data parallelism is employed, multiple nodes run identical training pipelines, resulting in the saving of identical checkpoints. The Replicator’s peer selection ensures these nodes are paired, eliminating the need for actual data replication. Instead, each node verifies the data integrity of its peers; no additional bandwidth is consumed, replication is instantaneous, and local storage usage is significantly reduced. If peers are misconfigured, standard checkpoint copying is maintained.

• Huge-model mode with data parallelism assumption: Beyond optimization, this mode caters to the largest models, where local node storage is insufficient to house both a node’s own checkpoint as well as a peer’s data. In such cases, the ML job configures the Replicator to assume data parallelism, drastically reducing local storage requirements. This extends to scenarios where dedicated nodes handle Cloud Storage backups rather than the nodes storing the most recent checkpoints themselves.

• Optimized Cloud Storage utilization: Leveraging data deduplication, all unique data is stored in Cloud Storage only once, optimizing storage space, bandwidth consumption, and associated costs.

• Automated garbage collection: The Replicator continuously monitors checkpoint saves across all nodes. Once the latest checkpoint is confirmed to have been successfully saved everywhere, it automatically initiates the deletion of older checkpoints, while ensuring that checkpoints still being backed up to Cloud Storage are retained until the process is complete.

A wide range of checkpointing solutions

At Google Cloud, we offer a comprehensive portfolio of checkpointing solutions to meet diverse AI training needs. Options like direct Cloud Storage and Cloud Storage FUSE are simpler approaches and serve smaller to medium-scale workloads very effectively. Parallel file systems such as Lustre offer high throughput for large clusters, while multi-tier checkpointing is purpose-built for the most demanding, highest-scale (>1K nodes) training jobs that require very frequent saves and rapid recovery.

Multi-tier checkpointing is currently in preview, focused on JAX for Cloud TPUs and PyTorch on GPUs. Get started with it today by following our user guide, and don’t hesitate to reach out to your account team if you have any questions or feedback.

Businesses that rely on real-time data for decision-making and application development need a robust and scalable streaming platform, and Apache Kafka has emerged as the leading solution.

At its core, Kafka is a distributed streaming platform that allows applications to publish and subscribe to streams of records, much like a message queue or enterprise messaging system, and goes beyond traditional messaging with features like high throughput, persistent storage, and real-time processing capabilities. However, deploying, managing, and scaling Kafka clusters can be challenging. This is what Google Cloud’s Managed Service for Apache Kafka solves. This managed Kafka service is open-source compatible and portable, easy to operate, and secure, allowing you to focus on building and deploying streaming applications without worrying about infrastructure management, software upgrades, or scaling. It’s also integrated for optimal performance with other Google Cloud data services such as BigQuery, Cloud Storage and Dataflow.

While Apache Kafka offers immense power, achieving optimal performance isn’t automatic. It requires careful tuning and benchmarking. This post provides a hands-on guide to optimize your deployments for throughput and latency.

Note: We assume a high-level understanding of Apache Kafka and BASH scripting. For an introduction and overview of Apache Kafka, visit the Apache Software Foundation website. For an introduction to BASH, please visit this Geeks for Geeks tutorial.

Benchmarking Kafka producers, consumers and latencies

Benchmarking your Kafka deployment is crucial for understanding its performance characteristics and ensuring it can serve your application’s requirements. This involves a deep dive into metrics like throughput and latency, along with systematic experimentation by optimizing your producer and consumer configurations. It’s important to note that this is done at a topic / application level and should be replicated for each topic.

Optimizing for throughput and latency

The Apache Kafka bundle includes two utilities – kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh – to assess producer and consumer performance as well as latencies.

Note: while we are using some config values in order to demonstrate tool usage, it’s recommended that you use configurations (e.g. message size, message rates, etc…) that mirror your workloads.

kafka-producer-perf-test

This tool simulates producer behavior by sending a specified number of messages to a topic while measuring throughput and latencies, and takes the following flags:

• topic (required): Specifies the target Kafka topic

• num-records (required): Sets the total number of messages to send 

• record-size (required): Defines the size of each message in bytes

• throughput (required): Sets a target throughput in messages per second (use -1 to disable throttling)

• producer-props:

• bootstrap.servers (required): Comma-separated list of Kafka bootstrap server or broker addresses.

• acks (optional): Controls the level of acknowledgment required from brokers (0, 1, or all). 0 for no broker, 1 for leader broker and ‘all’ for all brokers. The default value is ‘all’.

• batch.size (optional): The maximum size of a batch of messages in bytes. The default value is 16KB.

• linger.ms (optional): The maximum time to wait for a batch to fill before sending. The default value is 0 ms.

• Compression.type (optional): any one of: none, gzip, snappy, lz4, zstd. The default value is none.

Sample code block #1: Kafka producer performance test

Important considerations

The most crucial properties are acks, batch.size, linger.ms, and compression because they directly influence producer throughput and latency. While exact settings depend on your application, we suggest these baseline configurations:

• acks: acks=1 requires acknowledgement from the leader broker only. This will give the best performance unless you need acks from all the leaders and followers.

• batch.size: 10000B, or 10 KB, is a good baseline value to start with. Increasing the batch size allows producers to send more messages in a single request, reducing overhead.

• linger.ms: 10ms is a good value as a baseline. You can try within a range of 0-50ms. Increasing linger time further can result in increased latencies. 

• compression: The recommendation is to use compression to further increase your throughput and reduce latencies.

kafka-consumer-perf-test

This tool simulates consumer behavior by fetching messages from a Kafka topic and measuring the achieved throughput and latencies. Key properties include:

• topic (required): Specifies the Kafka topic to consume from.

• bootstrap-server (required): Comma-separated list of kafka bootstrap server or broker addresses.

• messages (required): The total number of messages to consume.

• group (optional): The consumer group ID.

• fetch-size (optional): The maximum amount of data to fetch in a single request. The default value is 1048576 bytes or 1.04MB.

Sample code block #2: Kafka consumer test

Important considerations

To achieve optimal consumer throughput, the fetch-size property is crucial for tuning. The default fetch-size configuration is largely determined by your consumption and throughput needs, and can range from up to 1MB for smaller messages to 1-50MB for larger ones. It’s advisable to analyze the effects of different fetch sizes on both application responsiveness and throughput. By carefully documenting these tests and examining the resulting information, you can pinpoint performance limitations and refine your settings accordingly.

How to benchmark throughput and latencies

Benchmarking the producer

When conducting tests to measure the throughput and latencies of Kafka producers, the key parameters are batch.size, or the maximum size of a batch of messages, and linger.ms, the maximum time to wait for a batch to fill before sending. For the purposes of this benchmark, we suggest keeping acks at 1 (acknowledgment from the leader broker) to balance durability and performance. This helps us to estimate the expected throughput and latencies for a producer. Note that message size is kept constant as 1KB.

Analysis and findings

• The impact of batch size: As expected, increasing batch size generally leads to higher throughput (messages/s and MBs). We see a significant jump in throughput as we move from 1KB to 10KB batch sizes. However, further increasing the batch size to 100KB does not show a significant improvement in throughput. This suggests that an optimal batch size exists beyond which further increases may not yield substantial throughput gains.

• Impact of linger time: Increasing the linger time from 10ms to 100ms with a 100KB batch size slightly reduced throughput (from 117,187 to 111,524 messages/s). This indicates that, in this scenario, a longer linger time might not be much beneficial for maximizing throughput.

• Latency considerations: Latency tends to increase with larger batch sizes. This is because messages wait longer to be included in a larger batch before being sent. This is clearly visible when batch_size is increased from 10KB to 100KB.

Together, these findings highlight the importance of careful tuning when configuring Kafka producers. Finding the optimal balance between batch.size and linger.ms is crucial for achieving desired throughput and latency goals.

Benchmarking for consumer

To assess consumer performance, we conducted a series of experiments using kafka-consumer-perf-test, systematically varying the fetch size.

Analysis and findings

• Impact of fetch size on throughput: The results clearly demonstrate a strong correlation between fetch.size and consumer throughput. As we increase the fetch size, both message throughput (messages/s) and data throughput (MBs) improve significantly. This is because larger fetch sizes allow the consumer to retrieve more messages in a single request, reducing the overhead of frequent requests and improving data transfer efficiency.

• Diminishing returns: While increasing fetch.size generally improves throughput, we observe diminishing returns as we move beyond 100MB. The difference in throughput between 100MB and 500MB is not significant, suggesting that there’s a point where further increasing the fetch size provides minimal additional benefit.

Scaling the Google Managed Service for Apache Kafka

Based on some more experiments, we explored optimal configurations for the managed Kafka cluster. Please note that for this exercise, we kept message size as 1KB, batch size as 10KB, the topic has 1000 partitions, and the replication number is 3. The results were as follows.

Scaling your managed Kafka cluster effectively is crucial to ensure optimal performance as your requirements grow. To determine the right cluster configuration, we conducted experiments with varying numbers of producer threads, vCPUs, and memory. Our findings indicate that vertical scaling, by increasing vCPUs and memory from 3 vCPUs/12GB to 12 vCPUs/48GB, significantly improved resource utilization. With two producer threads, the cluster’s byte_in_count metric doubled and CPU utilization increased to 56% from 24%. Your throughput requirements play a vital role. With 12 vCPUs/48GB, moving from 2 to 4 producer threads nearly doubled the cluster’s bytes_in_count. You also need to monitor resource utilization to avoid bottlenecks, as increasing throughput can increase CPU and memory utilization. Ultimately, optimizing managed Kafka service performance requires a careful balance between vertical scaling of the cluster and your throughput requirements, tailored to your specific workload and resource constraints.

Build the Kafka cluster you need

In conclusion, optimizing your Google Cloud Managed Service for Apache Kafka deployment involves a thorough understanding of producer and consumer behavior, careful benchmarking, and strategic scaling. By actively monitoring resource utilization and adjusting your configurations based on your specific workload demands, you can ensure your managed Kafka clusters deliver the high throughput and low latency required for your real-time data streaming applications.

Interested in diving deeper? Explore the resources and documentations linked below:

• Apache Kafka

• kafka-producer-perf-test

• kafka-consumer-perf-test

• Getting Started

• Authentication and Security 

• Cloud Monitoring and Logging for Google’s managed Kafka service

• Code Samples

As AI moves from promising experiments to landing core business impact, the most critical question is no longer “What can it do?” but “How well does it do it?”. 

Ensuring the quality, reliability, and safety of your AI applications is a strategic imperative. To guide you, evaluation must be your North Star—a constant process that validates your direction throughout the entire development lifecycle. From crafting the perfect prompt and choosing the right model to deciding if tuning is worthwhile and evaluating your agents, robust evaluation provides the answers. 

One year ago, we launched the Gen AI evaluation service, offering capabilities to evaluate various models including Google’s foundation models, open models, proprietary foundation models, and customized models. It provided online evaluation modes with pointwise and pairwise criteria, utilizing computation and Autorater methods. 

Since then, we’ve listened closely to your feedback and focused on addressing your most important needs. That’s why today we’re excited to dive into the new features of the Gen AI Evaluation Service, designed to help you scale your evaluations, evaluate your autorater, customize your autorater with rubrics and evaluate your agents in production.

1. Scale your valuation with Gen AI batch evaluation

One of the most pressing questions for AI developers is, “How can I run evaluation at scale”? Previously, scaling evaluations could be heavy-engineered, hard to maintain, and expensive. You have to build your own batch evaluation processes combining multiple Google Cloud services. 

The new batch evaluation feature simplifies this process, providing a single API for large datasets. This means you can evaluate large volumes of data efficiently, supporting all methods and metrics available in the Gen AI evaluation service in Vertex AI. It’s designed to be cheaper and more efficient than previous approaches. 

You can learn more about how to run batch evaluation with the Gemini API in Vertex AI in this tutorial. 

2. Scrutinize your autorater and build trust

A common and critical concern we hear from developers is, “How can I customize and truly evaluate my autorater?” While using an LLM to assess an LLM-based application offers scale and efficiency, it also introduces valid questions about its limitations, robustness, and potential biases. The fundamental challenge is building trust in its results.

We believe that trust isn’t given; it’s built through transparency and control. Our features are designed to empower you to rigorously scrutinize and refine your autorater. This is achieved through two key capabilities:

1. First, you can evaluate your autorater’s quality. By creating a benchmark dataset of human-rated examples, you can directly compare the autorater’s judgments against your “source of truth.” This allows you to calibrate its performance, measure its alignment with you, and gain a clear understanding of areas that need improvement.

2. Second, you can actively improve its alignment. We provide several approaches to customize your autorater’s behavior. You can refine the autorater’s prompt with specific criteria, chain-of-thought reasoning, and detailed scoring guidelines. Furthermore, advanced settings and the ability to bring and tune the autorater with your own reference data ensures it meets your specific needs and is able to capture unique use cases.

Here is an example of analysis you can build with the new autorater customization features.

Check out the Advanced judge model customization series in the official documentation to learn more about how to evaluate and configure the judge model. For a practical example, here is a tutorial on how to customize your evaluations using an open autorater with Vertex AI Gen AI Evaluation. 

3. Rubrics-driven evaluation 

Evaluating complex AI applications can sometimes present a frustrating challenge: how can you use a fixed set of criteria when every input is different? A generic list of evaluation criteria often fails to capture the nuance of a complex multimodal use case, such as image understanding.

To solve this, our rubrics-driven evaluation feature breaks the evaluation experience into a two-step approach.

1. Step 1 – Rubric generation: First, instead of asking users to provide a static list of criteria, the system acts like a tailored test-maker. For each individual data point in your evaluation set, it automatically generates a unique set of rubrics—specific, measurable criteria adapted to that entry’s content. You can review and customize these tests, if needed.

2. Step 2 – Targeted autorating: Next, the autorater uses these custom-generated rubrics to assess the AI’s response. This is like a teacher writing unique questions for each student’s essay based on its specific topic, rather than using the same generic questions for the whole class.

This process ensures that every evaluation is contextual and insightful. It enhances interpretability by tying every score to criteria that are directly relevant to the specific task, giving you a far more accurate measure of your model’s true performance.

Here, you can see an example of the rubric-driven pairwise evaluation you will be able to produce with Gen AI evaluation service on Vertex AI.

Check out these examples of running rubric-based evaluation for  instruction-following, multimodal, and text quality. Also, we have worked with our research team to implement rubrics-based autorater for text- to-image and text-to-video. 

4. Agent evaluation

We are at the beginning of the agentic era, where agents reason, plan, and use tools to accomplish complex tasks. However, evaluating these agents presents a unique challenge. It’s no longer sufficient to just assess the final response; we need to validate the entire decision-making process. “Did the agent choose the right tool?”, “Did it follow a logical sequence of steps?”, “Did it effectively store and use information to provide personalized answers?”. These are some of the critical questions that determine an agent’s reliability.

To address some of these challenges, the Gen AI evaluation service in Vertex AI introduces capabilities specifically for agent evaluation. You can evaluate not only the agent’s final output but also gain insights into its “trajectory”—the sequence of actions and tool calls it makes. With specialized metrics for trajectory, you can assess your agent’s reasoning path. Whether you’re building with Agent Development Kit, LangGraph, CrewAI, or other frameworks, and hosting them locally or on Vertex AI Agent Engine, you can analyze if the agent’s actions were logical and if the right tools were used at the right time. All results are integrated with Vertex AI Experiments, providing a robust system to track, compare, and visualize performance, enabling you to build more reliable and effective AI agents.

Here you can find a detailed documentation with several examples of agent evaluation with Gen AI evaluation service on Vertex AI. 

Finally, we recognize that evaluation remains a research frontier. We believe that collaborative efforts are key to addressing current challenges. Therefore, we are actively working with companies like Weights & Biases, Arize, and Maxim AI. Together, we aim to find solutions for open challenges such as the cold-start data problem, multi-agent evaluation, and real-world agent simulation for validation.

Get started today

Ready to build reliable LLMs applications ready for production on Vertex AI? The Gen AI evaluation service in Vertex AI addresses the most requested features from users, providing a powerful, comprehensive suite for evaluating your AI application. By enabling you to scale evaluations, build trust in your autorater, and assess multimodal and agentic use cases, we want to foster confidence and efficiency, ensuring your LLM-based applications perform as expected in production.

Check the comprehensive documentation and code examples for the Gen AI evaluation service.