Today, many organizations are moving towards lakehouse architectures to have a single copy of their data and use multiple engines for different workloads — without having to copy or move the data. However, managing a data lakehouse can be complex, often requiring custom pipelines that are hard to operate and that aren’t interoperable between query engines. Further, governance can be challenging when you have independent systems in multiple, local silos.
One way to succeed with a lakehouse architecture is to implement a metadata layer across your data engines. BigLake metastore is Google Cloud’s fully-managed, serverless, and scalable runtime metastore based on the industry-standard Apache Iceberg REST Spec, providing a standard REST interface for wider compatibility and interoperability across OSS engines like Apache Spark, as well as Google Cloud native engines such as BigQuery. Today, we’re excited to announce that support for the Iceberg REST Catalog is now generally available.
Now your users can query using their engine of choice across open-source engines such as Apache Spark and Trino, as well as native engines like BigQuery, all backed with the enterprise security offered by Google Cloud. For example, Spark users can utilize the BigLake metastore as a serverless Iceberg catalog to share the same copy of data with other engines, including BigQuery.
BigLake metastore also provides support for key authorization mechanisms such as credential vending, allowing users to access their tables without having direct access to the files in the underlying Google Cloud Storage bucket. Finally, BigLake metastore is integrated with Dataplex Universal Catalog so you get end-to-end governance complete with comprehensive lineage, data quality, and discoverability capabilities for BigLake Iceberg tables in BigQuery. Powered by Google’s planet-scale metadata management infrastructure based on Spanner, BigLake metastore removes the need to manage custom metastore deployments, giving you the benefits of an open and flexible lakehouse with the performance and interoperability of an enterprise-grade managed service.
Leading organizations building their lakehouses with Google’s Data Cloud are already seeing the benefits of BigLake metastore.
“Spotify is leveraging BigLake and BigLake metastore as part of our efforts to build a modern lakehouse platform. By utilizing open formats and open APIs, this platform provides an interoperable and abstracted storage interface for our data. BigLake helps us make our data accessible for processing by BigQuery, Dataflow and open-source, Iceberg-compatible engines.” – Ed Byne, Product Manager, Spotify
Simplify data management and unify governance
BigLake metastore has a new UX console in which you can create and update your Iceberg Catalog. For easy access, the console lets you access all your Cloud Storage and BigQuery storage data across multiple runtimes, including BigQuery, and open-source, Iceberg-compatible engines such as Spark and Trino. For example, a data engineer can create Iceberg tables in Spark and the same data can be accessed by a data analyst in BigQuery. This gives you a single view of all of your Iceberg tables across Google Cloud, whether they’re managed by BigLake or self-managed in Cloud Storage.
The BigLake UX console also lets you quickly create a catalog for your Iceberg data in Cloud Storage, rather than having to do it from the source.
With BigLake metastore, you can enjoy the following benefits:
Unified metadata: Shared runtime metadata across various engines, data formats and modalities, so you can understand and process the same underlying data without needing proprietary connectors or data copies. This enables data engineers to share the same data across multiple engines, leading to faster time to market for their key use cases.
Open APIs for interoperability: Supports interoperability with open-source and third-party engines through Iceberg REST Catalog, so different teams can use their preferred analytics tools on a single, unified dataset.
Broad storage support: Integrated access and processing with data stored in Cloud Storage or BigQuery, helping you maximize data utility and maintain flexible storage without moving or copying data.
Serverless: Reduced TCO due to serverless and no-ops environments and scalability for any workload size.
Enterprise readiness and scale: Backed by Google’s planet-scale infrastructure and Spanner, so your metadata can scale with your data. There’s also support for Cloud Storage Dual Region and Multi-Region buckets for data and catalog redundancy.
AI-powered governance: End-to-end governance complete with comprehensive lineage, data quality, and discoverability capabilities for BigLake Iceberg tables in BigQuery, and integrated with Dataplex Universal Catalog.
Unlock new AI use cases with your data lakehouse
Google’s Data Cloud is built on Google’s vast infrastructure and powered by AI, offering a unified platform for AI-ready data. This allows you to build open lakehouse architectures designed to handle both structured and multimodal data, so you can unlock new AI use cases. With BigLake and BigLake metastore, you can enable richer AI processing on your Iceberg data using BigQuery AI functions for text generation, text or unstructured data analysis, and translation. These functions access Gemini and partner LLM models available from Vertex AI, Cloud AI APIs, or built-in BigQuery models. Further, you can train, evaluate, and run ML models like linear regression, k-means clustering, or time-series forecasts directly on your Iceberg data using BigQuery ML.
Let’s take an example. Imagine you’re a data engineer at a large retail company, and a data analyst wants to access a product returns table to view a list of returned products. Some of the returns data is inserted into an Iceberg table by a data scientist on the Marketing team using Spark. Spark uses BigLake metastore Iceberg REST Catalog as the Catalog for the Iceberg table. Then, with the help of the Iceberg REST Catalog, the data scientist can immediately analyze the returns data, using BigQuery to list the returned products, BigQuery’s AI Generate function to describe the products, and BigQuery ML to plot a logistic regression model for the returns. The whole process is fast thanks to the use of the Cloud Storage FileIO implementation (GCSFileIO), while Dataplex Universal Catalog provides governance capabilities for BigLake Iceberg tables in BigQuery.
Learn more
With BigLake metastore, you now have a fully-managed, serverless, and scalable runtime metastore, enabling an open and interoperable lakehouse for your organization. Get started with BigLake metastore and the Iceberg REST Catalog today. And to learn how to build an AI-ready lakehouse with Apache Iceberg and BigLake, watch our most recent lakehouse webinar on demand where we dive deeper into the topic.
Running AI workloads in a hybrid fashion — in your data center and in the cloud — requires sophisticated, global networks that unify cloud and on-premises resources. While Google’s Cloud WAN provides the necessary unified network fabric to connect VPCs, data centers, and specialized hardware, this very interconnectedness exposes a critical, foundational challenge: IP address scarcity and overlapping subnets. As enterprises unify their private and cloud environments, manually resolving these pervasive address conflicts can be a big operational burden.
Resolving IPv4 address conflicts has been a longstanding challenge in networking. And now, with a growing number of IP-intensive workloads and applications, customers face the crucial question of how to ensure sufficient IP addresses for their deployments.
Google Cloud offers various solutions to address private IP address challenges and facilitate communication between non-routable networks, including Private Service Connect (PSC), IPv6 addressing, and network address translation (NAT) appliances. In this post, we focus on private NAT, a feature of the Cloud NAT service. This managed service simplifies private-to-private communication, allowing networks with overlapping IP spaces to connect without complex routing or managing proprietary NAT infrastructure.
Getting to know private NAT
Private NAT allows your Google Cloud resources to connect to other VPC networks or to on-premises networks with overlapping and/or non-routable subnets, without requiring you to manage any virtual machines or appliances.
Here are some of the key benefits of private NAT:
A managed service: As a fully managed service, private NAT minimizes the operational burden of managing and scaling your own NAT gateways. Google Cloud handles the underlying infrastructure, so you can focus on your applications.
Simplified management: Private NAT simplifies network architecture by providing a centralized and straightforward way to manage private-to-private communication — across workloads and traffic paths.
High availability: Being a distributed service, private NAT offers high availability, VM-to-VM line-rate performance, and resiliency, all without having to over-provision costly, redundant infrastructure.
Scalability: Private NAT is designed to scale automatically with your needs, supporting a large number of NAT IP addresses and concurrent connections.
Figure: Cloud NAT options
Common use cases
Private NAT provides critical address translation for the most complex hybrid and multi-VPC networking challenges
Unifying global networks with Network Connectivity Center
For organizations that use Network Connectivity Center to establish a central connectivity hub, private NAT offers the essential mechanism for linking networks that possess overlapping “ non-routable” IP address ranges. This solution facilitates two primary scenarios:
VPC spoke-to-spoke: Facilitates seamless private-to-private communication between distinct VPC networks (spokes) with overlapping subnets.
VPC-to-hybrid-spoke: Enables connectivity between a cloud VPC and an on-premises network (a hybrid spoke) connected via Cloud Interconnect or Cloud VPN. Learn more here.
Figure: Private NAT with Network Connectivity Center
Enabling local hybrid connectivity in shared VPC
Organizations with shared VPC architectures can establish connectivity from non-routable or overlapping network subnets to their local Cloud Interconnects or Cloud VPN tunnels. A single private NAT gateway can manage destination routes for all workloads within the VPC.
“Thanks to private NAT, we effortlessly connected our Orange on-prem data center with the Masmovil GCP environment, even with IP address overlaps after our joint venture. This was crucial for business continuity, as it allowed us to enable communications without altering our existing environment.” – Pedro Sanz Martínez, Head of Cloud Platform Engineering, MasOrange
Figure: Enabling local hybrid connectivity using private NAT
Accommodating Cloud Run and GKE workloads
Dynamic, IP-intensive workloads such as Google Kubernetes Engine (GKE) and Cloud Run often use Non-RFC 1918 ranges such as Class E to solve for IPv4 exhaustion. These workloads often need to access resources in an on-premises network or a partner VPC, so the ability for the on-premises network to accept non-RFC 1918 ranges is critical. In most cases, central network teams do not accept non-RFC 1918 address ranges.
You can solve this by applying a private NAT configuration to the non-RFC 1918 subnet. With private NAT, all egress traffic from your Cloud Run service or GKE workloads is translated, allowing it to securely communicate with the destination network despite being on non-routable subnets. Learn about how private NAT works with different workloads here.
Configuration in action: Example setups
Let’s look at how to configure private NAT for one of these use cases using gcloud commands.
Example: connecting to a partner network with overlapping IPs
Scenario: Your production-vpc contains an application subnet (app-subnet-prod, 10.20.0.0/24). You need to connect to a partner’s network over Cloud VPN, but the partner also uses the 10.20.0.0/24 range for the resources you need to access.
Solution: Configure a private NAT gateway to translate traffic from app-subnet-prod before it goes over the VPN tunnel.
1. Create a dedicated subnet for NAT IPs. This subnet’s range is used for translation and must not overlap with the source or destination.
3. Create a private NAT gateway. This configuration specifies that only traffic from app-subnet-prod to local dynamic (match is_hybrid) destinations should be translated using IPs from pnat-subnet-prod subnet.
Now, any VM in app-subnet-prod that sends traffic to the partner’s overlapping network will have its source IP translated to an address from the 192.168.1.0/24 range, resolving the conflict.
Google Cloud’s private NAT elegantly solves the common and complex problem of connecting networks with overlapping IP address spaces. As a fully managed, scalable, and highly available service, it simplifies network architecture, reduces operational overhead, and enables you to build and connect complex hybrid and multi-cloud environments with ease.
Learn more
Ready to get started with private NAT? Check out the official private NAT documentation and tutorials to learn more and start building your own solutions today.
We are thrilled to announce the integration of TimesFM into our leading data platforms, BigQuery and AlloyDB. This brings the power of large-scale, pre-trained forecasting models directly to your data within the Google Data Cloud, enabling you to predict future trends with unprecedented ease and accuracy.
TimesFM is a powerful time-series foundation model developed by Google Research, pre-trained on a vast dataset of over 400 billion real-world time-points. This extensive training allows TimesFM to perform “zero-shot” forecasting, meaning it can generate accurate predictions for your specific data without needing to be retrained. This dramatically simplifies the process of creating and deploying forecasting models, saving you time and resources.
Now, let’s dive into what this means for you in BigQuery and AlloyDB.
TimesFM in BigQuery
We launched the AI.FORECAST function in preview at Google Cloud Next ‘25. Today, we are announcing:
TimesFM 2.5 is now supported. By specifying `model => “TimesFM 2.5”`, you can use the latest TimesFM model to achieve better forecasting accuracy and lower latency.
AI.FORECAST supports dynamic context windows up to 15K: Multiple context windows from 64 to 15K are supported, by specifying `context_window`. If not specified, a context window is selected to match the time series input size.
AI.FORECAST supports displaying historical data: Displaying historical data together with forecasts is supported by setting `output_historical_time_series` to true. The option enhances usability by enabling easier and better visualizations.
We add AI.EVALUATE for model evaluation. Users can specify the actual data to evaluate the accuracy of the forecasted value.
In this example, you can use the TimesFM 2.5 model and specify the context window = 1024 in AI.FORECAST to use the latest 1024 points as the history data. You can specify output_historical_time_series = true to display historical data together with the forecasts.
code_block
<ListValue: [StructValue([(‘code’, “WITH citibike_trips AS (rn SELECT EXTRACT(DATE FROM starttime) AS date, COUNT(*) AS num_tripsrn FROM `bigquery-public-data.new_york.citibike_trips` GROUP BY date)rnSELECT *rnFROMrn AI.FORECAST(rn TABLE citibike_trips, — History Tablern data_col => ‘num_trips’,rn timestamp_col => ‘date’,rn horizon => 300,rn output_historical_time_series => TRUE,rn model => ‘TimesFM 2.5’,rn context_window => 1024);”), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c0c24f5e0>)])]>
The first 10 days forecasted values are:
You can also visualize the results by clicking the `Visualization` tab. The results should be similar to:
In this example of AI.EVALUATE, you can use the data before “2016-08-01” as history to evaluate the forecasted bike trips against the actual data after “2016-08-01”:
code_block
<ListValue: [StructValue([(‘code’, ‘WITH citibike_trips AS (rn SELECT EXTRACT(DATE FROM starttime) AS date, usertype, COUNT(*) AS num_tripsrn FROM `bigquery-public-data.new_york.citibike_trips` GROUP BY date, usertype)rnSELECT * rnFROMrn AI.EVALUATE(rn (SELECT * FROM citibike_trips WHERE date < ‘2016-08-01’), — History time seriesrn (SELECT * FROM citibike_trips WHERE date >= ‘2016-08-01’), — Actual time seriesrn data_col => ‘num_trips’,rn timestamp_col => ‘date’,rn id_cols => [“usertype”]);’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c0c24ff10>)])]>
The SQL generates evaluation metrics based on each `usertype`:
AI.DETECT_ANOMALIES
The addition of AI.DETECT_ANOMALIES lets you specify the target data to detect anomalies against the forecasted value.
In this example of AI.DETECT_ANOMALIES, you can use the data before “2016-08-01” as history to detect anomalies in the target data after “2016-08-01”:
code_block
<ListValue: [StructValue([(‘code’, ‘WITH citibike_trips AS (rn SELECT EXTRACT(DATE FROM starttime) AS date, usertype, COUNT(*) AS num_tripsrn FROM `bigquery-public-data.new_york.citibike_trips` GROUP BY date, usertype)rnSELECT * rnFROMrn AI.DETECT_ANOMALIES(rn (SELECT * FROM citibike_trips WHERE date < ‘2016-08-01’), — History time series rn (SELECT * FROM citibike_trips WHERE date >= ‘2016-08-01’), — Target time seriesrn data_col => ‘num_trips’,rn timestamp_col => ‘date’,rn id_cols => [“usertype”]);’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c0c24f280>)])]>
The SQL generates the anomalies per usertype for each data point that is after “2016-08-01”, an example of 10 rows of results are:
TimesFM in AlloyDB
AI.FORECASTis now available in AlloyDB in preview. AlloyDB provides built-in support for TimesFM for predictions directly from inside of AlloyDB. This enables you to make predictions leveraging operational and analytical data for use cases such as sales forecasting, inventory demand prediction, or operational load modeling, without needing to export data.
Forecasting sales with AlloyDB
Let’s walk through an example of how you can forecast sales leveraging data stored in AlloyDB. Traditionally you would have to set up and maintain an ETL pipeline to extract data from AlloyDB, pull it into a data science environment, potentially deploy a forecasting model, run predictions for the model and store them. But for time-sensitive applications, these steps can be costly.
Instead, suppose you are leveraging AlloyDB for your operational workloads. You have stored sales, stock and price data, along with metadata, in a table retail_sales. You know what happened last week in terms of sales, but you want to predict what will happen next week so that you can plan accordingly to the demand.
With AlloyDB’s latest integration, you can get started with just two simple steps.
1. Register the model. Register the TimesFM modelas a model endpoint within AlloyDB’s model endpoint management in order to point to the Vertex AI endpoint where the model is hosted. This allows AlloyDB to securely send time-series data to the model and receive predictions back. Here we point to a TimesFM model deployed on Vertex AI and choose a model id “timesfm_v2”.
code_block
<ListValue: [StructValue([(‘code’, “CALLrn ai.create_model(rn model_id => ‘timesfm_v2’,rn model_type => ‘ts_forecasting’,rn model_provider => ‘google’,rn model_qualified_name => ‘timesfm_v2’,rn model_request_url => ‘https://<REGION>-aiplatform.googleapis.com/v1/projects/<PROJECT_ID>/locations/<REGION>/endpoints/<ENDPOINT_ID>:predict’ — endpoint in Vertex AI Model Gardenrn);”), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c0c24f490>)])]>
2. Generate Predictions with AI.FORECAST.Once the model is registered, you can start leveraging the AI.FORECAST function. This function takes your time-series data and prediction parameters (like the forecast horizon) and returns the forecasted values.
In this example, we’ll forecast the next 11 days of sales based on the sales data stored in our database with a confidence level of .80.
This integrated approach means you can keep your data securely within your high-performance AlloyDB instance and immediately leverage Google’s state-of-the-art forecasting capabilities. The low latency of AlloyDB, combined with the zero-shot power of TimesFM, makes real-time predictive analytics a reality for your operational workloads. Read more about our integration in this blog post.
AI.FORECAST in Agents and MCP
In addition to supporting TimesFM (AI.FORECAST) via a SQL interface, you can leverage TimesFM’s prediction capabilities on BigQuery and AlloyDB via agentic interfaces such as Agent Development Kit (ADK), MCP toolbox for Databases, and the Gemini CLI extension for Google Data Cloud.
Use BigQuery built-in forecast tool
This blog post shows you how to write your agent with ADK’s built-in BigQuery forecast tool (via TimesFM) to do the forecast task with your data. Here is a quick peek of how you can run forecasting task via natural language with an agent built with ADK:
This blog post can walk you through how to install and configure the MCP extension and use the BigQuery forecast tool in the Gemini CLI.
Cloud SQL is a proven foundation for fully managed databases, offering production-ready MySQL, PostgreSQL, and SQL Server database engines without the operational headache. With Cloud SQL, there’s no need to worry about patches, backups, and scaling limits — just connect your app and start building.
Today, we’re announcing new free trial instances designed to help you experience the power of Cloud SQL for MySQL and PostgreSQL, with no upfront commitment. Whether you’re a seasoned Google Cloud developer or new to the platform, this 30-day free trial allows you to explore, test, and truly understand the value Cloud SQL brings to your database needs.
There are two editions of Cloud SQL currently available:
Cloud SQL Enterprise Plus edition: Designed for mission-critical applications, providing the highest performance and availability with a 99.99% SLA (including maintenance). It features near-zero downtime for planned maintenance, significant performance boosts through Data Cache (using local SSD), and enhanced write throughput.
Cloud SQL Enterprise edition: Suitable for most business applications, offering high availability and managed maintenance with a 99.95% SLA. It offers all the core capabilities of Cloud SQL, striking a good balance of performance, availability, and cost.
Cloud SQL Free Trial Instance ‘Get Started’ Page
Why a dedicated Cloud SQL free trial?
You might be familiar with the standard $300 Google Cloud free trial for new users. While that’s a fantastic starting point, customers have been asking us for a more specialized offering. They want a dedicated environment to test the full power of Cloud SQL, especially enterprise-grade configurationsfor Performance, High Availability (HA), and Data Cache. This new trial is our answer.
This trial provides a significantly enhanced experience for customers developing applications on top of Cloud SQL, allowing you to:
Experience enterprise-grade features: Test critical functionality like High Availability and Data Cache, both essential to robust and performant database operations.
Onboard new users: As a developer, get hands-on with Cloud SQL without the usual hurdle of getting expense approvals for running tests.
Perform preliminary performance testing: Evaluate Cloud SQL’s performance for your specific workloads, ensuring it meets your demands.
This new Cloud SQL free trial is designed for a wide range of users:
Existing Google Cloud customers: If you’re already using other Google Cloud products, but haven’t explored Cloud SQL,this is your chance!
New Google Cloud users: Complementing the existing standard $300 trial, this offers a deeper dive into Cloud SQL’s capabilities.
What’s included in the 30-day free trial?
We want you to get a comprehensive understanding of Cloud SQL’s key value pillars: price-performance, high availability, connectivity, security, observability, ease of manageability, and open-source compatibility. Your free trial instance will be configured to help you explore all of these areas, based on the following database instance:
When you’re ready to move your workload to production, upgrading to a paid instance is a simple one-click upgrade, at any time during the trial.
Not ready to upgrade quite yet? At the end of the 30-day free trial, we automatically suspend your free trial resources, keeping the instance in a “stopped” state for an additional 90 days, at no additional charge. This should give you ample time to upgrade and continue without interruption.
Ready to get started?
Ready to unlock the full potential of your data with Cloud SQL? Creating your free trial instance is easy. If you’re new to Google Cloud, just sign up for an account and follow the instructions to create your Cloud SQL free trial instance. This exciting offer is available in all Google Cloud regions. Start your free trial and see what Cloud SQL can do for your applications.
<ListValue: [StructValue([(‘title’, ‘Disclaimer: This guide is for informational and educational purposes only and is not a substitute for professional medical advice, diagnosis, or treatment.’), (‘body’, <wagtail.rich_text.RichText object at 0x7f7c104adc40>), (‘btn_text’, ”), (‘href’, ”), (‘image’, None)])]>
Artificial intelligence (AI) is revolutionizing healthcare, but how do you take a powerful, general-purpose AI model and teach it the specialized skills of a pathologist? This journey from prototype to production often begins in a notebook, which is exactly where we’ll start.
In this guide, we’ll take the crucial first step. We’ll walk through the complete process of fine-tuning the Gemma 3 variant MedGemma. MedGemma is Google’s family of open models for the medical community, to classify breast cancer histopathology images. We’re using the full precision MedGemma model because that’s what you’ll need in order to get maximum performance for many clinical tasks. If you’re concerned about compute costs, you can quantize and fine-tune by using MedGemma’s pre-configured fine-tuning notebook instead.
To complete our first step, we’ll use the Finetune Notebook. The notebook provides you with all of the code and a step-by-step explanation of the process, so it’s the perfect environment for experimentation. I’ll also share the key insights that I learned along the way, including a critical choice in data types that made all the difference.
After we’ve perfected our model in this prototyping phase, we’ll be ready for the next step. In an upcoming post, we’ll show you how to take this exact workflow and move it to a scalable, production-ready environment using Cloud Run jobs.
Setting the stage: Our goal, model, and data
Before we get to the code, let’s set the stage. Our goal is to classify microscope images of breast tissue into one of eight categories: four benign (non-cancerous) and four malignant (cancerous). This type of classification represents one of many crucial tasks that pathologists perform in order to make an accurate diagnosis, and we have a great set of tools for the job.
We’ll be using MedGemma, a powerful family of open models from Google that’s built on the same research and technology that powers our Gemini models. What makes MedGemma special is that it isn’t just a general model: it’s been specifically tuned for the medical domain.
The MedGemma vision component, MedSigLIP, was pre-trained on a vast amount of de-identified medical imagery, including the exact type of histopathology slides that we’re using. If you don’t need the predictive power of MedGemma, you can use MedSigLIP alone as a more cost-effective option for predictive tasks like image classification. There are multiple MedSigLIP tutorial notebooks that you can use for fine-tuning.
The MedGemma language component was also trained on a diverse set of medical texts, making the google/medgemma-4b-itversion that we’re using perfect for following our text-based prompts. Google provides MedGemma as a strong foundation, but it requires fine-tuning for specific use cases—which is exactly what we’re about to do.
Handling a 4-billion parameter model requires a capable GPU, so I used an NVIDIA A100 with 40 GB of VRAM onVertex AI Workbench. This GPU has the necessary power, and it also features NVIDIA Tensor Cores that excel with modern data formats, which we’ll leverage for faster training. In an upcoming post, we’ll explain how to calculate the VRAM that’s required for your fine tuning.
My float16 disaster: A crucial lesson in stability
My first attempt to load the model used the common float16 data type to save memory. It failed spectacularly. The model’s outputs were complete garbage, and a quick debugging check revealed that every internal value had collapsed into NaN (Not a Number).
The culprit was a classic numerical overflow.
To understand why, you need to know the critical difference between these 16-bit formats:
float16 (FP16): Has a tiny numerical range. It can’t represent any number that’s greater than 65,504. During the millions of calculations in a transformer, intermediate values can easily exceed this limit, causing an overflow that creates a NaN. When a NaN appears, it contaminates every subsequent calculation.
bfloat16 (BF16): This format, developed at Google Brain, makes a crucial trade-off. It sacrifices a little bit of precision to maintain the same massive numerical range as the full 32-bit float32 format.
The bfloat16 massive range prevents overflows, which keeps the training process stable. The fix was a simple one-line change, but it was based on this critical concept.
The successful code:
code_block
<ListValue: [StructValue([(‘code’, ‘# The simple, stable solutionrnmodel_kwargs = dict(rn torch_dtype=torch.bfloat16, # Use bfloat16 for its wide numerical rangern device_map=”auto”,rn attn_implementation=”sdpa”,rn)rnrnmodel = AutoModelForImageTextToText.from_pretrained(MODEL_ID, **model_kwargs)’), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c11187d30>)])]>
Lesson learned: For fine-tuning large models, always prefer bfloat16 for its stability. It’s a small change that saves you from a world of NaN-related headaches.
The code walkthrough: A step-by-step guide
Now, let’s get to the code. I’ll break down my Finetune Notebook into clear, logical steps.
Step 1: Setup and installations
First, you need to install the necessary libraries from the Hugging Face ecosystem and log into your account to download the model.
Hugging Face authentication and and the recommended approach to handle your secrets
⚠️ Important security note: You should never hardcode secrets like API keys or tokens directly into your code or notebooks, especially in a production environment. This practice is insecure and it creates a significant security risk.
In Vertex AI Workbench, the most secure and enterprise-grade approach to handle secrets (like your Hugging Face token) is to use Google Cloud’s Secret Manger.
If you’re just experimenting and you don’t want to set up Secret Manager yet, you can use the interactive login widget. The widget saves the token temporarily in the instance’s file system.
code_block
<ListValue: [StructValue([(‘code’, ‘# Hugging Face authentication using interactive login widget:rnfrom huggingface_hub import notebook_loginrnnotebook_login()’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c11187b50>)])]>
In our upcoming post, where we move this process to Cloud Run Jobs, we’ll show you the correct and secure way to handle this token by using Secret Manager.
Step 2: Load and prepare the dataset
Next, we download the BreakHis dataset from Kaggle using the kagglehub library. This dataset includes a Folds.csv file, which outlines how the data is split for experiments. The original study used 5-fold cross-validation, but to keep the training time manageable for this demonstration, we’ll focus on Fold 1 and we’ll only use images with 100X magnification. You can explore using other folds and magnifications for more extensive experiments.
code_block
<ListValue: [StructValue([(‘code’, ‘! pip install -q kagglehubrnimport kagglehubrnimport osrnimport pandas as pdrnfrom PIL import Imagernfrom datasets import Dataset, Image as HFImage, Features, ClassLabelrnrn# Download the dataset metadatarnpath = kagglehub.dataset_download(“ambarish/breakhis”)rnprint(“Path to dataset files:”, path)rnfolds = pd.read_csv(‘{}/Folds.csv’.format(path))rnrn# Filter for 100X magnification from the first foldrnfolds_100x = folds[folds[‘mag’]==100]rnfolds_100x = folds_100x[folds_100x[‘fold’]==1]rnrn# Get the train/test splitsrnfolds_100x_test = folds_100x[folds_100x.grp==’test’]rnfolds_100x_train = folds_100x[folds_100x.grp==’train’]rnrn# Define the base path for imagesrnBASE_PATH = “/home/jupyter/.cache/kagglehub/datasets/ambarish/breakhis/versions/4/BreaKHis_v1″‘), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7c111879d0>)])]>
Step 2.1: Balance the dataset
The initial train and test splits for the 100X magnification show an imbalance between benign and malignant classes. To address this, we’ll undersample the majority class in both the training and testing sets in order to create balanced datasets with a 50/50 distribution.
We’re converting our data into the Hugging Face datasets format because it’s the easiest way to work with the SFTTrainer from their Transformers library. This format is optimized for handling large datasets, especially images, because it can load them efficiently when needed. And it gives us handy tools for preprocessing, like applying our formatting function to all examples.
code_block
<ListValue: [StructValue([(‘code’, ‘CLASS_NAMES = [rn ‘benign_adenosis’, ‘benign_fibroadenoma’, ‘benign_phyllodes_tumor’,rn ‘benign_tubular_adenoma’, ‘malignant_ductal_carcinoma’,rn ‘malignant_lobular_carcinoma’, ‘malignant_mucinous_carcinoma’,rn ‘malignant_papillary_carcinoma’rn]rnrndef get_label_from_filename(filename):rn filename = filename.replace(‘\\’, ‘/’).lower()rn if ‘/adenosis/’ in filename: return 0rn if ‘/fibroadenoma/’ in filename: return 1rn if ‘/phyllodes_tumor/’ in filename: return 2rn if ‘/tubular_adenoma/’ in filename: return 3rn if ‘/ductal_carcinoma/’ in filename: return 4rn if ‘/lobular_carcinoma/’ in filename: return 5rn if ‘/mucinous_carcinoma/’ in filename: return 6rn if ‘/papillary_carcinoma/’ in filename: return 7rn return -1rnrntrain_data_dict = {rn ‘image’: [os.path.join(BASE_PATH, f) for f in train_filenames],rn ‘label’: [get_label_from_filename(f) for f in train_filenames]rn}rntest_data_dict = {rn ‘image’: [os.path.join(BASE_PATH, f) for f in test_filenames],rn ‘label’: [get_label_from_filename(f) for f in test_filenames]rn}rnfeatures = Features({rn ‘image’: HFImage(),rn ‘label’: ClassLabel(names=CLASS_NAMES)rn})rntrain_dataset = Dataset.from_dict(train_data_dict, features=features).cast_column(“image”, HFImage())rneval_dataset = Dataset.from_dict(test_data_dict, features=features).cast_column(“image”, HFImage())rnrnprint(train_dataset)rnprint(eval_dataset)’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411dfa0>)])]>
Step 3: Prompt engineering
This step is where we tell the model what we want it to do. We create a clear, structured prompt that instructs the model to analyze an image and to return only the number that corresponds to a class. This prompt makes the output simple and easy to parse. We then map this format across our entire dataset.
code_block
<ListValue: [StructValue([(‘code’, ‘# Define the instruction promptrnPROMPT = “””Analyze this breast tissue histopathology image and classify it.rnrnClasses (0-7):rn0: benign_adenosisrn1: benign_fibroadenomarn2: benign_phyllodes_tumorrn3: benign_tubular_adenomarn4: malignant_ductal_carcinomarn5: malignant_lobular_carcinomarn6: malignant_mucinous_carcinomarn7: malignant_papillary_carcinomarnrnAnswer with only the number (0-7):”””rnrndef format_data(example):rn “””Format examples into the chat-style messages MedGemma expects.”””rn example[“messages”] = [rn {rn “role”: “user”,rn “content”: [rn {“type”: “image”},rn {“type”: “text”, “text”: PROMPT},rn ],rn },rn {rn “role”: “assistant”,rn “content”: [rn {“type”: “text”, “text”: str(example[“label”])},rn ],rn },rn ]rn return examplernrn# Apply formattingrnformatted_train = train_dataset.map(format_data, batched=False)rnformatted_eval = eval_dataset.map(format_data, batched=False)rnrnprint(“✓ Data formatted with instruction prompts”)’), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411d940>)])]>
Step 4: Load the model and processor
Here, we load the MedGemma model and its associated processor. The processor is a handy tool that prepares both the images and text for the model. We’ll also make two key parameter choices for efficiency:
torch_dtype=torch.bfloat16: As we mentioned earlier, this format ensures numerical stability.
attn_implementation="sdpa":Scaled dot product attention is a highly optimized attention mechanism that’s available in PyTorch 2.0. Think of this mechanism as telling the model to use a super-fast, built-in engine for its most important calculation. It speeds up training and inference, and it can even automatically use more advanced backends like FlashAttention if your hardware supports it.
code_block
<ListValue: [StructValue([(‘code’, ‘MODEL_ID = “google/medgemma-4b-it”rnrn# Model configurationrnmodel_kwargs = dict(rn torch_dtype=torch.bfloat16,rn device_map=”auto”,rn attn_implementation=”sdpa”,rn)rnrnmodel = AutoModelForImageTextToText.from_pretrained(MODEL_ID, **model_kwargs)rnprocessor = AutoProcessor.from_pretrained(MODEL_ID)rnrn# Ensure right padding for trainingrnprocessor.tokenizer.padding_side = “right”‘), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411d6a0>)])]>
Step 5: Evaluate the baseline model
Before we invest time and compute in fine-tuning, let’s see how the pre-trained model performs on its own. This step gives us a baseline to measure our improvement against.
code_block
<ListValue: [StructValue([(‘code’, ‘# Helper functions to run evaluationrnaccuracy_metric = evaluate.load(“accuracy”)rnf1_metric = evaluate.load(“f1″)rnrndef compute_metrics(predictions, references):rn return {rn **accuracy_metric.compute(predictions=predictions, references=references),rn **f1_metric.compute(predictions=predictions, references=references, average=”weighted”)rn }rnrndef postprocess_prediction(text):rn “””Extract just the number from the model’s text output.”””rn digit_match = re.search(r’\b([0-7])\b’, text.strip())rn return int(digit_match.group(1)) if digit_match else -1rnrndef batch_predict(model, processor, prompts, images, batch_size=8, max_new_tokens=40):rn “””A function to run inference in batches.”””rn predictions = []rn for i in range(0, len(prompts), batch_size):rn batch_texts = prompts[i:i + batch_size]rn batch_images = [[img] for img in images[i:i + batch_size]]rnrn inputs = processor(text=batch_texts, images=images, padding=True, return_tensors=”pt”).to(“cuda”, torch.bfloat16)rn prompt_lengths = inputs[“attention_mask”].sum(dim=1)rnrn with torch.inference_mode():rn outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id)rnrn for seq, length in zip(outputs, prompt_lengths):rn generated = processor.decode(seq[length:], skip_special_tokens=True)rn predictions.append(postprocess_prediction(generated))rnrn return predictionsrnrn# Prepare data for evaluationrneval_prompts = [processor.apply_chat_template([msg[0]], add_generation_prompt=True, tokenize=False) for msg in formatted_eval[“messages”]]rneval_images = formatted_eval[“image”]rneval_labels = formatted_eval[“label”]rnrn# Run baseline evaluationrnprint(“Running baseline evaluation…”)rnbaseline_preds = batch_predict(model, processor, eval_prompts, eval_images)rnbaseline_metrics = compute_metrics(baseline_preds, eval_labels)rnrnprint(f”\n{‘BASELINE RESULTS’:-^80}”)rnprint(f”Accuracy: {baseline_metrics[‘accuracy’]:.1%}”)rnprint(f”F1 Score: {baseline_metrics[‘f1′]:.3f}”)rnprint(“-” * 80)’), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411ddf0>)])]>
The performance of the baseline model was evaluated on both 8-class and binary (benign/malignant) classification:
8-Class accuracy: 32.6%
8-Class F1 score (weighted): 0.241
Binary accuracy: 59.6%
Binary F1 score (malignant): 0.639
This output shows that the model performs better than random chance (12.5%), but there’s significant room for improvement, especially in the fine-grained 8-class classification.
A quick detour: Few-shot learning vs. fine-tuning
Before we start training, it’s worth asking: is fine-tuning the only way? Another popular technique is few-shot learning.
Few-shot learning is like giving a smart student a few examples of a new math problem right before a test. You aren’t re-teaching them algebra, you’re just showing them the specific pattern you want them to follow by providing examples directly in the prompt. This is a powerful technique, especially when you’re using a closed model through an API where you can’t access the internal weights.
So why did we choose fine-tuning?
We can host the model: Because MedGemma is an open model, we have direct access to its architecture. This access lets us perform fine-tuning to create a new, permanently updated version of the model.
We have a good dataset: Fine-tuning lets the model learn the deep, underlying patterns in our hundreds of training images far more effectively than just showing it a few examples in a prompt.
In short, fine-tuning creates a true specialist model for our task, which is exactly what we want.
Step 6: Configure and run fine-tuning with LoRA
This is the main event! We’ll use Low-Rank Adaptation (LoRA), which is much faster and more memory-efficient than traditional fine-tuning. LoRA works by freezing the original model weights and training only a tiny set of new adapter weights. Here’s a breakdown of our parameter choices:
r=8: The LoRA rank. A lower rank means fewer trainable parameters, which is faster but less expressive. A higher rank has more capacity, but risks overfitting on a small dataset. Rank 8 is a great starting point that balances performance and efficiency.
lora_alpha=16: A scaling factor for the LoRA weights. A common rule of thumb is to set it to twice the rank (2 × r).
lora_dropout=0.1: A regularization technique. It randomly deactivates some LoRA neurons during training to prevent the model from becoming overly specialized and failing to generalize.
code_block
<ListValue: [StructValue([(‘code’, ‘# LoRA Configurationrnpeft_config = LoraConfig(rn r=8,rn lora_alpha=16,rn lora_dropout=0.1,rn bias=”none”,rn target_modules=”all-linear”,rn task_type=”CAUSAL_LM”,rn)rnrn# Custom data collator to handle images and textrndef collate_fn(examples):rn texts, images = [], []rn for example in examples:rn images.append([example[“image”]])rn texts.append(processor.apply_chat_template(example[“messages”], add_generation_prompt=False, tokenize=False).strip())rn batch = processor(text=texts, images=images, return_tensors=”pt”, padding=True)rn labels = batch[“input_ids”].clone()rn labels[labels == processor.tokenizer.pad_token_id] = -100rn image_token_id = processor.tokenizer.convert_tokens_to_ids(processor.tokenizer.special_tokens_map[“boi_token”])rn labels[labels == image_token_id] = -100rn labels[labels == 262144] = -100rn batch[“labels”] = labelsrn return batchrnrn# Training argumentsrntraining_args = SFTConfig(rn output_dir=”medgemma-breastcancer-finetuned”,rn num_train_epochs=5,rn per_device_train_batch_size=1,rn per_device_eval_batch_size=1,rn gradient_accumulation_steps=8,rn gradient_checkpointing=True,rn optim=”paged_adamw_8bit”,rn learning_rate=5e-4,rn lr_scheduler_type=”cosine”,rn warmup_ratio=0.03, # Warm up LR for first 3% of trainingrn max_grad_norm=0.3, # Clip gradients to prevent instabilityrn bf16=True, # Use bfloat16 precisionrn logging_steps=10,rn save_strategy=”steps”,rn save_steps=100,rn eval_strategy=”epoch”,rn push_to_hub=False,rn report_to=”none”,rn gradient_checkpointing_kwargs={“use_reentrant”: False},rn dataset_kwargs={“skip_prepare_dataset”: True},rn remove_unused_columns=False,rn label_names=[“labels”], rn)rnrn# Initialize and run the trainerrntrainer = SFTTrainer(rn model=model,rn args=training_args,rn train_dataset=formatted_train,rn eval_dataset=formatted_eval,rn peft_config=peft_config,rn processing_class=processor,rn data_collator=collate_fn,rn)rnrnprint(“Starting training…”)rntrainer.train()rntrainer.save_model()’), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411d7c0>)])]>
The training took about 80 minutes on the A100 GPU with VRAM 40 GB. The results looked promising, with the validation loss steadily decreasing.
Important (time saving!) tip: If your training gets interrupted for any reason (like a connection issue or exceeding resource limits), you can resume the training process from a saved checkpoint by using the resume_from_checkpoint argument in trainer.train(). Checkpoints can save you valuable time because they’re saved at every save_steps interval as defined in TrainingArguments.
Step 7: The final verdict – evaluating our fine-tuned model
After training, it’s time for the moment of truth. We’ll load our new LoRA adapter weights, merge them with the base model, and then run the same evaluation that we ran for the baseline.
code_block
<ListValue: [StructValue([(‘code’, ‘# Clear memory and load the final modelrndel modelrntorch.cuda.empty_cache()rngc.collect()rnrn# Load base model againrnbase_model = AutoModelForImageTextToText.from_pretrained(rn MODEL_ID,rn torch_dtype=torch.bfloat16,rn device_map=”auto”,rn attn_implementation=”sdpa”rn)rnrn# Load LoRA adapters and merge them into a single modelrnfinetuned_model = PeftModel.from_pretrained(base_model, training_args.output_dir)rnfinetuned_model = finetuned_model.merge_and_unload()rnrn# Configure for generationrnfinetuned_model.generation_config.max_new_tokens = 50rnfinetuned_model.generation_config.pad_token_id = processor_finetuned.tokenizer.pad_token_idrnfinetuned_model.config.pad_token_id = processor_finetuned.tokenizer.pad_token_idrnrn# Load the processor and run evaluationrnprocessor_finetuned = AutoProcessor.from_pretrained(training_args.output_dir)rnfinetuned_preds = batch_predict(finetuned_model, processor_finetuned, eval_prompts, eval_images, batch_size=4)rnfinetuned_metrics = compute_metrics(finetuned_preds, eval_labels)’), (‘language’, ‘lang-py’), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7bf411ddc0>)])]>
Final results
So, how did the fine tuning impact performance? Let’s look at the numbers for 8-class accuracy and macro F1.
The results are great! After fine-tuning, we see a dramatic improvement:
8-Class: Accuracy jumped from 32.6% to 87.2% (+54.6%) and F1 from 0.241 to 0.865.
Binary: Accuracy increased from 59.6% to 99.0% (+39.4%) and F1 from 0.639 to 0.991.
This project shows the incredible power of fine-tuning modern foundation models. We took a generalist AI that was already pre-trained on relevant medical data, gave it a small, specialized dataset, and taught it a new skill with remarkable efficiency. The journey from a generic model to a specialized classifier is more accessible than ever, opening up exciting possibilities for AI in medicine and beyond.
The fastest way to transform your business is here. Today, we’re bringing Gemini 3, our most intelligent model, to every developer and enterprise team. It’s the best model in the world for multimodal understanding,and our most powerful agentic and vibe-coding model yet. Plus, Gemini 3 Pro tops the LMArena Leaderboard with a breakthrough score of 1501 Elo. You can learn more about the model capabilities here.
State-of-the-art reasoning and multimodality: Gemini 3 uses multimodal understanding and state-of-the-art reasoning to analyze text, video, and files all at once. Applications can range from analyzing X-rays and MRI scans to assist in faster diagnostics; to automatically generate transcripts and metadata for podcast content; or to analyzing streams of machine logs to anticipate equipment failure before it happens.
Powerful agentic coding and front-end creation: Gemini 3 is our most powerful agentic and vibe-coding model yet for transforming application development and design. With Gemini 3, enterprises can rapidly prototype full front-end interfaces with a single prompt and leverage agentic coding to quickly move from prototype to production.
Advanced tool use and planning: Gemini 3 enables advanced reasoning with large sets of tools, facilitating long-running tasks across your enterprise systems and data. Businesses can now leverage Gemini 3 to execute tasks like financial planning, supply chain adjustments, and contract evaluation.
Taken together, Gemini 3 is our most intelligent model for helping enterprises transform their businesses for the agentic future.
State-of-the-art reasoning and multimodality
Consider the friction your teams face every day – the data you need exists, but extracting meaning from it forces your smartest people to perform tedious, manual work. That’s why we built Gemini 3 from the ground up to synthesize information about any topic across multiple modalities, including text, images, video, audio, and code.
What this means for your business:
Deeply understand any topic or dataset: Gemini 3 is our most factually accurate model. You can produce personalized training and employee onboarding, perform legal and contract analysis, or handle procurement, with confidence in the model’s understanding of your business.
Make better, data-backed decisions: Gemini 3’s powerful multimodal understanding makes sense of your data, no matter where it comes from. For example, you can more accurately analyze videos, factory floor images, and customer calls alongside text reports, giving you a more unified view of your data.
How customers are already seeing impact:
“As organizations generate and work with vast amounts of unstructured data, Gemini 3 Pro brings a new level of multimodal understanding, planning, and tool-calling that transforms how Box AI interprets and applies your institutional knowledge. The result is content actively working for you to deliver faster decisions and execute across mission-critical workflows, from sales and marketing to legal and finance. We’re excited to offer Gemini 3 Pro to customers today through the Box AI Studio.”
Ben Kus, CTO, Box
“Presentations.AI uses Gemini 3’s multimodal reasoning to analyze company info, extract key strategic moves, and generate content that enables enterprise sales teams to walk into C-suite meetings with intelligence that took analysts 6 hours to compile – generated in 90 seconds.”
Sumanth Raghavendra, CEO and Co-founder, Presentaions.AI
“Gemini 3 represents a significant advancement in multimodal AI. Rakuten partnered with Google to perform alpha testing, and its ability to handle real-world conditions across both audio and vision modalities, especially in challenging scenarios like overlapping speakers or blurry images, sets it apart for enterprise applications. From accurately transcribing 3-hour multilingual meetings with superior speaker identification, to extracting structured data from poor-quality document photos, outperforming baseline models by over 50%, it showcased impressive capabilities that redefine enterprise potential.”
Yusuke Kaji General Manager, AI for Business, Rakuten Group, Inc.
Powerful agentic coding and front-end creation
Many technical teams and developers are often bogged down by the heavy lift of maintaining brittle legacy systems and the cognitive load of juggling disconnected tools.
Gemini 3 has powerful agentic coding capabilities to enable legacy code migration and software testing that act as a force multiplier for technical teams. With a 1M token context window that leads the industry on long context performance, Gemini 3 outperforms previous generations and can consume entire code bases to help developers be more efficient than ever before. Finally, with dramatic improvements in frontend quality, developers can now use Gemini 3 to generate and render richer aesthetics and more sophisticated UI components faster and more reliably.
Accessible through the terminal via Gemini CLI, as well as Google’s new agentic development platform, Google Antigravity, Gemini 3’s powerful intelligence enables it to better synthesize disparate pieces of code and following complex user instructions to handle multi-step development tasks simultaneously. Third party coding platforms like Cursor, GitHub, JetBrains, Manus, Replit, and moreare already integrating Gemini 3 Pro into their tools for developers.
What this means for your business:
Accelerate the move from concept to execution: The enhanced zero-shot generation and exceptional instruction following of Gemini 3 allows development teams to rapidly generate everything from well-organized wireframes to stunning high-fidelity frontend prototypes with superior aesthetics and sophisticated UI components.
Help technical teams do more, safely: Because Gemini 3 is the best vibe coding and agentic coding model we’ve ever built, it’s even better at updating old code, running software tests, and handling complex operations – all with our most comprehensive set of safety evaluations to date.
How customers are already seeing impact:
“We’re excited to partner with Google to launch Gemini 3 in Cursor! Gemini 3 Pro shows noticeable improvements in frontend quality, and works well for solving the most ambitious tasks.”
Sualeh Asif, Co-founder and Chief Product Officer, Cursor
“With Gemini 3 Pro in Figma Make, teams have a strong foundation to explore and steer their ideas with code-backed prototypes. The model translates designs with precision and generates a wide, inventive range of styles, layouts, and interactions. As foundation models get better, Figma gets better — and I’m excited to see how Gemini 3 Pro helps our community unlock new creative possibilities.”
Loredana Crisan, Chief Design Officer, Figma
“By bringing Gemini 3 Pro to GitHub Copilot, we’re seeing promising gains in how quickly and confidently developers can move from idea to code. In our early testing in VS Code, Gemini 3 Pro demonstrated 35% higher accuracy in resolving software engineering challenges than Gemini 2.5 Pro. That’s the kind of potential that translates to developers solving real-world problems with more speed and effectiveness.”
Joe Binder, VP of Product, GitHub
“At JetBrains, we pride ourselves on code quality, so we challenged Gemini 3 Pro with demanding frontline tasks: from generating thousands of lines of front-end code to even simulating an operating-system interface from a single prompt. The new Gemini 3 Pro model advances the depth, reasoning, and reliability of AI in developer tools, showing more than a 50% improvement over Gemini 2.5 Pro in the number of solved benchmark tasks. In collaboration with Google, we’re now integrating Gemini 3 Pro into Junie and AI Assistant, to deliver smarter, more context-aware experiences to millions of developers worldwide.”
Vladislav Tankov, Director of AI, JetBrains
“Gemini 3 Pro truly stands out for its design capabilities, offering an unprecedented level of flexibility while creating apps. Like a skilled UI designer, it can range from well-organized wireframes to stunning high-fidelity prototypes.”
Michele Catasta, President & Head of AI, Replit
Advanced tool use and planning
When using AI to work through complex problems, clarity is key. It’s why we trained Gemini 3 to be stronger at tool use and planning so it could be a reliable collaborator when you’re creating sophisticated agents for long-running complex business tasks . Whether you’re building agents to complete multi-step tasks, create plans, or do business planning, Gemini 3 helps you achieve the right outcomes.
What this means for your business:
Build agents that help you forecast: Gemini 3 is the best vibe coding and agentic coding model we’ve ever built – making our products more autonomous and boosting developer productivity.Combined with state-of-the-art reasoning, it means you can execute and forecast quarterly planning, customer support needs, demand campaigns, and more.
Pair strategy with agent execution: Gemini 3’s advanced tool use and reasoning capabilities means you can connect your high-level strategy with the business tools that will carry out the actual work to assist in items like budgeting to full-cycle customer support.
How customers are already seeing impact:
“Gemini 3 Pro is significantly enhancing our user experience on complex agent tasks that require multi-step planning. We immediately achieved a 10% boost in the relevancy of responses for a complex code-generation task used for data retrieval and noted a further 30% reduction in tool-calling mistakes. Ultimately this means our customers get correct answers more often, and more quickly.”
Bob Bradley, Vice President, Data Science & AI Engineering, Geotab
“We’re delighted to see the launch of Gemini 3! With this release, we’ve observed even stronger performance in the model’s reasoning and problem-solving capabilities. Many of Manus’ recent advancements—such as Wide Research and the web-building capabilities introduced in Manus 1.5—have become significantly more powerful with Gemini 3’s support. We look forward to continuing our partnership and delivering even better experiences for our users together.”
Tao Zhang, Co-Founder and Chief Product Officer, Manus AI
“Gemini 3 is a major leap forward for agentic AI. It follows complex instructions with minimal prompt tuning and reliably calls tools, which are critical capabilities to build truly helpful agents. This advancement accelerates Shopify’s ability to build agentic AI tools that solve complex commerce challenges for our merchants.”
“Our early evaluations indicate that Gemini 3 is delivering state-of-the-art reasoning with depth and nuance. We have observed measurable and significant progress in both legal reasoning and complex contract understanding. We deeply value the opportunity to collaborate closely with Google DeepMind to validate how these improvements translate into real-world, professional-grade performance for our users. This partnership is vital to bringing the most advanced AI to market with confidence and transparency.”
Joel Hron, Chief Technology Officer, Thomson Reuters
“At Wayfair, we’ve been piloting Google’s Gemini 3 Pro to turn complex partner support SOPs into clear, data-accurate infographics for our field associates. Compared with Gemini 2.5 Pro, it’s a clear step forward in handling structured business tasks that require precision and consistency — helping our teams grasp key information faster and support partners more effectively.”
Fiona Tan, CTO, Wayfair
At WRTN, we leverage Gemini 3 across the full spectrum of our business—from powering Story Generation in Crack and delivering contextual Companion Chat to driving Memory Management and complex B2B Agent Projects. Gemini 3’s multi-lingual capabilities are stellar, especially in high-fidelity languages like Korean, where every model iteration becomes dramatically more natural and stable across all domains. This stability is critical for our agentic planning workflows. The direct and iterative partnership with the Gemini team is what makes this collaboration truly game-changing.
DJ Lee, Chief Product Officer, WRTN Technologies Inc.
Get started with Gemini 3
Today, you can safely put our most powerful agentic and vibe-coding model to work. We’re making Gemini 3 available where your teams already are:
For business teams: You can access Gemini 3 Pro in preview on Gemini Enterprise, our advanced agentic platform for teams to discover, create, share, and run AI agents all in one secure platform.
For developers: You can start building with Gemini 3 Pro in preview on Vertex AI today. Gemini 3 is also available in Google Antigravity, Gemini CLI, AI Studio, and more.
At Google Cloud, we take our role in the financial ecosystem in Europe very seriously. We also firmly believe that digital operational resilience is vital to safeguarding and enhancing innovation.
Today, we mark a significant milestone in our long-term commitment to the European financial services sector. The European Supervisory Authorities (ESAs) have officially designated Google Cloud EMEA Limited (Google Cloud EMEA), together with its subsidiaries, as a critical Information and Communication Technology (ICT) third-party service provider (CTPP) under the EU Digital Operational Resilience Act (DORA).
This designation acknowledges the systemic importance of the financial entities that rely on our services, as well as the importance of the workloads they have deployed. We welcome this new phase under DORA, and we remain committed to working with our customers and our regulators under DORA to drive towards even greater resilience for the European financial system.
Embracing direct oversight
Google Cloud EMEA has been assigned a dedicated Lead Overseer who will assess our strength in managing ICT risks through oversight. This oversight establishes a direct communication channel between Google Cloud and financial regulators in the EU, and provides a significant opportunity to enhance understanding, transparency, and trust between all parties.
We are confident that this structured dialogue will help us learn and contribute to improved risk management and resilience across the entire sector. We will approach our relationship with the ESAs and our Lead Overseer with the same commitment to ongoing transparency, collaboration, and assurance that we offer our customers and their regulators today.
Keeping customer success in focus
Along with our commitment to successful oversight, we remain focused on supporting our customers’ DORA compliance journeys with helpful resources like our Register of Information Guide and our ICT Risk Management Customer Guide. If you haven’t already, we also encourage our financial entity customers to consider our DORA-specific contract and subcontractor resources. Please contact your Google Cloud representative for further details.
As all financial entities subject to DORA will know, CTPP oversight does not replace your own responsibilities under DORA. That said, by supplementing risk management by financial entities and creating a clear mechanism for information and learnings to flow between CTPPs and key EU and national supervisory stakeholders, we feel confident that customers and users will benefit from the oversight of CTPPs.
Looking ahead
We value the constructive dialogue the ESAs have fostered with industry, and look forward to continuing this collaboration with our Lead Overseer. We believe that together we can help to build a more resilient and secure financial sector in Europe.
As we move forward in this new era of direct oversight, our goal remains to make Google Cloud the best possible service for sustainable, digital transformation for all European organizations on their terms.
AI is shifting from single-response models to complex, multi-step agents that can reason, use tools, and complete sophisticated tasks. This increased capability means you need an evolution in how you evaluate these systems. Metrics focused only on the final output are no longer enough for systems that make a sequence of decisions.
A core challenge is that an agent can produce a correct output through an inefficient or incorrect process—what we call a “silent failure”. For instance, an agent tasked with reporting inventory might give the correct number but reference last year’s report by mistake. The result looks right, but the execution failed. When an agent fails, a simple “wrong” or “right” doesn’t provide the diagnostic information you need to determine where the system broke down.
To debug effectively and ensure quality, you must understand multiple aspects of the agent’s actions:
The trajectory—the sequence of reasoning and tool calls that led to the result.
The overall agentic interaction – the full conversation between the user and the agent (Assuming a chat agent)
Whether the agent was manipulated into its actions.
This article outlines a structured framework to help you build a robust, tailored agent evaluation strategy so you can trust that your agent can move from a proof-of-concept (POC) to production.
Start with success: Define your agent’s purpose
An effective evaluation strategy is built on a foundation of clear, unambiguous success criteria. You need to start by asking one critical question: What is the definition of success for this specific agent? These success statements must be specific enough to lead directly to measurable metrics.
Vague goal (not useful)
Clear success statement (measurable)
“The agent should be helpful.”
RAG agent: Success is providing a factually correct, concise summary that is fully grounded in known documents.
“The agent should successfully book a trip.”
Booking agent: Success is correctly booking a multi-leg flight that meets all user constraints (time, cost, airline) with no errors.
By defining success first, you establish a clear benchmark for your agent to meet.
A purpose-driven evaluation framework
A robust evaluation should have success criteria and associated testable metrics that cover three pillars.
Pillar 1: Agent success and quality
This assesses the complete agent interaction, focusing on the final output and user experience. Think of this like an integration test where the agent is tested exactly as it would be used in production.
What it measures: The end result.
Example metrics: Interaction correctness, task completion rate, conversation groundedness, conversation coherence, and conversation relevance.
Pillar 2: Analysis of process and trajectory
This focuses on the agent’s internal decision-making process. This is critical for agents that perform complex, dynamic reasoning. Think of this like a series of unit tests for each decision path of your agent.
What it measures: The agent’s reasoning process and tool usage.
Key metrics: Tool selection accuracy, reasoning logic, and efficiency.
Pillar 3: Trust and safety assessment
This evaluates the agent’s reliability and resilience under non-ideal conditions. This is to prevent adversarial interactions with your agents. The reality is that when your agents are in production, they may be tested in unexpected ways, so it’s important to build trust that your agent can handle these situations.
What it measures: Reliability under adverse conditions.
Key metrics: Robustness (error handling), security (resistance to prompt injection), and fairness (mitigation of bias).
Define your tests: Methods for evaluation
With a framework in place, you can define specific tests that should be clearly determined by the metrics you chose. We recommend a multi-layered approach:
Human evaluation
Human evaluation is essential to ground your entire evaluation suite in real-world performance and domain expertise. This process establishes ground truth by identifying the specific failure modes the product is actually exhibiting and where it’s not able to meet your success criteria.
LLM-as-a-judge
Once human experts identify and document specific failure modes, you can build scalable, automated tests using an LLM to score agent performance. LLM-as-a-judge processes are used for complex, subjective failures and activities and can be used as rapid, repeatable tests to determine agent improvement. Before deployment, you should align the LLM judge to the human evaluation by comparing the judge’s output against the original manual human output, groundtruthing the results.
Code-based evaluations
These are the most inexpensive and deterministic tests, often identified in Pillar 2 by observing the agent trajectories. They are ideal for failure modes that can be checked with simple Python functions or logic, such as ensuring the output is JSON or meets specific length requirements.
Method
Primary Goal
Evaluation Target
Scalability and Speed
Human evaluation
Establish “ground truth” for subjective quality and nuance.
High and fast; ideal for automated regression testing.
Adversarial testing
Test agent robustness and safety against unexpected/malicious inputs.
The agent’s failure mode (whether the agent fails safely or produces a harmful output).
Medium; requires creative generation of test cases.
Generate high-quality evaluation data
A robust framework is only as good as the data it runs on. Manually writing thousands of test cases creates a bottleneck. The most robust test suites blend multiple techniques to generate diverse, relevant, and realistic data at scale.
Synthesize conversations with “dueling LLMs”: You can use a second LLM to role-play as a user, generating diverse, multi-turn conversational data to test your agent at scale. This is great for creating a dataset to be used for Pillar 1 assessments.
Use and anonymize production data: Use anonymized, real-world user interactions to create a “golden dataset” that captures actual use patterns and edge cases.
Human-in-the-loop curation: Developers can save valuable interactive sessions from logs or traces as permanent test cases, continuously enriching the test suite with meaningful examples.
Do I need a golden dataset?
You always need evaluation data, such as logs or traces, to run any evaluation. However, you don’t always need a pre-labeled golden dataset to start. While a golden dataset—which provides perfect, known-good outputs—is crucial for advanced validation (like understanding how an agent reaches a known answer in RAG or detecting regressions), it shouldn’t be a blocker.
How to start without one
It’s possible to get started with just human evaluation and vibes-based evaluation metrics to determine initial quality. These initial, subjective metrics and feedback can then be adapted into LLM-as-a-Judge scoring for example:
Aggregate and convert early human feedback into a set of binary scores (Pass/Fail) for key dimensions like correctness, conciseness, or safety tested by LLM-as-a-Judge. The LLM-as-a-Judge then automatically scores the agent interaction against these binary metrics to determine overall success or failure. The agent’s overall quality can then be aggregated and scored with a categorical letter grading system for example ‘A’ – All binary tests pass, ‘B’ – ⅔ of binary tests pass, ‘C’ – ⅓ of binary tests pass etc.
This approach lets you establish a structured quality gate immediately while you continuously build your golden dataset by curating real-world failures and successes.
Operationalize the process
A one-time evaluation is just a snapshot. To drive continuous improvement, you must integrate the evaluation framework into the engineering lifecycle. Operationalizing evaluation changes it into an automated, continuous process.
Integrate evaluation into CI/CD
Automation is the core of operationalization. Your evaluation suite should act as a quality gate that runs automatically with every proposed change to the agent.
Process: The pipeline executes the new agent version against your reference dataset, computes key metrics, and compares the scores against predefined thresholds.
Outcome: If performance scores fall below the threshold, the build fails, which prevents quality regressions from reaching production.
Monitor performance in production
The real world is the ultimate test. You should monitor for:
Operational metrics: Tool call error rates, API latencies, and token consumption per interaction.
Quality and engagement metrics: User feedback (e.g., thumbs up/down), conversation length, and task completion rates.
Drift detection: Monitor for significant changes in the types of user queries or a gradual decrease in performance over time.
Create a virtuous feedback loop
The final step is to feed production data back into your evaluation assets. This makes your evaluation suite a living entity that learns from real-world use.
Review: Periodically review production monitoring data and conversation logs.
Identify: Isolate new or interesting interactions (especially failures or novel requests) that aren’t in your current dataset.
Curate and add: Anonymize these selected interactions, annotate them with the “golden” expected outcome, and add them to your reference dataset.
This continuous cycle ensures your agent becomes more effective and reliable with every update. You can track and visualize the results from these cycles by exporting the runs of these tests and leveraging dashboarding tools to see how the quality of your agent is evolving over time.
Today, we’re announcing Dhivaru, a new Trans-Indian Ocean subsea cable system that will connect the Maldives, Christmas Island and Oman. This investment will build on the Australia Connect initiative, furthering the reach, reliability, and resilience of digital connectivity across the Indian Ocean.
Reach, reliability and resilience are integral to the success of AI-driven services for our users and customers. Tremendous adoption of groundbreaking services such as Gemini 2.5 Flash Image (aka Nano Banana) and Vertex AI, mean resilient connectivity has never been more important for our users. The speed of AI adoption is also outpacing anyone’s predictions, and Google is investing to meet this long-term demand.
“Dhivaru” is the line that controls the main sail on traditional Maldivian sailing vessels, and signifies the skill, strength, and experience of the early sailors navigating the seas.
In addition to the cable investment, Google will be investing in creating two new connectivity hubs for the region. The Maldives and Christmas Island are naturally positioned for connectivity hubs to help improve digital connectivity for the region, including Africa, the Middle East, South Asia and Oceania.
“Google’s decision to invest in the Maldives is a strong signal of confidence in our country’s stable and open investment environment, and a direct contribution to my vision for a diversified, inclusive, and digitized Maldivian economy. As the world moves rapidly toward an era defined by digital transformation and artificial intelligence, this project reflects how the Maldives is positioning itself at the crossroads of global connectivity — leveraging our strategic geography to create new economic opportunities for our people and to participate meaningfully in the future of the global economy.” – His Excellency the President of Republic of Maldives
“We are delighted to partner with Google on this landmark initiative to establish a new connectivity hub in the Maldives. This project represents a major step forward in strengthening the nation’s digital infrastructure and enabling the next wave of digital transformation. As a leading digital provider, Ooredoo Maldives continues to expand world-class connectivity and digital services nationwide. This progress opens new opportunities for businesses such as tourism, enabling smarter operations, improved customer experiences and greater global reach. We are proud to be powering the next phase of the Digital Maldives.” – Ooredoo Maldives CEO and MD, Khalid Al Hamadi.
“Dhiraagu is committed to advancing the digital connectivity of the Maldives and empowering our people, communities, and businesses. Over the years, we have made significant investments in building robust subsea cable systems — transforming the digital landscape — connecting the Maldives to the rest of the world and enabling the rollout of high-speed broadband across the nation. We are proud and excited to partner with Google on their expansion of subsea infrastructure and the establishment of a new connectivity hub in Addu City, the southernmost city of the Maldives. This strategic collaboration with one of the world’s largest tech players marks another milestone in strengthening the nation’s presence within the global subsea infrastructure, and further enhances the reliability and resiliency of our digital ecosystem.” – Ismail Rasheed, CEO & MD, DHIRAAGU
Connectivity hubs for the Indian Ocean region
Connectivity hubs are strategic investments designed to future-proof regional connectivity and accelerate the delivery of next-generation services through three core capabilities: Cable switching, content caching, and colocation.
Cable switching: Delivering seamless resilience
Google carefully selects the locations for our connectivity hubs to minimize the distance data has to travel before it has a chance to ‘switch paths’.. This capability improves resilience, and ensures robust, high-availability connectivity across the region. The hubs also allow automatic re-routing of traffic between multiple cables. If one cable experiences a fault, traffic will automatically select the next best path and continue on its way. This ensures high availability not only for the host country, but minimizes downtime for services and users across the region.
Content caching: Accelerating digital services
Low latency is critical for optimal user experience. One of Google’s objectives is to serve content from as close to our users and customers as possible. By caching — storing copies of the most popular content locally — Google can reduce the latency to retrieve or view this content, improving the quality of services.
Colocation: Fostering a local ecosystem
Connectivity hubs are often in locations where users have limited access to high quality data centers to house their services and IT hardware, such as islands. Although these facilities are not very large as compared to a Google data center, Google understands the benefits of shared infrastructure, and is committed to providing rack space to carriers and local companies.
Energy efficiency
Subsea cables are very energy efficient. As a result, even when supporting multiple cables, content storage and colocation, a Google connectivity hub requires far less power than a typical data center. They are primarily focused on networking and localized storage and not the large demands supporting AI, cloud and other important building blocks of the Internet. Of course, the power required for a connectivity hub can still be a lot for some smaller locations, and where it is, Google is exploring using its power demand to accelerate local investment in sustainable energy generation, consistent with its long history of stimulating renewable energy solutions.
These new connectivity hubs in the Maldives and Christmas Island are ideally situated to deepen the resilience of internet infrastructure in the Indian Ocean Region. The facilities will help power our products, strengthen local economies and bring AI benefits to people and businesses around the world. We look forward to announcing future subsea cables and connectivity hubs and further enhancing the Internet’s reach, reliability, and resilience.
At Google Cloud, we have the honor of partnering with some of the most brilliant and inventive individuals across the world. Each year, the Google Cloud Partner All-stars program honors these remarkable people for their dedication to innovation and commitment to excellence. Our 2025 All-stars are pushing our industry forward, and we’re thrilled to celebrate them.
2025 Spotlight: AI Innovation
For 2025, we’re excited to introduce a new category that recognizes strategic leaders in enterprise-wide AI adoption. These honorees are trusted advisors, helping customers transform their business using Google AI. This includes implementing agentic AI to transform core processes, create new revenue streams, or redefine operating models.
These All-stars showcase a holistic vision for how AIintegrates into a customer’s culture and strategy to drive lasting, measurable transformation that fundamentally alters business processes.
What sets Partner All-stars apart? The following qualities define what it means to be a Partner All-star:
AI Innovation
Guides customers through profound business transformation by driving enterprise-wide AI adoption
Establishes a strategic vision for integrating AI and autonomous agents into a customer’s operating model
Leverages agentic AI to redefine core processes, create new revenue streams, and transform business outcomes
Delivers lasting, measurable results that fundamentally alter a customer’s business processes
Delivery Excellence
Top-ranked personnel on Google Cloud’s Delivery Readiness Portal (DRP)
Displays commitment to technical excellence by passing advanced delivery challenge labs and other advanced technical training
Demonstrates excellent knowledge and adoption of Google Cloud delivery enablement methods, assets, and offerings
Exhibits expertise through customer project and deployment experience
Marketing
Drives strategic programs and key events that address customer concerns and priorities
Works with cross-functional teams to ensure the success of campaigns and events
Takes a data-driven approach to marketing, investing resources and time in programs that drive the biggest impact
Always explores areas of opportunity to improve future work
Sales
Embodies commitment to the customer transformation journey
Consistently meets and exceeds sales targets
Aligns on goals to deliver amazing end-to-end customer experiences
Prioritizes long-term customer relationships over short-term sales
Solutions Engineering
Delivers superior customer experiences by keeping professional skills up to date, earning at least one Google technical certification
Embraces customer challenges head-on, taking responsibility for end-to-end solutioning
Works with purpose, providing deliverables in a timely manner without compromising quality
Works effectively across joint product areas, leveraging technology in innovative ways to address customer needs
Celebrating excellence in 2025
On behalf of the entire Google Cloud team, I want to extend a much-deserved congratulations to our 2025 Google Cloud Partner All-stars. Their commitment to innovation is an inspiration to us and a driving force of success to our customers.
Follow the celebration and engage with #PartnerAllstars on social media to learn more about these exceptional leaders.
Written by: Mohamed El-Banna, Daniel Lee, Mike Stokkel, Josh Goddard
Overview
Last year, Mandiant published a blog post highlighting suspected Iran-nexus espionage activity targeting the aerospace, aviation, and defense industries in the Middle East. In this follow-up post, Mandiant discusses additional tactics, techniques, and procedures (TTPs) observed in incidents Mandiant has responded to.
Since mid-2024, Mandiant has responded to targeted campaigns by the threat group UNC1549 against the aerospace, aviation and defense industries. To gain initial access into these environments, UNC1549 employed a dual approach: deploying well-crafted phishing campaigns designed to steal credentials or deliver malware and exploiting trusted connections with third-party suppliers and partners.
The latter technique is particularly strategic when targeting organizations with high security maturity, such as defense contractors. While these primary targets often invest heavily in robust defenses, their third-party partners may possess less stringent security postures. This disparity provides UNC1549 a path of lesser resistance, allowing them to circumvent the primary target’s main security controls by first compromising a connected entity.
Operating in late 2023 through 2025, UNC1549 employed sophisticated initial access vectors, including abuse of third-party relationships to gain entry (pivoting from service providers to their customers), VDI breakouts from third parties, and highly targeted, role-relevant phishing.
Once inside, the group leverages creative lateral movement techniques, such as stealing victim source code for spear-phishing campaigns that use lookalike domains to bypass proxies, and abusing internal service ticketing systems for credential access. They employ custom tooling, notably DCSYNCER.SLICK—a variant deployed via search order hijacking to conduct DCSync attacks.
UNC1549’s campaign is distinguished by its focus on anticipating investigators and ensuring long-term persistence after detection. They plant backdoors that beacon silently for months, only activating them to regain access after the victim has attempted eradication. They maintain stealth and command and control (C2) using extensive reverse SSH shells (which limit forensic evidence) and domains strategically mimicking the victim’s industry.
Threat Activity
Initial Compromise
A primary initial access vector employed by UNC1549 involved combining targeted social engineering with the exploitation of compromised third-party accounts. Leveraging credentials harvested from vendors, partners, or other trusted external entities, UNC1549 exploited legitimate access pathways inherent in these relationships.
Third-Party Services
Notably, the group frequently abused Citrix, VMWare, and Azure Virtual Desktop and Application services provided by victim organizations to third party partners, collaborators, and contractors. Utilizing compromised third-party credentials, they authenticated to the supplier’s infrastructure, establishing an initial foothold within the network perimeter. Post-authentication, UNC1549 used techniques designed to escape the security boundaries and restrictions of the virtualized Citrix session. This breakout granted them access to the underlying host system or adjacent network segments, and enabled the initiation of lateral movement activities deeper within the target corporate network.
Spear Phishing
UNC1549 utilized targeted spear-phishing emails as one of the methods to gain initial network access. These emails used lures related to job opportunities or recruitment efforts, aiming to trick recipients into downloading and running malware hidden in attachments or links. Figure 1 shows a sample phishing email sent to one of the victims.
Figure 1: Screenshot of a phishing email sent by UNC1549
Following a successful breach, Mandiant observed UNC1549 pivoting to spear-phishing campaigns specifically targeting IT staff and administrators. The goal of this campaign was to obtain credentials with higher permissions. To make these phishing attempts more believable, the attackers often perform reconnaissance first, such as reviewing older emails in already compromised inboxes for legitimate password reset requests or identifying the company’s internal password reset webpages, then crafted their malicious emails to mimic these authentic processes.
Establish Foothold
To maintain persistence within compromised networks, UNC1549 deployed several custom backdoors. Beyond MINIBIKE, which Mandiant discussed in the February 2024 blog post, the group also utilizes other custom malware such as TWOSTROKE and DEEPROOT. Significantly, Mandiant’s analysis revealed that while the malware used for initial targeting and compromises was not unique, every post-exploitation payload identified, regardless of family, had a unique hash. This included instances where multiple samples of the same backdoor variant were found within the same victim network. This approach highlights UNC1549’s sophistication and the considerable effort invested in customizing their tools to evade detection and complicate forensic investigations.
Search Order Hijacking
UNC1549 abused DLL search order hijacking to execute CRASHPAD, DCSYNCER.SLICK, GHOSTLINE, LIGHTRAIL, MINIBIKE, POLLBLEND, SIGHTGRAB, and TWOSTROKE payloads. Using the DLL search order hijacking techniques, UNC1549 achieved a persistent and stealthy way of executing their tooling.
Throughout the different investigations, UNC1549 demonstrated a comprehensive understanding of software dependencies by exploiting DLL search order hijacking in multiple software solutions. UNC1549 has deployed malicious binaries targeting legitimate Fortigate, VMWare, Citrix, Microsoft, and NVIDIA executables. In many cases, the threat actor installed the legitimate software after initial access in order to abuse SOH; however, in other cases, the attacker leveraged software that was already installed on victim systems and then replaced or added the malicious DLLs within the legitimate installation directory, typically with SYSTEM privileges.
TWOSTROKE
TWOSTROKE, a C++ backdoor, utilizes SSL-encrypted TCP/443 connections to communicate with its controllers. This malware possesses a diverse command set, allowing for system information collection, DLL loading, file manipulation, and persistence. While showing some similarities to MINIBIKE, it’s considered a unique backdoor.
Upon execution of TWOSTROKE, it employs a specific routine to generate a unique victim identifier. TWOSTRIKE retrieves the fully qualified DNS computer name using the Windows API function GetComputerNameExW(ComputerNameDnsFullyQualified). This retrieved name then undergoes an XOR encryption process, utilizing the static key. Following the encryption, the resulting binary data is converted into a lowercase hexadecimal string.
Finally, TWOSTROKE extracts the first eight characters of this hexadecimal string, reverses it, and uses it as the victim’s unique bot ID for later communication with the C2 server.
Functionalities
After sending the check in request to the C2 server, the TWOSTROKE C2 server returns with a hex-encoded payload that contains multiple values separated by “@##@.” Depending on the received command, TWOSTROKE can execute one of the following commands:
1: Upload a file to the C2
2: Execute a file or a shell command
3: DLL execution into memory
4: Download file from the C2
5: Get the full victim user name
6: Get the full victim machine name
7: List a directory
8: Delete a file
LIGHTRAIL
UNC1549 was observed downloading a ZIP file from attacker-owned infrastructure. This ZIP file contained the LIGHTRAIL tunneler asVGAuth.dll and was executed through search order hijacking using the VGAuthCLI.exe executable. LIGHTRAIL is a custom tunneler, likely based on the open-source Socks4a proxy, Lastenzug, that communicates using Azure cloud infrastructure.
There are several distinct differences between the LIGHTRAIL sample and the LastenZug source code. These include:
Increasing the MAX_CONNECTIONS from 250 to 5000
Static configuration inside the lastenzug function (wPath and port)
No support for using a proxy server when connecting to the WebSocket C2
Compiler optimizations reducing the number of functions (26 to 10)
Additionally, LastenZug is using hashing for DLLs and API function resolving. By default, the hash value is XOR’d with the value 0x41507712, while the XOR value in the observed LIGHTRAIL sample differs from the original source code – 0x41424344(‘ABCD’).
After loading the necessary API function pointers, the initialization continues by populating the server name (wServerName), the port, and URI (wPath) values. The port is hardcoded at 443 (for HTTPS) and the path is hardcoded to “/news.” This differs from the source code where these values are input parameters to the lastenzug function.
The initWSfunction is responsible for establishing the WebSocket connection, which it does using the Windows WinHTTP API. The initWSfunction has a hard-coded User-Agent string which it constructs as a stack string:
Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10136
Mandiant identified another LIGHTRAIL sample uploaded to VirusTotal from Germany. However, this sample seems to have been modified by the uploader as the C2 domain was intentionally altered.
GET https://aaaaaaaaaaaaaaaaaa.bbbbbb.cccccccc.ddddd.com/page HTTP/1.1
Host: aaaaaaaaaaaaaaaaaa.bbbbbb.cccccccc.ddddd.com
Connection: Upgrade
Upgrade: websocket
User-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.37 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10136
Sec-WebSocket-Key: 9MeEoJ3sjbWAEed52LdRdg==
Sec-WebSocket-Version: 13
Figure 2: Modified LIGHTRAIL network communication snippet
Most notable is that this sample is using a different URL path for its communication, but also the User-Agent in this sample is different from the one that was observed in previous LIGHTRAIL samples and the LastenZug source code.
DEEPROOT
DEEPROOT is a Linux backdoor written in Golang and supports the following functionalities: shell command execution, system information enumeration and file listing, delete, upload, and download. DEEPROOT was compiled to be operating on Linux systems; however, due to Golang’s architecture DEEPROOT could also be compiled for other operating systems. At the time of writing, Mandiant has not observed any DEEPROOT samples targeting Windows systems.
DEEPROOT was observed using multiple C2 domains hosted in Microsoft Azure. The observed DEEPROOT samples used multiple C2 servers per binary, suspected to be used for redundancy in case one C2 server has been taken down.
Functionalities
After sending the check in request to the C2 server, the DEEPROOT C2 server returns with a hex-encoded payload that contains multiple values separated by ‘-===-’
sleep_timeout is the time in milli-seconds to wait before making the next request.
command_id is an identifier for the C2 command, used by the backdoor when responding to the C2 with the result.
command is the command number and it’s one of the following:
1 – Get directory information (directory listing), the directory path is received in argument_1.
2 – Delete a file, the file path is received in argument_1.
3 – Get the victim username.
4 – Get the victim’s hostname.
5 – Execute a shell command, the shell command is received in argument_1.
6 – Download a file from the C2, the C2 file path is received in argument_1 and the local file path is received in argument_2.
7 – Upload a file to the C2, the local file path is received in argument_1.
argument_1 and argument_2 are the command arguments and it is optional.
GHOSTLINE
GHOSTLINE is a Windows tunneler utility written in Golang that uses a hard-coded domain for its communication. GHOSTLINE uses the go-yamux library for its network connection.
POLLBLEND
POLLBLEND is a Windows tunneler that is written in C++. Earlier iterations of POLLBLEND featured multiple hardcoded C2 servers and utilized two hardcoded URI parameters for self-registration and tunneler configuration download. For the registration of the machine, POLLBLEND would reach out to/register/ and sent a HTTP POST request with the following JSON body.
{"username": "<computer_name>"}
Figure 4: POLLBLEND body data
Code Signing
Throughout the tracking of UNC1549’s activity across multiple intrusions, the Iranian-backed threat group was observed signing some of their backdoor binaries with legitimate code-signing certificates—a tactic also covered by Check Point—likely to help their malware evade detection and bypass security controls like application allowlists, which are often configured to trust digitally signed code. The group employed this technique to weaponize malware samples, including variants for GHOSTLINE, POLLBLEND, and TWOSTROKE. All identified code-signing certificates have been reported to the relevant issuing Certificate Authorities for revocation.
Escalate Privileges
UNC1549 has been observed using a variety of techniques and custom tools aimed at stealing credentials and gathering sensitive data post-compromise. This included a utility, tracked as DCSYNCER.SLICK, designed to mimic the DCSync Active Directory replication feature. DCSync is a legitimate function domain controllers use for replicating changes via RPC. This allowed the attackers to extract NTLM password hashes directly from the domain controllers. Another tool, dubbed CRASHPAD, focused on extracting credentials saved within web browsers. For visual data collection, they deployed SIGHTGRAB, a tool capable of taking periodic screenshots, potentially capturing sensitive information displayed on the user’s screen. Additionally, UNC1549 utilized simpler methods, such as deploying TRUSTTRAP, which presented fake popup windows prompting users to enter their credentials, which were then harvested by the attackers.
UNC1549 frequently used DCSync attacks to obtain NTLM password hashes for domain users, which they then cracked in order to facilitate lateral movement and privilege escalation. To gain the necessary directory replication rights for DCSync, the threat actor employed several methods. They were observed unconventionally resetting passwords for domain controller computer accounts using net.exe. This action typically broke the domain controller functionality of the host and caused an outage, yet it successfully enabled them to perform the DCSync operation and extract sensitive credentials, including those for domain administrators and Azure AD Connect accounts. UNC1549 leveraged other techniques to gain domain replication rights, including creating rogue computer accounts and abusing Resource-Based Constrained Delegation (RBCD) assignments. They also performed Kerberoasting, utilizing obfuscated Invoke-Kerberoast scripts, for credential theft.
net user DC-01$ P@ssw0rd
Figure 5: Example of an UNC1549 net.exe command to reset a domain controller computer account
In some cases, shortly after gaining a foothold on workstations, UNC1549 discovered vulnerable Active Directory Certificate Services templates. They used these to request certificates, allowing them to impersonate higher-privileged user accounts.
UNC1549 also frequently targeted saved credentials within web browsers, either through malicious utilities or by RDP session hijacking. In the latter, the threat actor would identify which user was logged onto a system through quser.exe or wmic.exe, and then RDP to that system with the user’s account to gain access to their active and unlocked web browser sessions.
DCSYNCER.SLICK
DCSYNCER.SLICK is a Windows executable that is based on the Open source Project DCSyncer and is based on Mimikatz source code. DCSYNCER.SLICK has been modified to use Dynamic API resolution and has all its printf statements removed.
Additionally, DCSYNCER.SLICK collects and XOR-encrypts the credentials before writing them to a hardcoded filename and path. The following hardcoded filenames and paths were observed being used by DCSYNCER.SLICK:
To evade detection, UNC1549 executed the malware within the context of a compromised domain controller computer account. They achieved this compromise by manually resetting the account password. Instead of utilizing the standardnetdomcommand, UNC1549 used the Windows commandnet user <computer_name> <password>. Subsequently, they used these newly acquired credentials to execute the DCSYNCER.SLICK payload. This tactic would give the false impression that replication had occurred between two legitimate domain controllers.
CRASHPAD
CRASHPAD is a Windows executable that is written in C++ that decrypts the content of the file config.txtinto the file crash.logby impersonating the explorer.exe user privilege and through the CryptUnprotectDataAPI.
The contents of these files could not be determined because UNC1549 deleted the output after CRASHPAD was executed.
The CRASHPAD configuration and output file paths were hardcoded into the sample, similar to the LOG.txt filename found in the DCSYNCER.SLICK binary.
SIGHTGRAB
SIGHTGRAB is a Windows executable written in C that autonomously captures screen shots at regular intervals and saves them to disk. Upon execution SIGHTGRAB loads several Windows libraries dynamically at runtime including User32.dll, Gdi32.dll, and Ole32.dll. SIGHTGRAB implements runtime API resolution through LoadLibraryA and GetProcAddress calls with encoded strings to access system functions. SIGHTGRAB uses XOR encryption with a single-byte key of 0x41 to decode API function names.
SIGHTGRAB retrieves the current timestamp and uses string interpolation of YYYY-MM-DD-HH-MM on the timestamp to generate the directory name. In this newly created directory, SIGHTGRAB saves all the taken screenshots incrementally.
Figure 6: Examples of screenshot files created by SIGHTGRAB on disk
Mandiant observed UNC1549 strategically deploy SIGHTGRAB on workstations to target users in two categories: those handling sensitive data, allowing for subsequent data exposure and exfiltration, and those with privileged access, enabling privilege escalation and access to restricted systems.
TRUSTTRAP
A malware that serves a Windows prompt to trick the user into submitting their credentials. The captured credentials are saved in cleartext to a file. Figure 7 shows a sample popup by TRUSTTRAP mimicking the Microsoft Outlook login window.
Figure 7: Screenshot showing the fake Microsoft Outlook login window
TRUSTTRAP has been used by UNC1549 since at least 2023 for obtaining user credentials used for lateral movement.
Reconnaissance and Lateral Movement
For internal reconnaissance, UNC1549 leveraged legitimate tools and publicly available utilities, likely to blend in with standard administrative activities. AD Explorer, a valid executable signed by Microsoft, was used to query Active Directory and inspect its configuration details. Alongside this, the group employed native Windows commands like net user and net group to enumerate specific user accounts and group memberships within the domain, and PowerShell scripts for ping and port scanning reconnaissance on specific subnets, typically those associated with privileged servers or IT administrator workstations
UNC1549 uses a wide variety of methods for lateral movement, depending on restrictions within the victim environment. Most frequently, RDP was used. Mandiant also observed the use of PowerShell Remoting, Atelier Web Remote Commander (“AWRC”), and SCCM remote control, including execution of variants of SCCMVNC to enable SCCM remote control on systems.
Atelier Web Remote Commander
Atelier Web Remote Commander (AWRC) is a commercial utility for remotely managing, auditing, and supporting Windows systems. Its key distinction is its agentless design, meaning it requires no software installation or pre-configuration on the remote machine, enabling administrators to connect immediately.
Leveraging the capabilities of AWRC, UNC1549 utilized this publicly available commercial tool to facilitate post-compromise activities. These activities included:
Established remote connections: Used AWRC to connect remotely to targeted hosts within the compromised network
Conducted reconnaissance: Employed AWRC’s built-in functions to gather information by:
Enumerating running services
Enumerating active processes
Enumerating existing RDP sessions
Stole credentials: Exploited AWRC to exfiltrate sensitive browser files known to contain stored user credentials from remote systems
Deployed malware: Used AWRC as a vector to transfer and deploy malware onto compromised machines
SCCMVNC
SCCMVNC is a tool designed to leverage the existing Remote Control feature within Microsoft System Center Configuration Manager (SCCM/ConfigMgr) to achieve a VNC-like remote access experience without requiring additional third-party modules or user consent/notifications.
SCCM.exe reconfig /target:[REDACTED]
Figure 8: Example of an UNC1549 executing SCCMVNC command
The core functionality of SCCMVNC lies in its ability to manipulate the existing Remote Control feature of SCCM. Instead of deploying a separate VNC server or other remote access software, the tool directly interacts with and reconfigures the settings of the native SCCM Remote Control service on a client workstation. This approach leverages an already present and trusted component within the enterprise environment.
A key aspect of SCCMVNC is its capacity to bypass the standard consent and notification mechanisms typically associated with SCCM Remote Control. Normally, when an SCCM remote control session is initiated, the end-user is prompted for permission, and various notification icons or connection bars are displayed. SCCMVNC effectively reconfigures the underlying SCCM settings (primarily through WMI interactions) to disable these user-facing requirements. This alteration allows for a significantly more discreet and seamless remote access experience, akin to what one might expect from a VNC connection where the user might not be immediately aware of the ongoing session.
Command and Control
UNC1549 continued to use Microsoft Azure Web Apps registrations and cloud infrastructure for C2. In addition to backdoors including MINIBUS, MINIBIKE, and TWOSTROKE, UNC1549 relied heavily on SSH reverse tunnels established on compromised systems to forward traffic from their C2 servers to compromised systems. This technique limited the availability of host-based artifacts during investigations, since security telemetry would only record network connections. For example, during data collection from SMB shares, outbound connections were observed from the SSH processes to port 445 on remote systems, but the actual data collected could not be confirmed due to no staging taking place within the victim environment, and object auditing being disabled.
Figure 9: Example of an UNC1549 reverse SSH command
Mandiant also identified evidence of UNC1549 deploying a variety of redundant remote access methods, including ZEROTIER and NGROK. In some instances, these alternative methods weren’t used by the threat actor until victim organizations had performed remediation actions, suggesting they are primarily deployed to retain access.
Complete Mission
Espionage
UNC1549’s operations appear strongly motivated by espionage, with mission objectives centering around extensive data collection from targeted networks. The group actively seeks sensitive information, including network/IT documentation, intellectual property, and emails. Furthermore, UNC1549 often leverages compromised organizations as a pivot point, using their access to target other entities, particularly those within the same industry sector, effectively conducting third-party supplier and partner intrusions to further their intelligence-gathering goals.
Notably, Mandiant responded to one intrusion at an organization in an unrelated sector, and assessed that the intrusion was opportunistic due to the initial spear phishing lure being related to a job at an aerospace and defense organization. This demonstrated UNC1549’s ability to commit resources to expanding access and persistence in victim organizations that don’t immediately meet traditional espionage goals.
Defense Evasion
UNC1549 frequently deleted utilities from compromised systems after execution to avoid detection and hinder investigation efforts. The deletion of forensic artifacts, including RDP connection history registry keys, was also observed. Additionally, as described earlier, the group repeatedly used SSH reverse tunnels from victim hosts back to their infrastructure, a technique which helped hide their activity from EDR agents installed on those systems. Combined, this activity demonstrated an increase in the operational security of UNC1549 over the past year.
reg delete "HKEY_CURRENT_USERSoftwareMicrosoftTerminal Server ClientDefault" /va /f
reg delete "HKEY_CURRENT_USERSoftwareMicrosoftTerminal Server ClientServers" /f
Figure 10: Examples of UNC1549 commands to delete RDP connection history registry keys
Acknowledgement
This analysis would not have been possible without the assistance from across Google Threat Intelligence Group, Mandiant Consulting and FLARE. We would like to specifically thank Greg Sinclair and Mustafa Nasser from FLARE, and Melissa Derr, Liam Smith, Chris Eastwood, Alex Pietz, Ross Inman, and Emeka Agu from Mandiant Consulting.
MITRE ATT&CK
TACTIC
ID
Name
Description
Collection
T1213.002
Data from Information Repositories: SharePoint
UNC1549 browsed Microsoft Teams and SharePoint to download files used for extortion.
Collection
T1113
Screen Capture
UNC1549 was observed making screenshots from sensitive data.
Reconnaissance
T16561598.003
Phishing for Information
UNC1549 used third party vendor accounts to obtain privileged accounts using a Password Reset portal theme.
Credential Access
T1110.003
Brute Force: Password Spraying
UNC1549 was observed performing password spray attacks against the Domain.
Credential Access
T1003.006
OS Credential Dumping: DCSync
UNC1549 was observed using DCSYNCER.SLICK to perform DCSync on domain controller level.
Defense Evasion
T1574.001
Hijack Execution Flow: DLL Search Order Hijacking
UNC1549 was observed using Search Order Hijacking to execute both LIGHTRAIL and DCSYNCER.SLICK.
Initial Access
T1078
Valid Accounts
UNC1549 used valid compromised accounts to gain initial access
Initial Access
T1199
Trusted Relationship
UNC1549 used trusted third party vendor accounts for both initial access and lateral movement.
Google SecOps customers receive robust detection for UNC1549 TTPs through curated threat intelligence from Mandiant and Google Threat Intelligence. This frontline intelligence is operationalized within the platform as custom detection signatures and advanced YARA-L rules.
We’re excited to launch the Production-Ready AI with Google Cloud Learning Path, a free series designed to take your AI projects from prototype to production.
This page is the central hub for the curriculum. We’ll be updating it weekly with new modules from now through mid-December.
Why We Built This: Bridging the Prototype-to-Production Gap
Generative AI makes it easy to build an impressive prototype. But moving from that proof-of-concept to a secure, scalable, and observable production system is where many projects stall. This is the prototype-to-production gap. It’s the challenge of answering hard questions about security, infrastructure, and monitoring for a system that now includes a probabilistic model.
It’s a journey we’ve been on with our own teams at Google Cloud. To solve for this ongoing challenge, we built a comprehensive internal playbook focused on production-grade best practices. After seeing the playbook’s success, we knew we had to share it.
We’re excited to share this curriculum with the developer community. Share your progress and connect with others on the journey using the hashtag #ProductionReadyAI. Happy learning!
The Curriculum
Module 1: Developing Apps that use LLMs
Start with the fundamentals of building applications and interacting with models using the Vertex AI SDK.
The landscape of generative AI is shifting. While proprietary APIs are powerful, there is a growing demand for open models—models where the architecture and weights are publicly available. This shift puts control back in the hands of developers, offering transparency, data privacy, and the ability to fine-tune for specific use cases.
To help you navigate this landscape, we are releasing two new hands-on labs featuring Gemma 3, Google’s latest family of lightweight, state-of-the-art open models.
Why Gemma?
Built from the same research and technology as Gemini, Gemma models are designed for responsible AI development. Gemma 3 is particularly exciting because it offers multimodal capabilities (text and image) and fits efficiently on smaller hardware footprints while delivering massive performance.
But running a model on your laptop is very different from running it in production. You need scale, reliability, and hardware acceleration (GPUs). The question is: Where should you deploy?
Best for: Developers who want an API up and running instantly without managing infrastructure, scaling to zero when not in use.
If your priority is simplicity and cost-efficiency for stateless workloads, Cloud Run is your answer. It abstracts away the server management entirely. With the recent addition of GPU support on Cloud Run, you can now serve modern LLMs without provisioning a cluster.
aside_block
<ListValue: [StructValue([(‘title’, ‘Start the lab!’), (‘body’, <wagtail.rich_text.RichText object at 0x7f1d25d64040>), (‘btn_text’, ”), (‘href’, ”), (‘image’, None)])]>
Path 2: The Platform Approach (GKE)
Best for: Engineering teams building complex AI platforms, requiring high throughput, custom orchestration, or integration with a broader microservices ecosystem.
When your application graduates from a prototype to a high-traffic production system, you need the control of Kubernetes. GKE Autopilot gives you that power while still handling the heavy lifting of node management. This path creates a seamless journey from local testing to cloud production.
aside_block
<ListValue: [StructValue([(‘title’, ‘Start the lab!’), (‘body’, <wagtail.rich_text.RichText object at 0x7f1d25d64d30>), (‘btn_text’, ”), (‘href’, ”), (‘image’, None)])]>
Which Path Will You Choose?
Whether you are looking for the serverless simplicity of Cloud Run or the robust orchestration of GKE, Google Cloud provides the tools to take Gemma 3 from a concept to a deployed application.
Cloud infrastructure reliability is foundational, yet even the most sophisticated global networks can suffer from a critical issue: slow or failed recovery from routing outages. In massive, planetary-scale networks like Google’s, router failures or complex, hidden conditions can prevent traditional routing protocols from restoring service quickly, or sometimes at all. These brief but costly outages — what we call slow convergence or convergence failure — critically disrupt real-time applications with low tolerance to packet loss and, most acutely, today’s massive, sensitive AI/ML training jobs, where a brief network hiccup can waste millions of dollars in compute time.
To solve this problem, we pioneered Protective ReRoute (PRR), a radical shift that moves the responsibility for rapid failure recovery from the centralized network core to the distributed endpoints themselves. Since putting it into production over five years ago, this host-based mechanism has dramatically increased Google’s network’s resilience, proving effective in recovering from up to 84%1 of inter-data-center outages that would have been caused by slow convergence events. Google Cloud customers with workloads that are sensitive to packet loss can also enable it in their environments — read on to learn more.
The limits of in-network recovery
Traditional routing protocols are essential for network operation, but they are often not fast enough to meet the demands of modern, real-time workloads. When a router or link fails, the network must recalculate all affected routes, which is known as reconvergence. In a network the size of Google’s, this process can be complicated by the scale of the topology, leading to delays that range from many seconds to minutes. For distributed AI training jobs with their wide, fan-out communication patterns, even a few seconds of packet loss can lead to application failure and costly restarts. The problem is a matter of scale: as the network grows, the likelihood of these complex failure scenarios increases.
Protective ReRoute: A host-based solution
Protective ReRoute is a simple, effective concept: empower the communicating endpoints (the hosts) to detect a failure and intelligently re-steer traffic to a healthy, parallel path. Instead of waiting for a global network update, PRR capitalizes on the rich path diversity built into our network. The host detects packet loss or high latency on its current path, and then immediately initiates a path change by modifying carefully chosen packet header fields, which tells the network to use an alternate, pre-existing path.
This architecture represents a fundamental shift in network reliability thinking. Traditional networks rely on a combination of parallel and series reliability. Serialization of components tends to reduce the reliability of a system; in a large-diameter network with multiple forwarding stages, reliability degrades as the diameter increases. In other words, every forwarding stage affects the whole system. Even if a network stage is designed with parallel reliability, it creates a serial impact on the overall network while the parallel stage reconverges. By adding PRR at the edges, we treat the network as a highly parallel system of paths that appear as a single stage, where the overall reliability increases as the number of available paths grows exponentially, effectively circumventing the serialization effects of slow network convergence in a large-diameter network. The following diagram contrasts the system reliability model for a PRR-enabled network with that of a traditional network. Traditional network reliability is in inverse proportion to the number of forwarding stages; with PRR the reliability of the same network is in direct proportion to the number of composite paths, which is exponentially proportional to the network diameter.
How Protective ReRoute works
The PRR mechanism has three core functional components:
End-to-end failure detection: Communicating hosts continuously monitor path health. On Linux systems, the standard mechanism uses TCP retransmission timeout (RTO) to signal a potential failure. The time to detect a failure is generally a single-digit multiple of the network’s round-trip time (RTT). There are also other methods for end-to-end failure detection that have varying speed and cost.
Packet-header modification at the host: Once a failure is detected, the transmitting host modifies a packet-header field to influence the forwarding path. To achieve this, Google pioneered and contributed the mechanism that modifies the IPv6 flow-label in the Linux kernel (version 4.20+). Crucially, the Google software-defined network (SDN) layer provides protection for IPv4 traffic and non-Linux hosts as well by performing the detection and repathing on the outer headers of the network overlay.
PRR-aware forwarding: Routers and switches in the multipath network respect this header modification and forward the packet onto a different, available path that bypasses the failed component.
Proof of impact
PRR is not theoretical; it is a continuously deployed, 24×7 system that protects production traffic worldwide. Its impact is compelling: PRR has been shown to reduce network downtime caused by slow convergence and convergence failures by up to the above-mentioned 84%. This means that up to 8 out of every 10 network outages that would have been caused by a router failure or slow network-level recovery are now avoided by the host. Furthermore, host-initiated recovery is extremely fast, often resolving the problem in a single-digit multiple of the RTT, which is vastly faster than traditional network reconvergence times.
Key use cases for ultra-reliable networking
The need for PRR is growing, driven by modern application requirements:
AI/ML training and inference: Large-scale workloads, particularly those distributed across many accelerators (GPUs/TPUs), are uniquely sensitive to network reliability. PRR provides the ultra-reliable data distribution necessary to keep these high-value compute jobs running without disruption.
Data integrity and storage: Significant numbers of dropped packets can result in data corruption and data loss, not just reduced throughput. By reducing the outage window, PRR improves application performance and helps guarantee data integrity.
Real-time applications: Applications like gaming and services like video conferencing and voice calls are intolerant of even brief connectivity outages. PRR reduces the recovery time for network failures to meet these strict real-time requirements.
Frequent short-lived connections: Applications that rely on a large number of very frequent short-lived connections can fail when the network is unavailable for even a short time. By reducing the expected outage window, PRR helps these applications reliably complete their required connections.
Activating Protective ReRoute for your applications
The architectural shift to host-based reliability is an accessible technology for Google Cloud customers. The core mechanism is open and part of the mainline Linux kernel (version 4.20 and later).
You can benefit from PRR in two primary ways:
Hypervisor mode: PRR automatically protects traffic running across Google data centers without requiring any guest OS changes. Hypervisor mode provides recovery in the single digit seconds for traffic of moderate fanout in specific areas of the network.
Guest mode: For critical, performance-sensitive applications with high fan-out and in any segment of the network, you can opt into guest-mode PRR, whichenables the fastest possible recovery time and greatest control. This is the optimal setting for demanding mission-critical applications, AI/ML jobs, and other latency-sensitive services.
To activate guest-mode PRR for critical applications follow the guidance in the documentation and be ready to ensure the following:
Your VM runs a modern Linux kernel (4.20+).
Your applications use TCP.
The application traffic uses IPv6. For IPv4 protection, the application needs to use the gVNIC driver.
Get started
The availability of Protective ReRoute has profound implications for a variety of Google and Google Cloud users.
For cloud customers with critical workloads: Evaluate and enable guest-mode PRR for applications that are sensitive to packet loss and that require the fastest recovery time, such as large-scale AI/ML jobs or real-time services.
For network architects: Re-evaluate your network reliability architectures. Consider the benefits of designing for rich path diversity and empowering endpoints to intelligently route around failures, shifting your model from series to parallel reliability.
For the open-source community: Recognize the power of host-level networking innovations. Contribute to and advocate for similar reliability features across all major operating systems to create a more resilient internet for everyone.
With the pace of scientific discovery moving faster than ever, we’re excited to join the supercomputing community as it gets ready for its annual flagship event, SC25, in St. Louis from November 16-21, 2025. There, we’ll share how Google Cloud is poised to help with our lineup of HPC and AI technologies and innovations, helping researchers, scientists, and engineers solve some of humanity’s biggest challenges.
Redefining supercomputing with cloud-native HPC
Supercomputers are evolving from a rigid, capital-intensive resource into an adaptable, scalable service. To go from “HPC in the cloud” to “cloud-native HPC,” we leverage core principles of automation and elastic infrastructure to fundamentally change how you consume HPC resources, allowing you to spin up purpose-built clusters in minutes with the exact resources you need.
This cloud-native model is very flexible. You can augment an on-premises cluster to meet peak demand or build a cloud-native system tailored with the right mix of hardware for your specific problem — be it the latest CPUs, GPUs, or TPUs. With this approach, we’re democratizing HPC, putting world-class capabilities into the hands of startups, academics, labs, and enterprise teams alike.
Key highlights at SC25:
Next-generation infrastructure: We’ll be showcasing our latest H4D VMs, powered by 5th generation AMD EPYC processors and featuring Cloud RDMA for low-latency networking. You’ll also see our latest accelerated compute resources including A4X and A4X Max VMs featuring the latest NVIDIA GPUs with RDMA.
Powering your essential applications: Run your most demanding simulations at massive scale — from Computational Fluid Dynamics (CFD) with Ansys, to Computer-Aided Engineering with Siemens, computational chemistry with Schrodinger, and risk modeling in FSI.
Dynamic Workload Scheduler: Discover how Dynamic Workload Scheduler and its innovative Flex Start mode, integrated with familiar schedulers like Slurm, is reshaping HPC consumption. Move beyond static queues toward flexible, cost-effective, and efficient access to high-demand compute resources.
Easier HPC with Cluster Toolkit: Learn how Cluster Toolkit can help you deploy a supercomputer-scale cluster with less than 50 lines of code.
High-throughput, scalable storage: Get a deep dive into Google Cloud Managed Lustre, a fully managed, high-performance parallel file system that can handle your most demanding HPC and AI workloads.
Hybrid for the enterprise: For our enterprise customers, especially in financial services, we’re enabling hybrid cloud with IBM Spectrum Symphony Connectors, allowing you to migrate or burst workloads to Google Cloud and reduce time-to-solution.
AI-powered scientific discovery
There’s a powerful synergy between HPC and AI — where HPC builds more powerful AI, and AI makes HPC faster and more insightful. This complementary relationship is fundamentally changing how research is done, accelerating discovery in everything from drug development and climate modeling to new materials and engineering. At Google Cloud, we’re at the forefront of this transformation, building the models, tools, and platforms that make it possible.
What to look for:
AI for scientific productivity: We’ll be showcasing Google’s suite of AI tools designed to enhance the entire research lifecycle. From Idea Generation agent to Gemini Code Assist with Gemini Enterprise, you’ll see how AI can augment your capabilities and accelerate discovery.
AI-powered scientific applications: Learn about the latest advancements in our AI-powered scientific applications including AlphaFold 3 and Weather Next
The power of TPUs: Explore Google’s TPUs, including the latest seventh-generation Ironwood model, and discover how they can enhance AI workload performance and efficiency.
Join the Google Cloud at SC25: At Google Cloud, we believe the cloud is the supercomputer of the future. From purpose-built HPC and AI infrastructure to quantum breakthroughs and simplified open-source tools, let Google Cloud be the platform for your next discovery.
We invite you to connect with our experts and learn more. Join the Google Cloud Advanced Computing Community to engage in discussions with our partners and the broader HPC, AI, and quantum communities.
We can’t wait to see what you discover.
See us at the show:
Visit us in booth #3724: Stop by for live demos of our latest HPC and AI solutions, including Dynamic Workload Scheduler, Cluster Toolkit, our latest AI agents, and even see our TPUs. Our team of experts will be on hand to answer your questions and discuss how Google Cloud can meet your needs.
Attend our technical talks: Keep an eye on our SC25 schedule for Google Cloud presentations and technical talks, where our leaders and partners will share deep dives, insights, and best practices.
Passport program: Grab a passport card from the Google booth and visit our demos, labs, and talks to collect stamps and learn about how we’re working with organizations across the HPC ecosystem to democratize HPC. Come back to the Google booth with your completed passport card to choose your prize!
Play a game: Join us in the Google booth and at our events to enjoy some Gemini-driven games — test your tech trivia knowledge or compete head-to-head with others to build the best LEGO creation!
Join our community kickoff: Are you a member of the Google Cloud Advanced Computing Community? Secure your spot today for our SC25 Kickoff Happy Hour!
Celebrate with NVIDIA and Google Cloud: We’re proud to co-host a reception with NVIDIA, and we look forward to toasting another year of innovation with our customers and partners. Register today to secure your spot!
Editor’s note: The post is part of a series that highlights how organizations leverage Google Cloud’s unique data science capabilities over alternative cloud data platforms. Google Cloud’s vector embedding generation and search features are unique for their end-to-end, customizable platform that leverages Google’s advanced AI research, offering features like task-optimized embedding models and hybrid search to deliver highly relevant results for both semantic and keyword-based queries.
Zeotap’s customer intelligence platform (CIP) helps brands understand their customers and predict behaviors, so that they can improve customer engagement. Zeotap partners with Google Cloud to build a customer data platform that offers privacy, security, and compliance. Zeotap CIP, built with BigQuery, enables digital marketers to build and use AI/ML models to predict customer behavior and personalize the customer experienc
The Zeotap platform includes a customer segmentation feature called lookalike audience extensions. A lookalike audience is a group of new potential customers identified by machine learning algorithms who share similar characteristics and behaviors with an existing, high-value customer base. However, sparse or incomplete first-party data can make it hard to create effective lookalike audiences, preventing advertising algorithms from accurately identifying the key characteristics of valuable customers that they need to find similar new prospects. For such rare features, Zeotap uses multiple machine learning (ML) methodologies that combine Zeotap’s multigraph algorithm and high-quality data assets to more accurately extend customers’ audiences between the CDP and lookalike models.
In this blog, we dive into how Zeotap uses BigQuery, including BigQuery ML and Vector Search to solve the end-to-end lookalike problem. By taking a practical approach, we transformed a complex nearest-neighbour problem into a simple inner-join problem, overcoming challenges of cost, scale and performance without a specialized vector database. We break down each step of the workflow, from data preparation to serving, highlighting how BigQuery addresses core challenges along the way. We illustrate one of the techniques, Jaccard similarity with embeddings, to address the low-cardinality categorical columns that dominate user-profile datasets.
The high-level flow is as follows, and happens entirely within the BigQuery ecosystem. Note: In this blog, we will not be covering the flow of high-cardinality columns.
Jaccard similarity
Among a couple of other similarity indexes, which return the most similar vector that are closest in embedding space, Zeotap prefers the Jaccard similarity to be a fitting index for low-cardinality features, which is a measure of overlap between two sets with a simple formula: (A B) / (AB). The Jaccard similarity answers the question, “Of all the unique attributes present in either of the two users, what percentage of them are shared?” It only cares about the features that are present in at least one of the entities (e.g., the 1s in a binary vector) and ignores attributes that are absent in both.
Jaccard similarity shines because it is simple and easily explainable over many other complex distance metrics and similarity indexes that only measure distance in the embeddings space — a real Occam’s razor, as it were.
Implementation blueprint
Generating the vector embeddings After selecting the low-cardinality features, we create our vectors using BigQuery one-hot encoding andmulti-hot encoding for primitive and array-based columns.
Again, it helps to visualize a sample vector table:
Challenge: Jaccard distance is not directly supported in BigQuery vector search!
BigQuery vector search supports three distance types: Euclidean, Cosine and Dot product, but not Jaccard distance — at least not natively. However, we can represent the choice of binary vectors where the Jaccard Distance (1 – Jaccard Similarity) as:
Jd(A,B) = 1 – |A∩B|/|A∪B| = (|A∪B| – |A∩B|)/|A∪B|
Using only the dot product, this can be rewritten as:
So we can, in fact, arrive at the Jaccard distance using the dot product. We found BigQuery’s out-of-the-box LP_NORM function for calculating theManhattan norm useful, as the Manhattan norm for a binary vector is the dot product with itself. In other words, using the Manhattan norm function, we found that we can support the Jaccard distance in a way that it can be calculated using the supported “dot product” search in BigQuery.
Building the vector index
Next, we needed to build our vector index. BigQuery supports two primary vector index types: IVF (Inverted File Index) and TREE_AH (Tree with Asymmetric Hashing), each tailored to different scenarios. The TREE_AH vector index type combines a tree-like structure with asymmetric hashing (AH), based onGoogle’s ScaNN algorithm, which has performed exceptionally well on variousANN benchmarks. Also, since the use case was for large batch queries (e.g., hundreds of thousands to millions of users), this offered reduced latency and cost compared to alternate vector databases.
Lookalike delivery
Once we had a vector index to optimize searches, we asked ourselves, “Should we run our searches directly using the VECTOR_SEARCH function in BigQuery?” Taking this approach over the base table yielded a whopping 118 million user-encoded vectors for just one client! Additionally, and most importantly, since this computation called for a Cartesian product, our in-memory data sizes became very large and complex quickly. We needed to devise a strategy that would scale to all customers.
The rare feature strategy
A simple but super-effective strategy is to avoid searching for ubiquitous user features. In a two-step rare-feature process, we identify the “omnipresent” features, then proceed to create a signal-rich table that includes users who possess at least one of the rarer/discriminative features. Right off the bat, we achieved up to 78% reduction in search space. BigQuery VECTOR_SEARCH allows you to do this with pre-filtering, wherein you use a subquery to dynamically shrink the search space. The catch is that the subquery cannot be a classic join, so we introduce a “flag” column and make it part of the index. Note: If a column is not stored in the index, then the WHERE clause in the VECTOR_SEARCH will execute a post-filter.
Use the BQUI or system tables to see if a vector is used to accelerate queries
Batch strategy
Vector search compares query users (N, the users we’re targeting) against base users (M, the total user pool, in this case 118M). The complexity increases with (M × N), making large-scale searches resource-intensive. To manage this, we applied batches to the N query users, processing them in groups (e.g., 500,000 per batch), while M remained the full base set. This approach reduced the computational load, helping to efficiently match the top 100 similar users for each query user.We then used grid search to determine the optimal batch size for high-scale requirements.
To summarize
We partnered with Google Cloud to enable digital marketers to build and use AI/ML models for customer segmentation and personalized experiences, driving higher conversion rates and lower acquisition costs. We addressed the challenge of Jaccard distance not being directly supported in BigQuery Vector Search by using the dot product and Manhattan norm. This practical approach, leveraging BigQuery ML and vector offerings, allowed us to create bespoke lookalike models with just one single SQL script and overcome challenges of cost, scale, and performance without a specialized vector database.
Using BigQuery ML and vector offerings, coupled with its robust, serverless architecture, we were able to release bespoke lookalike models catering to individual customer domains and needs. Together, Zeotap and Google Cloud look forward to partnering to help marketers expand their reach everywhere.
The Built with BigQuery advantage for ISVs and data providers
Built with BigQuery helps companies like Zeotap build innovative applications with Google Data Cloud. Participating companies can:
Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices.
Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
BigQuery gives ISVs the advantage of a powerful, highly scalable unified Data Cloud for the agentic era, that’s integrated with Google Cloud’s open, secure, sustainable platform. Click here to learn more about Built with BigQuery.
In the fast-evolving world of agentic development, natural language is becoming the standard for interaction. This shift is deeply connected to the power of operational databases, where a more accurate text-to-SQL capability is a major catalyst for building better, more capable agents. From empowering non-technical users to self-serve data, to accelerating analyst productivity, the ability to accurately translate natural language questions into SQL is a game-changer. As end-user engagements increasingly happen over chat, conversations become the fundamental connection between businesses and their customers.
In an earlier post, “Getting AI to write good SQL: Text-to-SQL techniques explained,” we explored the core challenges of text-to-SQL — handling complex business context, ambiguous user intent, and subtle SQL dialects — and the general techniques used to solve them.
Today, we’re moving from theory to practice. We’re excited to share that Google Cloud has scored a new state-of-the-art result on the BIRD benchmark’s Single Trained Model Track. We scored 76.13, ahead of any other single-model solution (higher is better). In general, the closer you get to the benchmark of human performance (92.96), the harder it is to score incremental gains.
BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation) is an industry standard for testing text-to-SQL solutions. BIRD spans over 12,500 unique question-SQL pairs from 95 databases with a total size of 33 GB. The Single Trained Model Track is designed to measure the raw, intrinsic capability of the model itself, restricting the use of complex preprocessing, retrieval, or agentic frameworks often used to boost model accuracy. In other words, success here reflects an advancement in the model’s core ability to generate SQL.
Gemini scores #1 place in BIRD (October ‘25)
From research to industry-leading products
This leap in more accurate natural-language-to-SQL capability, often referred to as NL2SQL, isn’t just an internal research or engineering win; it fundamentally elevates the customer experience across several key data services,and our state-of-the-art research in this field is enabling us to create industry-leading products that customers leverage to activate their data with agentic AI.
Consider AlloyDB AI’s natural language capability, a tool that customers use to allow end users to query the most current operational data using natural language. For instance, companies like Hughes, an Echostar Corporation, depend on AlloyDB’s NL2SQL for critical tasks like call analytics. Numerous other retail, technology, and industry players also integrate this capability into their customer-facing applications. With NL2SQL that is near-100% accurate, customers gain the confidence to build and deploy applications in production workloads that rely on real-time data access.
The benefits of NL2SQL extend to analysis, as exemplified with conversational analytics in BigQuery. This service lets business users and data analysts explore data, run reports, and extract business intelligence from vast historical datasets using natural language. The introduction of a multi-turn chat experience, combined with a highly accurate NL2SQL engine, helps them make informed decisions with the confidence that the responses from BigQuery-based applications are consistently accurate.
Finally, developers are finding new efficiencies. They have long relied on Google Code Assist (GCA) for code generation, aiding their application development with databases across Spanner, AlloyDB, and Cloud SQL Studio. With the availability of more accurate NL2SQL, developers will be able to use AI coding assistance to generate SQL code too.
BIRD: a proving ground for core model capability
BIRD benchmark is one of the most commonly used benchmarks in the text-to-SQL field. It moves beyond simple, single-table queries to cover real-worldchallenges our models must handle, such as reasoning over very large schemas, dealing with ambiguous values, and incorporating external business knowledge. Crucially, BIRD measures a critical standard: execution-verified accuracy. This means a query is not just considered ‘correct’ if it appears right; it must also successfully run and return the correct data.
We specifically targeted the Single Trained Model Track because it allows us to isolate and measure the model’s core ability to solve the text-to-SQL task (rather than an ensemble, a.k.a., a system with multiple components such as multiple parallel models, re-rankers, etc.). This distinction is critical, as text-to-SQL accuracy can be improved with techniques like dynamic few-shot retrieval or schema preprocessing; this track reflects the model’s true reasoning power. By focusing on a single-model solution, these BIRD results demonstrate that enhancing the core model creates a stronger foundation for systems built on top of it.
Our method: Specializing the model
Achieving a state-of-the-art score doesn’t happen only by using a powerful base model. The key is to specialize the model. We developed a recipe designed to transform the model from a general-purpose reasoner into a highly specialized SQL-generation expert.
This recipe consisted of three critical phases applied before inference:
Rigorous data filtering: Ensuring the model learns from a flawless, “gold standard” dataset.
Multitask learning: Teaching the model not just to translate, but to understand the implicit subtasks required for writing a correct SQL query.
Test-time scaling: “Self consistency” a.k.a., picking the best answer.
Let’s break down each step.
Our process for achieving SOTA result
Step 1: Start with a clean foundation (data filtering)
One important tenet of fine-tuning is “garbage in, garbage out.” A model trained on a dataset with incorrect, inefficient, or ambiguous queries may learn incorrect patterns. The training data provided by the BIRD benchmark is powerful, but like most large-scale datasets, it’s not perfect.
Before we could teach the model to be a SQL expert, we had to curate a gold-standard dataset. We used a rigorous two-stage pipeline: first, execution-based validation to execute every query and discard any that failed, returned an error, or gave an empty result. Second, we used LLM-based validation, where multiple LLMs act as a “judge” to validate the semantic alignment between the question and the SQL, catching queries that run but don’t actually answer the user’s question. This aggressive filtering resulted in a smaller, cleaner, and more trustworthy dataset that helped our model learn from a signal of pure quality rather than noise.
Step 2: Make the model a SQL specialist (multitask learning)
With a clean dataset, we could move on to the supervised fine-tuning itself. This is the process of taking a large, general-purpose model — in our case, Gemini 2.5-pro — and training it further on our narrow, specialized dataset to make it an expert in a specific task.
To build these skills directly into the model, we leveraged the publicly available Supervised Tuning API for Gemini on Vertex AI. This service provided the foundation for our multitask supervised finetuning (SFT) approach, where we trained Gemini-2.5-pro on several distinct-but-related tasks simultaneously.
We also extended our training data to cover tasks outside of the main Text-to-SQL realm, helping enhance the model’s reasoning, planning, and self-correction capabilities.
By training on this combination of tasks in parallel, the model learns a much richer, more robust set of skills. It goes beyond simple question-to-query mapping — it learns to deeply analyze the problem, plan its approach, and refine its own logic, leading to drastically improved accuracy and fewer errors.
Step 3: Inference accuracy + test-time scaling with self-consistency
The final step was to ensure we could reliably pick the model’s single best answer at test time. For this, we used a technique called self-consistency.
With self-consistency, instead of asking the model for just one answer, we ask it to generate several query candidates for the same question. We then execute these queries, cluster them by their execution results, and select a representative query from the largest cluster. This approach is powerful because if the model arrives at the same answer through different reasoning paths, that answer has a much higher probability of being correct.
It’s important to note that self-consistency is a standard, efficient method, but it is not the only way to select a query. More complex, agentic frameworks can achieve even higher accuracy. For example, our team’s own research on CHASE-SQL (our state-of-the-art ensembling methodology) demonstrates that using diverse candidate generators and a trained selection agent can significantly outperform consistency-based methods.
For this benchmark, we wanted to focus on the model’s core performance. Therefore, we used the more direct self-consistency method: we generated several queries, executed them, and selected a query from the group that produced the most common result. This approach allowed us to measure the model’s raw text-to-SQL ability, minimizing the influence of a more complex filtering or reranking system.
The BIRD Single-Model Track explicitly allows for self-consistency, which reflects the model’s own internal capabilities. The benchmark categorizes submissions based on the number of candidates used (‘Few’, ‘Many’, or ‘Scale’). We found our “sweet spot” in the “Few” (1-7 candidates) category.
This approach gave us the final, critical boost in execution accuracy that pushed our model to the top of the leaderboard. More importantly, it proves our core thesis: by investing in high-quality data and instruction tuning, you can build a single model that is powerful enough to be production-ready without requiring a heavy, high-latency inference framework.
A recipe for customizing Gemini for text-to-SQL
A combination of clean data, multi-task learning, and efficient self-consistencyallowed us to take the powerful Gemini 2.5-pro model and build a specialist that achieved the top-ranking score on the BIRD single-model benchmark.
Our fine-tuned model represents a much stronger baseline for text-to-SQL. However, it’s important to note that this score is not the upper bound of accuracy. Rather, it is the new, higher baseline we have established for the core model’s capability in a constrained setting. These results can be further amplified by either
creating an ensemble, aka integrating this specialist model into a broader system that employs preprocessing (like example retrieval) or agentic scaffolding (like our CHASE-SQL research), or
optimizing model quality for your unique database by enhancing metadata and/or query examples (which is how our customers typically deploy production workloads).
Nevertheless, the insights from this research are actively informing how we build our next-generation AI-powered products for Google Data Cloud, and we’ll continue to deliver these enhancements in our data services.
Explore advanced text-to-SQL capabilities today
We’re constantly working to infuse our products with these state-of-the-art capabilities, starting with bringing natural language queries to applications built on AlloyDB and BigQuery. For AI-enhanced retrieval, customers especially value AlloyDB and its AI functions. AlloyDB integrates AI capabilities directly into the database, allowing developers to run powerful AI models using standard SQL queries without moving data. It offers specialized operators such as AI.IF() for intelligent filtering, AI.RANK() for semantic reranking of search results, and AI.GENERATE() for in-database text generation and data transformation.
And if you want to write some SQL yourself, Gemini Code Assist can help. With a simple prompt, you can instruct Gemini as to the query you want to create. Gemini will generate your code and you can immediately test it by executing it against your database. We look forward to hearing about what you build with it!
Editor’s note: Waze (a division of Google parent company Alphabet) depends on vast volumes of dynamic, real-time user session data to power its core navigation features, but scaling that data to support concurrent users worldwide required a new approach. Their team built a centralized Session Server backed by Memorystore for Redis Cluster, a fully managed service with 99.99% availability that supports partial updates and easily scales to Waze’s use case of over 1 million MGET commands per second with ~1ms latency. This architecture is the foundation for Waze’s continued backend modernization.
Real-time data drives the Waze app experience. Our turn-by-turn guidance, accident rerouting, and driver alerts depend on up-to-the-millisecond accuracy. But keeping that experience seamless for millions of concurrent sessions requires robust and battle hardened infrastructure that is built to manage a massive stream of user session data. This includes active navigation routes, user location, and driver reports that can appear and evolve within seconds.
Behind the scenes, user sessions are large, complex objects that update frequently and contribute to an extremely high volume of read and write operations. Session data was once locked in a monolithic service, tightly coupled to a single backend instance. That made it hard to scale and blocked other microservices from accessing the real-time session state. To modernize, we needed a shared, low-latency solution that could handle these sessions in real time and at global scale. Memorystore for Redis Cluster made that possible.
aside_block
<ListValue: [StructValue([(‘title’, ‘Build smarter with Google Cloud databases!’), (‘body’, <wagtail.rich_text.RichText object at 0x7f65a9750eb0>), (‘btn_text’, ”), (‘href’, ”), (‘image’, None)])]>
Choosing the right route
As we planned the move to a microservices-based backend, we evaluated our options, including Redis Enterprise Cloud, a self-managed Redis cluster, or continuing with our existing Memcached via Memorystore deployment. In the legacy setup, Memcached stored session data behind the monolithic Realtime (RT) server, but it lacked the replication, advanced data types, and partial update capabilities we wanted. We knew Redis had the right capabilities, but managing it ourselves or through a third-party provider would add operational overhead.
Memorystore for Redis Cluster offered the best of both worlds. It’s a fully managed service from Google Cloud with the performance, scalability, and resilience to meet Waze’s real-time demands. It delivers a 99.99% SLA and a clustered architecture for horizontal scaling. With the database decision made, we planned a careful migration from Memcached to Memorystore for Redis using a dual-write approach. For a period, both systems were updated in parallel until data parity was confirmed. Then we cut over to Redis with zero downtime.
Waze’s new data engine
From there, we built a centralized Session Server – our new command center for active user sessions – as a wrapper around Memorystore for Redis Cluster. This service became the single source of truth for all active user sessions, replacing the tight coupling between session data and the monolithic RT server. The Session Server exposes simple gRPC APIs, allowing any backend microservice to read from or write to the session state directly, including RT during the migration. This eliminated the need for client affinity, freed us from routing all session traffic through a single service, and made session data accessible across the platform.
We designed the system for resilience and scale from the ground up. Redis clustering and sharding remove single points of contention, letting us scale horizontally as demand grows. Built-in replication and automatic failover are designed to keep sessions online. While node replacements may briefly increase failure rates and latency for a short period, sessions are designed to stay online, allowing the navigation experience to quickly stabilize.And with support for direct gRPC calls from the mobile client to any backend service, we can use more flexible design patterns while shaving precious milliseconds off the real-time path.
Fewer pit stops, faster rides
Moving from Memcached’s 99.9% SLA to Memorystore for Redis Cluster’s 99.99% means higher availability and resiliency from the service. Load testing proved the new architecture can sustain full production traffic, comfortably handling bursts of up to 1 million MGET commands per second with a stable sub-millisecond service latency.
Because Memorystore for Redis supports partial updates, we can change individual fields within a session object rather than rewriting the entire record. That reduces network traffic, speeds up write performance, and makes the system more efficient overall – especially important when sessions can grow to many megabytes in size. These efficiencies translate directly into giving our engineering teams more time to focus on application-level performance and new feature development.
Session data in Memorystore for Redis Cluster is now integral to Waze’s core features, from evaluating configurations to triggering real-time updates for drivers. It supports today’s demands and is built to handle what’s ahead.
The road ahead
By proving Memorystore for Redis Cluster in one of Waze’s most critical paths, we’ve built the confidence to use it in other high-throughput caching scenarios across the platform. The centralized Session Server and clustered Redis architecture are now standard building blocks in our backend, which we can apply to new services without starting from scratch.
With that initial critical path complete, our next major focus is the migration of all remaining legacy session management from our RT server. This work will ultimately give every microservice independent access to update session data. Looking ahead, we’re also focused on scaling Memorystore for Redis Cluster to meet future user growth and fine-tuning it for both cost and performance.
Learn more
Waze’s story showcases the power and flexibility of Memorystore for Redis Cluster, a fully managed service with 99.99% availability for high-scale, real-time workloads.
Learn more about the power of Memorystore and get started for free.
Welcome back to The Agent Factory! In this episode, we’re joined by Ravin Kumar, a Research Engineer at DeepMind, to tackle one of the biggest topics in AI right now: building and training open-source agentic models. We wanted to go beyond just using agents and understand what it takes to build the entire factory line—from gathering data and supervised fine-tuning to reinforcement learning and evaluations.
This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.
Before diving into the deep research, we looked at the latest developments in the fast-moving world of AI agents.
Gemini 2.5 Computer Use: Google’s new model can act as a virtual user, interacting with computer screens, clicking buttons, typing in forms, and scrolling. It’s a shift from agents that just know things to agents that can do tasks directly in a browser.
Vibe Coding in AI Studio: A new approach to app building where you describe the “vibe” of the application you want, and the AI handles the boilerplate. It includes an Annotation Mode to refine specific UI elements with simple instructions like “Change this to green.”
DeepSeek-OCR and Context Compression: DeepSeek introduced a method that treats documents like images to understand layout, compressing 10-20 text tokens into a single visual token. This drastically improves speed and reduces cost for long-context tasks.
Google Veo 3.1 and Flow: The new update to the AI video model adds rich audio generation and powerful editing features. You can now use “Insert” to add characters or “Remove” to erase objects from existing video footage, giving creators iterative control.
Ravin Kumar on Building Open Models
We sat down with Ravin to break down the end-to-end process of creating an open model with agent capabilities. It turns out the process mirrors a traditional ML lifecycle but with significantly more complex components.
Ravin explained that training data for agents looks vastly different from standard text datasets. It starts with identifying what users actually need. The data itself is a collection of trajectories, complex examples of the model making decisions and using tools. Ravin noted that they use a mix of human-curated data and synthetic data generated by their own internal “teacher” models and APIs to create a playground for the open models to learn in.
Training Techniques: SFT and Reinforcement Learning
Once the data is ready, the training process involves a two-phase approach. First comes Supervised Fine-Tuning (SFT), where frameworks update the model’s weights to nudge it into new behaviors based on the examples. However, to handle generalization—new situations not in the original trainin data—they rely on Reinforcement Learning (RL). Ravin highlighted the difficulty of setting rewards in RL, warning that models are prone to “reward hacking,” where they might collect intermediate rewards without ever completing the final task.
Ravin emphasized that evaluation is the most critical and high-stakes part of the process. You can’t just trust the training process; you need a rigorous “final exam.” They use a combination of broad public benchmarks to measure general capability and specific, custom evaluations to ensure the model is safe and effective for its intended user use case.
Conclusion
This conversation with Ravin Kumar really illuminated that building open agentic models is a highly structured, rigorous process. It requires creating high-quality trajectories for data, a careful combination of supervised and reinforcement learning, and, crucially, intense evaluation.
Your turn to build
As Ravin advised, the best place to start is at the end. Before you write a single line of training code, define what success looks like by building a small, 50-example final exam for your agent. If you can’t measure it, you can’t improve it. We also encourage you to try mixing different approaches; for example, using a powerful API model like Gemini as a router and a specialized open-source model for specific tasks.
Check out the full episode for more details, and catch us next time!
In a world of increasing data volume and demand, businesses are looking to make faster decisions and separate insight from noise. Today, we’re bringing Conversational Analytics to general availability in Looker, delivering natural language queries to everyone in your organization, removing BI bottlenecks. With Conversational Analytics, we’re transforming the way you get answers, cutting through stale dashboards and accelerating data discovery. Our goal: make analytics and AI as easy and scalable as performing a Google search, extending BI to the broader enterprise as you go from prompt to full data exploration in seconds.
Instant AI-powered insights with Conversational Analytics in Looker
Now, with Conversational Analytics, getting an answer from your data is as simple as chatting with your most knowledgeable colleague. By tapping into human conversation, Conversational Analytics relieves you from struggling with complex dashboard filters, obscure field names, or the need to write custom SQL.
“At YouTube, we’re focused on helping creators succeed and bring their creativity to the world. We’ve been testing Conversational Analytics in Looker to give our partner managers instant, actionable data that lets them quickly guide creators and optimize creator support.” – Thomas Seyller, Senior Director, Technology & Insights, YouTube Business
The general availability of Conversational Analytics combines the reasoning power of Gemini, new capabilities in Google’s agentic frameworks, and the trusted data modeling of the Looker platform. Together, these set the stage for the next chapter in self-service analytics, making reliable data insights accessible to the entire enterprise. Conversational Analytics agents can understand your questions and provide insightful answers to questions about your data.
New at general availability is the ability to analyze data across domains. You can ask questions that integrate insights from up to five distinct Looker Explores (pre-joined views), spanning multiple business areas. Additionally, you can share the agents you build with colleagues, giving them faster access to a single source of truth, speeding consensus, and driving uniform decisions.
You can build and share agents with colleagues to have a consistent data picture.
Built on a trusted, governed foundation
The power of Conversational Analytics isn’t just in the conversation it enables; it’s in the trust of the underlying data. Conversational Analytics is grounded in Looker’s semantic layer, which ensures that every metric, field, and calculation is centrally defined and consistent, acting as a crucial context engine for AI. As more of your colleagues rapidly use these expanded capabilities, you need to know the results they see and act on are accurate.
For analysts looking to explore data or everyday users receiving insights in the context of their business, Conversational Analytics also improves data self-service, minimizing technical friction that can create bottlenecks and leaves insights locked away.
You can now:
Ask anything, anytime: Get instant answers to simple questions like “Show me our website traffic last month for shoe sales,” leading to deeper questions and greater insights across business areas and domains.
Deepen the discovery: Move beyond the constraints of static dashboards and ask open-ended questions like, “Show me the trend of website traffic over the past six months and filter it by the California region.” The system intelligently generates the appropriate query and visualization instantly.
Extend enterprise BI: Connect your Looker models to your enterprise BI ecosystem, centralize and share agents, and create new dashboards, starting with a prompt. Built on top of Looker Explores, Conversational Analytics’ natural language interface usesLookML for fine tuning and output accuracy.
Pivot quickly: The conversational interface supports multi-turn questions, so you can iterate on your findings. Ask for total sales, then follow up with, “Now show me that as an area chart, broken down by payment method.”
Gain full transparency: To build confidence and data literacy, the “How was this calculated?” feature provides a clear, natural language explanation of the underlying query that generated the results, so that you understand the source of your findings.
Empower the BI analyst and business user
Conversational Analytics is democratizing data for business teams, helping them govern the business’s data. At the same time, it’s also enhancing productivity and influence for data analysts and developers.
When business users can self-serve trusted data insights, data analysts see fewer interruptions and “ad-hoc” ticket requests, and can instead focus on high-impact work. Analysts can customize their client teams’ BI experiences by building Conversational Analytics agents that define common questions, filters, and style guidelines, so different teams can act on the same data in different ways.
Get ready to start talking
Conversational Analytics is available now for all Looker platform users. Your admin can enable it in your Looker instance today and you will discover how easy it is to move from simply asking “What?” to confidently determining “What’s next?” For more information, review the product documentation or watch this video tutorial.
