gcp

As cybersecurity threats have grown more sophisticated and prevalent, we’ve seen organizations develop robust cyber threat intelligence (CTI) programs to help bolster defenses. However, creating and maturing a CTI program remains a challenge because it requires people, processes, technologies, and metrics to validate success.

To help organizations better operationalize threat intelligence, we’ve published the CTI Program Design Playbook. This training track is the inaugural release from Mandiant Academy’s newest on-demand training, developed for professionals who actively defend networks. It’s designed to provide you with the essential knowledge and skills to design, build, operate, and optimize a CTI program from the ground up.

What you’ll learn: An overview

This track includes three two-hour courses addressing different aspects of CTI program development.

In the first course, Designing and Building a CTI Program, you will learn how to establish the foundations of a CTI program, including a CTI program charter and roadmap.

In the second course, Operating an Effective CTI Program, you will learn how to effectively identify and maintain a CTI program’s stakeholders. This course also describes the resources necessary to collect, analyze, and disseminate threat intelligence while accommodating stakeholders’ unique requirements.

Students who are part of an active CTI program but would like to mature their team’s capabilities may benefit from the third course, Optimizing an Efficient CTI Program. This course teaches you helpful ways to accurately measure a CTI program’s effectiveness and, where necessary, make improvements.

Who should take these courses?

This training track is the inaugural release from Mandiant Academy’s newest approach to on-demand training, which breaks the core tenets of effective cybersecurity strategies into discrete two-hour courses. This new style aims to address the needs of busy practitioners who already possess the knowledge and skills needed to defend networks but have limited time to enhance their capabilities. 

The CTI Program Design Playbook was developed for practitioners:

• Intelligence analysts supporting network defenders at the tactical, operational, or strategic levels

• Managers of active intelligence programs who would like to improve their team’s capabilities

• Cybersecurity practitioners who desire an intelligence capability

• Practitioners tasked with founding their organization’s first intelligence capability

Start learning today

To learn more about the CTI Program Design Playbook and to sign up for the course, please visit our website. You can access a wealth of knowledge through Mandiant Academy’s on-demand, instructor-led, and experiential training options. We hope this course proves helpful in your efforts to defend your organization against cyber threats.

As a Kubernetes platform engineer, you’ve probably followed the buzz around eBPF and its revolutionary impact on Kubernetes networking. Perhaps you’ve explored Cilium, a popular solution leveraging eBPF, and wondered how Google Kubernetes Engine (GKE) harnesses this power. That curiosity may have led you to GKE Dataplane V2. And you’ve undoubtedly noticed that where eBPF goes, enhanced observability follows.

In this article, we’ll shine a light on a tool that you can leverage with GKE Dataplane V2: Hubble. This powerful observability tool provides deep visibility with added Kubernetes context into your network packet flow.

GKE Dataplane V2 observability provides a Managed Hubble solution. This includes the following components:

• Hubble Relay: a service that collects network telemetry data about your pods and nodes

• Hubble CLI: a command-line interface tool providing live traffic information within the cluster

Enabling Dataplane V2 observability in GKE, either through the Google Cloud console or the gcloud command, triggers the deployment of a dedicated hubble-relay deployment with a single pod. This pod houses both the Hubble Relay and CLI components as dedicated containers, providing immediate access to network telemetry data and command-line tools for analysis.

While the Hubble UI is not included in the initial deployment, you can easily enhance your observability setup by adding it. This web-based interface offers a visual and interactive way to explore and analyze the data collected by Hubble Relay, making it easier to identify trends, spot anomalies, and troubleshoot issues.

With Hubble CLI and UI at our disposal, let’s explore how these tools can be used to effectively troubleshoot your Kubernetes environment.

Before you start

• Make sure you’re comfortable using the Hubble CLI (link to documentation).

• Know how to deploy the Hubble UI binary distribution (link to documentation).

Setting the stage

For this example, we’ll use two pods, each deployed in a separate namespace and exposed as a service.

Scenario 1: Tracing a simple request

In this scenario, we observe a basic interaction between a pod and a service.

1. Initiate a request: Execute a curl request from the store-frontend pod to backend-svc service:

$ kubectl exec store-frontend -n fe -it -- curl “backend-svc.be.svc.cluster.local.”

2. Observe with Hubble: Use the Hubble CLI to examine the traffic:

$ hubble observe --pod fe/store-frontend --follow

Hint: For a full list of Hubble filters and flags, run hubble observe –help

Hubble captures the entire flow: the store-frontend pod contacting kube-dns for service name resolution, followed by the TCP connection to the store-backend pod. See how Hubble adds the namespace and pod names to the packets? This Kubernetes-aware insight makes it much easier to understand and analyze your network traffic.

Scenario 2: When DNS goes wrong

Let’s simulate a scenario where our DNS server is overwhelmed. We deploy an application that floods the cluster with DNS requests, putting significant strain on the kube-dns pods. Then, we replicate the request from Scenario 1.

By examining the timestamps of the DNS request and response in the Hubble CLI output, we can clearly observe a delay of approximately 5 seconds for kube-dns to respond. This highlights the impact of the overloaded DNS server and explains the latency observed in our system.

Let’s intensify the load on our DNS server even further, pushing it to the point where DNS requests fail outright due to an inability to resolve service names.

$ kubectl exec -it store-frontend -n fe -- curl "backend-svc.be.svc.cluster.local."
curl: (6) Could not resolve host: backend-svc.be.svc.cluster.local.

In this extreme scenario, the Hubble drop reveals that our DNS queries are simply not being answered, indicating a complete breakdown in name resolution.

We finally will make the kube-dns pods unavailable by taking them down.

Hubble’s flow logs provide a clear picture of what’s happening. You can see that the store-frontend pod attempts to reach the kube-dns service, but the request fails because there are no healthy kube-dns pods to handle it.

Hubble CLI isn’t just a troubleshooting tool; it’s also powerful for optimization. For example, while running the first scenario, you may have noticed a surprisingly high number of DNS requests generated from a single curl command:

A quick investigation reveals the root cause: a missing trailing dot in the service name, and the pod’s resolv.conf being configured with ndots:5. This combination meant the query wasn’t treated as absolute and was instead expanded with all five search domains listed in resolv.conf, resulting in six DNS requests for every single curl call.

Scenario 3: Network policy mishap!

It appears there was a slight oversight with a new Ingress NetworkPolicy. Let’s just say Monday mornings and network policies don’t always mix…! The front-end application was accidentally left out of the allowlist.

Communications between the front-end and the back-end applications hang and timeout. The front-end engineer rushes to Hubble CLI. Hubble shows that the front-end application traffic is being denied and dropped by a network policy.

Thankfully, the Hubble CLI allows you to filter events by type using the –type flag. For instance, --type policy-verdict shows only events related to network policies. You can further refine this by using --verdict DROPPED to see only events where traffic was dropped due to a policy.

Hubble UI

While the Hubble CLI is undeniably powerful, the Hubble UI offers a complementary and equally valuable perspective. Beyond replicating the CLI’s functionality, the UI provides a visual map of your cluster’s network traffic. This allows you to easily grasp how pods and Services connect, both internally and to the outside world. With filtering options like namespaces and labels, you can quickly isolate specific traffic flows and troubleshoot potential problems.

Explore the depths of GKE Dataplane V2 with Hubble

This exploration of Hubble, while introductory, provides a solid foundation for understanding its capabilities. Remember that far more intricate queries and filters can be crafted to gain deeper insights into specific network issues. Simply knowing about Hubble and how to access it is a significant first step towards effectively operating and troubleshooting Cilium-based Dataplane V2 clusters. As you dive deeper into Hubble’s capabilities, you’ll discover its full potential for observability and optimization within your GKE environment.

When it comes to AI, large language models (LLMs) and machine learning (ML) are taking entire industries to the next level. But with larger models and datasets, developers need distributed environments that span multiple AI accelerators (e.g. GPUs and TPUs) across multiple compute hosts to train their models efficiently. This can lead to orchestration, resource management, and scalability challenges.

We’re here to help. At Google Cloud, we provide a robust suite of GPU and TPU resources alongside advanced orchestration tools as part of AI Hypercomputer architecture to simplify distributed, large-scale training. In this blog, we’ll guide you through the orchestration tools available for GPU accelerators on Google Cloud that can help you streamline and scale your machine learning workflows.

Choose the right accelerator family

A key element of distributed training lies in selecting the right GPU. Google Cloud’s specialized machine families offer tailored solutions for varying needs of performance and cost efficiency. The A3 machine series, featuring NVIDIA H100 and NVIDIA H200 (upcoming) GPUs, delivers strong GPU-to-GPU bandwidth that’s a great fit for large-scale training workloads. In contrast, the A2 machine series with NVIDIA A100 GPUs is designed for scenarios that require minimal inter-node communication such as streamlined, single-node training. Additionally, the versatile G2 machine family, equipped with NVIDIA L4 GPUs, provides the flexibility necessary for inference and testing workloads.

We also offer multiple GPU consumption models to meet the needs of large-scale training:

• Committed Use Discounts (CUDs) provide significant cost savings and guaranteed capacity in return for a long-term commitment.

• Dynamic Workload Scheduler (DWS) comes in two modes, which are designed for various workloads that need assurance or can be flexible about start time; the capacity is available for a defined duration and offered at a lower list price.

• On-demand consumption is the most flexible, with no upfront commitments, although the capacity availability is not guaranteed.

• Spot VMs provide drastically lower costs but are preemptible, requiring resilient and disruption-tolerant job designs.

To further accelerate your distributed training, we’ll explore three powerful orchestration strategies on Google Cloud: Google Kubernetes Engine (GKE), Cluster Toolkit, and Vertex AI custom training pipeline. Each approach brings its unique strengths, enabling you to leverage Google Cloud’s powerful infrastructure to drive your machine learning projects forward quickly and scalably.

Let’s walk through each of the options to better understand how Google Cloud’s advanced orchestration tools can help you optimize resources, reduce complexity, and achieve strong performance in your ML initiatives.

Option 1: GKE for unified workload management

Enterprises with robust platform teams often want a unified environment on which to run all their workloads, including custom training, for simpler management. GKE is a good choice in this context, providing the flexibility and scalability to handle diverse workloads on a single platform. With GKE, platform teams gain centralized control and visibility, while optimizing resource utilization and streamlining management.

Here’s how to orchestrate ML workloads running on GKE:

1. GKE cluster and nodepool provisioning
If you have reservation (CUD or DWS calendar) and prefer to use Terraform, follow the instructions from cluster provisioning templates, and specify the parameter file (terraform.tfvars):

Then execute the following command to provision the GKE cluster and nodepool:

In addition, Cluster Toolkit includes terraform based example blueprints to provision A3 or A3 Mega GKE clusters and nodepool. 

If you prefer to use the gcloud command, follow the step-by-step instructions from this tutorial to create a GKE cluster and nodepool with A3/A3 Mega VMs. 

For DWS Flex, you can create a DWS enabled node-pool with these gcloud commands:

2. Enable GPU direct communication with A3 TCPX/A3 Mega TCPXO and perform an initial benchmark test
Follow these steps to install GPUDirect for TCPX/TCPXO libraries, configure NCCL, and deploy a test workload to perform your initial benchmark tests.

Validate the output of allgather benchmark tests for two A3 Mega nodes:

In the above benchmark output table, the first column is message size, while the algbw and busbw columns on the right indicate per GPU bandwidth. Usually, we use the in/out-of-place busbw column with the biggest message size (highlighted row) to determine cross-node bandwidth. For A3 Mega nodes, we expect a range of 185-190GB/s per GPU; this may indicate near cross-node 1600gbps network bandwidth for A3 Mega nodes with 8 NVIDIA H100 GPUs and 8 NICs.

You may expand the NCCL tests from two nodes to 8, 16, 32, etc. to ensure your cross-node network performance is within a decent range and that all the nodes are healthy.

3. Configure distributed training batch workload
You can use JobSet, a Kubernetes-native API for managing a group of k8s Jobs as a unit using a unified API, to deploy distributed HPC (e.g., MPI) and AI/ML training workloads (PyTorch, Jax, Tensorflow etc.) on Kubernetes.

The following example illustrates a JobSet yaml manifest for A3 with GPUDirect-TCPX, which includes:

1. Key JobSet configuration elements

b. Training job settings, including pytorch main container
c. gcsfuse, tcpx (A3 high), tcpxo (A3 Mega) RxDM container
d. NCCL environment variables

For DWS batch workloads, please refer to the following A3 Mega-based example, integrating Kueue and JobSet settings.

Lastly, refer to this Helmchart example to see how to perform Megatron LM (Llama2) training on A3 Mega.

Option 2: Slurm via Cluster Toolkit

Slurm is one of the most popular high-performance computing (HPC) job schedulers. Used by researchers in both academia and industry, it offers a robust solution for LLM training orchestration with familiar semantics. Support for Slurm on Google Cloud is provided by Cluster Toolkit, formerly known as Cloud HPC Toolkit, open-source software that simplifies the process of deploying HPC, AI, and ML workloads on Google Cloud. It is designed to be highly customizable and extensible, and to address the deployment needs of a broad range of use cases, including deploying infrastructure for large-scale LLM training.

1. Provisioning A3-high and A3-mega clusters
Install Cluster Toolkit using the configuration instructions in the public documentation. Be sure to note some of the prerequisites including supported versions of Go, Terraform, and Packer.

Once you have a working Cluster Toolkit installation including the downloaded github repository, navigate to the examples/machine-learning blueprints directory. Here, you will have two folders for deploying H100 clusters based on the A3-series machine shapes, a3-highgpu-8g and a3-megagpu-8g. In this example, we’ll explore the blueprint in the a3-megagpu-8g folder.

Google Cloud Cluster Toolkit blueprints are Infrastructure as Code (IaC) documents that describe the infrastructure you would like to deploy, and are conceptually similar to Terraform or other IaC tooling. For the a3-megagpu-8g blueprint, there are three main files that control the deployment: 

1. slurm-a3mega-base.yaml – includes creating the necessary VPC networks along with the filestore instance used for a common home filesystem on the cluster nodes.
2. slurm-a3mega-image.yaml – creates the Compute Engine image instance that is used by Slurm to provision nodes based on the cluster’s definitio
3. slurm-a3mega-cluster.yaml – sets up the main cluster components, including the Slurm controller (the main orchestrator for Slurm jobs), the Slurm login node (a host used for job submission) and the a3mega partition (the working nodes in the cluster)

While you can customize each of the blueprint components if needed, you can easily get started by simply specifying the details for your working environment in the deployment-base.yaml and the deployment-image-cluster.yaml.

2. Enable GPUDirect-TCPXO optimized NCCL communication
Once the Slurm cluster is created, follow this tutorial to enable GPUDirect-TCPXO for optimized NCCL communication on the GPU networks. To validate the environment and ensure the TCPXO plugin is being properly loaded, build and compile the NCCL tests. Then, run sbatch run-nccl-tests.sh from the login node, being sure to change the number of nodes in the script to match those in your cluster. This runs a distributed all_gather test across the GPUs and nodes indicated in the script.

When working as intended, the NCCL tests should show output results indicating high-speed bandwidth throughput at various message sizes. A common measure of performance is to use the busbw value in GB/s from the second or last row of the output table, which shows the 4Gb and 8Gb message size values. A cluster with TCPXO active should report around 190 GB/s busbw throughput. See the performance page in the NVIDIA NCCL-tests repository for more details around these metrics.

3. Run an NeMo training workload
Follow this NeMo training tutorial to run an example NeMo Framework Pre-Training job using the following steps:

Step 1:

• Creates a NeMo Framework-derived container with the necessary TCPXO environment variables

• Submits a Slurm job to copy the framework launcher scripts and a few other auxiliary files into your working directory

Step 2:

• Establishes a Python virtual environment and installs NeMo Framework python package dependencies

Step 3:

• This command runs distributed training of a 5B parameter GPT3 model across eight nodes for 10 steps using mock data as the input.

Option 3: Vertex AI

For teams seeking managed infrastructure experience as well as access to leading open models such as Llama 3.1, Mistral, etc., Vertex AI Model Garden and Custom Training Job service presents an attractive option. This fully managed solution removes most of the orchestration burden and provides end-to-end ML platform operations, allowing you to focus on model development and experimentation. Vertex AI’s end-to-end training support further simplifies the process, offering an integrated workflow from data preparation to deployment.

Let’s look at how to perform  single-node or multi-node fine-tuning/training workload on Vertex.   

Single-node multi-GPU fine-tuning/training on Vertex
This notebook demonstrates fine-tuning and deploying Llama 3.1 models with the Vertex AI SDK. All of the examples in this notebook use parameter-efficient finetuning (PEFT) methods with Low-Rank Adaptation (LoRA) to reduce training and storage costs. LoRA is one approach of PEFT, where pretrained model weights are frozen and rank decomposition matrices representing the change in model weights are trained during fine-tuning.

Multi-node distributed fine-tuning/training on Vertex AI

This Vertex sample training repo provides examples on how to launch multi-node distributed training on A3 Mega (8 x NVIDIA H100) on Vertex. 

The NeMo example illustrates how to perform pre-training, continued pre-training and supervised fine-tuning (SFT). In addition, NeMo allows optimized training as a popular approach to evaluate the AI accelerator (A3 Mega in this case). To benchmark, you can rely on the reported metrics such as epoch time, step-time, etc. Since NeMo runs on most NVIDIA GPU types, it can be helpful for comparing different AI chips for a given task. Read on to learn how to run the example on Vertex with A3 Mega node types.

launch.sh is the main entry point to launch NeMo distributed training with command parameters:

Example:

At the end of launch.sh script, we use curl command to call Vertex customJobs API to launch NeMo training job in Vertex:

Job configurations in vertex-payload.json are part of curl command to launch Nemo training, it includes job specifications on resource requirements as showed:

The job configuration arguments “${TRANSFER_MODEL_CMD} ${LAUNCH_CMD}” in turn embed full content from the job training script, which also includes all the NCCL environments required by A3 Mega, while other pytorch launch commands are executed by Vertex CustomJob.

Optionally, build your own custom job container image as an “imageUri” parameter in vertex-payload.json, using this Dockerfile as your reference.

DIY enthusiasts: Building custom training environments

Lastly, we recognize many organizations prefer a more hands-on approach and have specific orchestration tools or frameworks that they wish to use. If that describes you, Google Compute Engine provides the foundation to build your own tailored training environments, letting you create and configure virtual machines (VMs) with your desired specifications, including the type and number of GPUs, CPU, memory, and storage. This granular control lets you optimize your infrastructure for your specific training workloads and integrate your preferred orchestration tools.

To facilitate this process, we provide example code snippets demonstrating how to use the gcloud compute instance create and gcloud compute instance bulk create API calls to create and manage your vanilla A3 Mega instances. Whether you need to create a single VM or provision a large-scale cluster, these resources can help streamline your infrastructure setup.

Conclusion

With the right orchestration strategy and Google Cloud’s robust and leading AI infrastructure, you can achieve your training goals and transform your business objectives into reality.

To learn more about distributed training, please review GKE example, Cluster Toolkit example, and Vertex AI example.

Search is at the heart of how we interact with the digital ecosystem, from online shopping to finding critical information. Enter generative AI, and user expectations are higher than ever. For applications to meet diverse user needs, they need to deliver fast, accurate and contextually relevant results, regardless of how queries are framed. For example:

• Online shoppers expect to find “waterproof hiking boots with ankle support” just as easily as a specific “SummitEdge Pro” model.

• Legal professionals need to pinpoint precise case citations or explore nuanced legal concepts with varied search terms.

• Doctors require precision when searching for critical patient information. A doctor looking for “allergy to penicillin” must locate the record accurately, whether the information is labeled as “drug sensitivities” or misspelled as “peniciln”. 

Spanner, Google’s always-on multi-model database with virtually unlimited scale, addresses these challenges with AI-powered hybrid search capabilities. Spanner allows developers to combine vector search, full-text search, and machine learning (ML) model reranking capabilities in a unified platform directly integrated with the operational data store, using a familiar SQL interface. 

In this post, we will explore how you can build a customized search engine for an ecommerce marketplace using Spanner.

Building a tailored search engine on Spanner

For ecommerce – along with many other industries – a single search method often falls short, resulting in dissatisfied users, incomplete information, or lost revenue. Keyword search excels at precision but struggles with alternate phrasing or natural language; vector search captures semantics but may overlook specific terms. Combining the strengths of both would enable organizations to deliver a more effective search experience.

SpanMart, a hypothetical ecommerce marketplace, allows users to search for products using keywords or natural language. Its products table supports multiple search methods with two specialized columns and associated indexes:

• A description_tokens column: This is a tokenized version of the description column, breaking down the text into individual terms. A search index (products_by_description) on this column accelerates full-text search, acting like an inverted index in information retrieval.

• An embedding column: This stores vector representations of the product descriptions, capturing semantic meaning rather than individual words. Similar descriptions are mapped close together in the “embedding space”. These embeddings are generated using models like Vertex AI Embeddings. A vector index (products_by_embedding) organizes these embeddings using a ScaNN tree structure for efficient semantic searches.

Here’s how the products table and its indexes are defined Spanner:

With these components in place, SpanMart can build an intelligent search pipeline that integrates: 

• Vector search for semantic relevance.
• Full-text search for precise keyword matching.
• Result fusion for combining the results from different retrieval methods. 
• ML model reranking for advanced result refinement.

This pipeline operates entirely within Spanner, where the operational data is stored. By avoiding integration with separate search engines or vector databases, Spanner eliminates the need for multiple technical stacks, complex ETL pipelines, and intricate application logic for inter-system communication. This reduces architectural and operational overhead and avoids potential performance inefficiencies.

The diagram below illustrates a high-level overview of how these components work together in Spanner.

Combining the power of vector and full-text search

When a user searches for products on SpanMart, the system first uses the embedding model to convert the user query into a vector that captures its semantic meaning. Then, SpanMart can build two queries:

• Approximate nearest neighbors (ANN) vector search:

• Full-text search (FTS):

These two queries excel in different scenarios and complement each other. For instance, when a user searches for a specific product model number, such as “Supercar T-6468”, the full-text search query can accurately find the exact model, while the vector search query suggests similar items. Conversely, for more complex natural language queries, such as “gift for an 8-year-old who enjoys logical reasoning but not a toy”, full-text search may struggle to yield useful results, whereas vector search can provide a relevant list of recommendations. Combining both queries would produce robust results for both styles of searches.

Reciprocal rank fusion (RRF)

RRF is a simple yet effective technique for combining results from multiple search queries. It calculates a relevance score for each record based on its position in all result sets, rewarding records ranked highly in individual searches. This method is particularly useful when the relevance scores from the individual searches are calculated in different spaces, making them difficult to compare directly. RRF addresses this by focusing on the relative rankings instead of scores within each result set.

Here’s how RRF works in our example:

• Calculate rank reciprocals: For each product, calculate its rank reciprocal in each result set by taking the inverse of its rank after adding a constant (e.g., 60). This constant prevents top-ranked products from dominating the final score and allows lower-ranked products to contribute meaningfully. For instance, a product ranked 5th in one result set would have a rank reciprocal of 1/(5 + 60) = 1/65 in that result set.

• Sum rank reciprocals: Sum the rank reciprocals from all result sets to get the final RRF score of a product.

The formula for RRF is –

 – where:

• d is a product description

• R is the set of retrievers (in this case, the two search queries)

• rank_r (?) is the rank of product description d in the results of retriever r.

• k is a constant

Implementing RRF within Spanner’s SQL interface is relatively straightforward. Here’s how:

Explanations:

• Common table expressions (CTEs): These are the WITH clauses, which are used in this query to improve readability. However, due to a current limitation, they may cause the query optimizer to default to an older version that lacks full-text search support. For now, the query uses the @{optimizer_version=7} hint to suggest a more recent optimizer version. 

• ANN CTE: This is the same as the previous ANN query, but with a twist. We assign a rank to each product in the results. While Spanner doesn’t support a direct way to assign ranks, there’s a workaround. By converting the results into an array of structs, we can use the offset of each element within the array as its rank. Since array offsets start at zero, we use offset + 1 to represent the actual rank. Note that this is purely a SQL language workaround without performance impact. The query planner effectively optimizes away the array conversion and directly assigns an offset to each row in the result set.

• FTS CTE: Similarly, this part mirrors the previous full-text search query, with the rank assigned using the array offset.

• Combining and ranking: The results from both CTEs are unioned, and grouped by the product id. For each product, we calculate the rrf_score and then select the top 50 products.

While RRF is an effective technique, Spanner’s versatile SQL interface empowers application developers to explore and implement various other result fusion methods. For instance, developers can normalize scores across different searches to a common range and then combine them using a weighted sum, assigning different importance to each search method. This flexibility allows for fine-grained control over the search experience and enables developers to tailor it to specific application requirements.

Using an ML model to rerank search results 

ML model-based reranking is a powerful way of refining search results to deliver improved results to the users. It applies an advanced yet computationally expensive model to a narrowed set of initial candidates, retrieved using methods like vector search, full-text search, or their combination, as discussed earlier. Due to its high computational cost, ML model-based reranking is applied after the initial retrieval reduces the result set to a small set of promising candidates.

Spanner’s integration with Vertex AI makes it possible to perform ML model-based reranking directly within Spanner. You can use a model deployed to your Vertex AI endpoint, including those available from the Vertex AI Model Garden. Once the model is deployed, you can create a corresponding reranker MODEL in Spanner.

In this example, SpanMart employs a Cross-Encoder model for reranking. This model takes two text inputs –  text and text_pair – and outputs a relevance score indicating how well the two texts align. Unlike vector search, which uses an embedding model to independently map each text into a fixed-dimensional space before measuring their similarity, a Cross-Encoder directly evaluates the two texts together. This allows the Cross-Encoder to capture richer contextual and semantic nuances in complex queries, such as “gift for an 8-year-old who enjoys logical reasoning but not a toy”. In a more advanced setup, the reranker could leverage a custom-trained model that incorporates additional signals such as product reviews, promotions, and user-specific data like browsing and purchase history, to offer an even more comprehensive search experience.

Once this model is defined in Spanner, we can proceed to add reranking on the initial search results using the following query:

Explanations:

• ANN, FTS and RRF CTEs: These are the same previously defined approximate nearest neighbors, full-text search and reciprocal rank fusion queries, respectively.

• ML.PREDICT Ranking: This step applies the reranker model to each product description as text from the RRF results, along with the search query as text_pair. The model assigns a relevance score to each product. The products are then sorted by these scores, and the top 10 are selected.

Get started

In this post, we demonstrated one approach to combine full-text search and vector search in Spanner, but developers are encouraged to explore other approaches, such as refining full-text search results with vector search, or combining multiple search results with customized fusion methods.

Learn more about Spanner and try it out today. For additional information, check out:

• Spanner vector search documentation and codelab

• Spanner full-text search overview

• Spanner vertex AI integration overview and tutorial on generating ML predictions using SQL

For enterprises, brilliance isn’t just about individual genius – it’s about the collective intelligence within an organization. But this brilliance is often hidden in silos, inaccessible to those who need it most, when they need it most. Our studies show that enterprise workers use an average of four to six tools just to ask and answer a question¹.

Generative AI holds immense promise for employee productivity. That’s why today, we’re introducing Google Agentspace. It unlocks enterprise expertise for employees with agents that bring together Gemini’s advanced reasoning, Google-quality search, and enterprise data, regardless of where it’s hosted. Google Agentspace makes your employees highly productive by helping them accomplish complex tasks that require planning, research, content generation, and actions – all with a single prompt.   

Three ways in which Google Agentspace unlocks enterprise expertise:

1. New ways to interact and engage with your enterprise data using NotebookLM: We built NotebookLM to help users make sense of complex information, and now we’re bringing this capability to enterprises. With NotebookLM Plus, your employees can upload information to synthesize, uncover insights, and enjoy new ways of engaging with data, such as podcast-like audio summaries and more. It’s the same experience millions of users love, now enhanced with security and privacy features for work. We’re also starting to roll out the experimental version of Gemini 2.0 Flash in NotebookLM. You can sign up for the early access program for NotebookLM for enterprise today. 

2. Information discovery across the enterprise: Google Agentspace gives employees a single, company-branded multimodal search agent that acts as a central source of enterprise truth for your entire organization. Building on the best of Google’s search capabilities, Agentspace can provide conversational assistance, answer complex questions, make proactive suggestions, and take actions based on your company’s unique information. Google Agentspace can do this across unstructured data – such as documents and emails – and structured data such as tables. We’ve also built translation in, so that you can understand all information, even if it originates in another language. With pre-built connectors for the most commonly used third-party applications, such as Confluence, Google Drive, Jira, Microsoft SharePoint, ServiceNow, and more, your employees can now easily access and query relevant data sources, and make better decisions.

3. Expert agents to automate your business functions: Google Agentspace is the launch point for custom AI agents that apply generative AI contextually. Now, enterprises can empower their employees in marketing, finance, legal, engineering, and more to conduct better research, draft content quickly, and automate repetitive tasks, including multi-step workflows. With Google Agentspace, enterprises can scale AI by providing a single location where employees can easily discover and access agents for their organization. Soon, employees will be able to use a low-code visual tool to build and tune their own expert agents with Google Agentspace.

With Google Agentspace, business analysts can effortlessly uncover industry trends and create compelling, data-driven presentations fueled by AI-generated insights. HR teams can revolutionize the employee experience with streamlined onboarding, even for complex tasks like 401k selection. Software engineers can proactively identify and resolve bugs, enabling them to build and iterate with greater efficiency, and accelerate deployment cycles. Marketers can unlock deeper performance analysis, optimize content recommendations, and fine-tune campaigns to achieve better results. And with multi-language support, all employees – no matter their region – can unlock enterprise expertise with agents. Sign up for the early access program. 

Finally, security is always top of mind. Google Agentspace is built on Google Cloud’s secure by design infrastructure, giving you the peace of mind to confidently deploy AI agents across your organization. Plus, we provide granular IT controls, including role-based access control (RBAC), VPC service controls, and IAM integration, ensuring your data remains protected and compliant at all times. 

Hear from our customers 

Our enterprise customers are already seeing great promise around connecting their employees with the wealth of their enterprise’s information. 

“At Deloitte, knowledge fuels our work. Our knowledge management teams are impressed with Google Agentspace’s ability to unify information scattered across various data silos. With Agentspace, our professionals can find the information they need instantly, boosting productivity and efficiency. Beyond search, the AI agentic capabilities within Google Agentspace can empower teams to take action on this knowledge, enabling us to deliver solutions and achieve client outcomes faster. In the long term, we see Agentspace as a key driver in transforming how we serve clients, fostering deeper collaboration, and unlocking new levels of insight and innovation across our organization.” – Kevin Laughridge, Alphabet Google US Lead Alliance Partner, Deloitte Consulting LLP 

“At Nokia, we create technology that helps the world act together. to connect and improve the world. Google Agentspace has the potential to revolutionize how our teams across Nokia find and leverage critical insights. We’re particularly excited by Google Agentspace’s ability to blend various data sources quotes and deliver personalized, contextually relevant answers. By unifying our knowledge resources, providing AI powered assistance and automating workflows, we strive towards reduced time spent searching for information, faster decision-making, and improved collaboration and productivity.” – Alan Triggs, Chief Digital Officer, Nokia 

“At Decathlon, we’re driven by a passion for sports and a commitment to innovation. Google Agentspace’s ability to connect teams with the right information from across organizations and provide self-service assistance is incredibly promising. Google Agentspace could become an essential enabler for product designers, marketers, and researchers, enabling them to make faster, more informed decisions and ultimately delivering even better experiences for customers. Our cross-functional teams — from data to workplace tools — see it as a great promise to meet our needs,” – Youssef BAKKALI, Group Product Manager, Decathlon

“At Onix we help businesses harness the power of technology to drive innovation and growth. Google Agentspace has immense potential to transform how enterprises deploy agents and access enterprise data at scale. By connecting disparate data sources and leveraging Google’s advanced AI for agentic workflows, Google Agentspace empowers organizations to unlock new levels of creativity. These last few months Onix has been partnering with Google to onboard early customers, and we are excited to see our clients take their first steps on this transformative journey.” – Ramnish Singh, SVP Solutions Engineering, Onix 

“Banco BV is committed to providing our team with the most innovative tools and technologies for collaboration. Google Agentspace is helping us realize the vision of enabling our employees to leverage some of the most advanced generative AI technologies for search, assistance and actions across all our critical systems in a secure and compliant way. We are particularly excited to see how our early users are leveraging Google Agentspace for faster and more comprehensive analysis and engaging with content in new multimodal ways that reflect our vision of work as more relaxed and collaborative.” – Fabio Jabur, Head of Data and AI, Banco BV 

“Google Agentspace has the potential to redefine how enterprises leverage knowledge, particularly through the power of AI agents. We’re excited to partner with Google Cloud to bring this to our customers, enabling them to securely use generative AI to enhance productivity, streamline workflows, and drive innovation. Google Agentspace’s unique ability to personalize search, provide AI assistance, and leverage innovations like NotebookLM will revolutionize how knowledge workers discover, generate, and utilize information. We believe that agents represent the future of knowledge work, and Google Agentspace is at the forefront of this exciting evolution.” – Gaurav Johar, Head of Applied AI, Quantiphi

“Our team of over 12,000 dedicated staff is united by a shared purpose: to make every day a little better for our customers by keeping daily essentials within reach for all. As Singapore’s largest food retailer, we understand that our people are the key to delivering on our social mission. To empower them with the tools they need to put their best foot forward every day, we are building an organization-wide research and assistance platform with Google Agentspace, with the aim of reducing manual effort. Google Agentspace will equip our employees with both a seamless research assistant that searches across all our documents, internal systems, and even third-party applications, and a task assistant using natural language. Our ultimate goal is to improve knowledge discovery and streamline processes across our enterprise. By enabling our people to access and share knowledge effortlessly, we can foster improved customer experiences and enhanced operational efficiency.” – Dennis Seah, Chief Digital & Technology Officer, FairPrice 

Hear from users at Deloitte Consulting LLP

“My work involves various market analyses as a strategy professional. It normally takes us a couple of weeks to read through all the research material. NotebookLM allowed us to get the initial insights in minutes and spend the time going deeper. It helps my team collaborate from a single source, pooling notes and links, freeing up more of our research time for strategic thinking and brainstorming allowing us to deliver compelling insights to our clients.” – Parinda Gandevikar, Senior Consultant, Deloitte Consulting LLP  

“NotebookLM’s podcast uncovered hidden insights in our GenAI report, revealing differences between industry sectors. Despite having read it multiple times, I had not caught this nuance. It was a real ‘aha’ moment.” – Laura Shact, Human Capital AI Lead, Deloitte Consulting LLP

“It’s a game changer for drawing insights from meetings. We took the recordings and uploaded it to NotebookLM and it did an amazing job of synthesizing the key findings.” – Tom Cahill, Senior Consultant, Deloitte Consulting LLP

“Before a long drive, I upload earnings call transcripts of recent tech companies and listen to the audio overview on my drive. It’s great how the key insights are distilled and interwoven with broader context that’s not even in those documents.” – Vivek Kulkarni, GenAI Transformation Lead, Deloitte USA LLP 

“I run a practice which involves creating and sharing a lot of different types of enablement content for the practice. It was delightful for our team to go from paging through dozens of documents to quickly finding the information we need, with the source document just one click away in NotebookLM.” – Gopal Srinivasan, Alphabet Google AI Alliance Lead, Deloitte Consulting LLP

Get started

Unlocking your organization’s collective brilliance is an investment in its future success. By connecting employees to enterprise expertise, you can create a more informed, innovative, and agile organization. 

Learn more about Google Agentspace and sign up for early access today.

^{1. Source: Survey of Knowledge Worker Communication Needs, Google Inc. (2024)}

Written by: Muhammad Umair

Here at Mandiant FLARE, malware reverse engineering is a regular part of our day jobs. At times we are required to perform basic triages on binaries, where every hour saved is critical to incident response timelines. At other times we examine complicated samples for days developing comprehensive analysis reports. As we face larger and more complex malware, often written in modern languages like Rust, knowing where to go, what to look at, and developing a “map” of the malware forms a significant effort that directly impacts our response times and triage effectiveness.

Today we introduce a new tool, XRefer (pronounced eks-reffer), which aims to shoulder some of this burden for anyone who endeavors to go down these rabbit holes like us, helping analysts get to the important parts faster while maintaining the context of their investigation.

Introduction

XRefer provides a persistent companion view to assist analysts in navigating and understanding binaries. It’s a modular and extensible tool that comes in the form of an IDA Pro plugin. Figure 1 shows the XRefer interface.

At its core, XRefer offers two complementary navigation paradigms:

1. Gemini-powered cluster analysis, which decomposes the binary into functional units and leverages the large language model (LLM) to describe their purpose and relationships. Think of this like viewing a city from Google Maps: you can quickly identify the business districts, residential areas, and green spaces. In binary terms, this feature helps identify functional groupings like command-and-control communication, persistence mechanisms, or information-gathering routines, giving you a strategic view of the malware’s architecture.

2. A context-aware view, which dynamically updates based on your current location in the code. This view presents both immediate artifacts of the current function and those from related functions along the same execution path. It’s similar to standing outside a shopping mall with X-ray vision: without entering each store, you can see the restaurant menus, shop inventories, and services offered on each floor. This allows you to make informed decisions about which areas deserve deeper investigation. Just as a mall directory helps you efficiently plan your shopping route, XRefer’s context-aware view helps analysts quickly identify relevant code paths by surfacing APIs, strings, capa matches, library information, and other artifacts that might otherwise require manual exploration of multiple functions. 

Let’s take a closer look at each of these paradigms, beginning with cluster-based navigation.

The Birds-Eye View: Cluster-Based Binary Navigation

One of XRefer’s key features is its ability to break down a binary into functional units, providing an immediate high-level understanding of its architecture. To demonstrate this capability, let’s examine an ALPHV ransomware sample written in Rust. Despite containing over 2,700 functions, XRefer’s analysis organizes key functionality of this complex binary into clear functional clusters, as shown in the Cluster Relationship graph in Figure 2.

These functional clusters are descriptively labelled as follows:

• Ransomware Main Module

• Configuration Parsing Module    

• User Profile and Process Information Module

• Privilege Escalation, System Information, and AntiAnalysis Module

• File Processing Pipeline Module

• Network Communication and Cluster Management Module

• Thread Synchronization and Keyed Events Module

• File Path and Encryption Key Generation Module

• Console Clearing Module

• UI Rendering and Console Output Module

• Image Generation and Encoding Module

• Data Encoding and Hashing Module

• File Discovery and Dispatch Module

• Thread Synchronization and Time Management Module

While each cluster contains deeper sub-clusters that analysts can explore, we’ll focus on the high-level view for now. The clustering and relationship identification is performed through static analysis. XRefer then leverages Gemini to provide natural language descriptions of each cluster and how they relate to one another. Figure 3 illustrates key components in the graph navigation interface.

At the top, the view provides a brief description of the binary’s functionality and its category. Next, it describes the currently selected cluster and its relationships to other clusters. For convenient navigation, the cross-references of that cluster are listed, followed by a visual graph representation. For readability, these details are transcribed in Table 1.

The clusters can also be viewed in a linear format within XRefer’s interface, as shown in Figure 1.

To better demonstrate cluster navigation visually, we’ve used a lightweight backdoor that displays more clearly on screen. Figure 4 provides a quick glimpse of the cluster navigation workflow, showing how analysts can quickly browse clusters and navigate to their respective functions in the disassembly or pseudocode views.

The navigation can automatically sync with clusters—when you navigate to a function that belongs to a known cluster, XRefer can automatically open that cluster’s view and highlight the current function within it. XRefer offers two approaches to clustering:

1. Cluster all paths that are part of XRefer’s analysis (XRefer’s analysis is discussed later)

2. Cluster a focused subset of functions, pre-filtered by Gemini based on their artifacts

Note: Throughout this blog post, we use the term “artifacts” to refer to binary elements like strings, API calls, library references, and other extractable information that help understand program behavior.

By default, XRefer employs the first method. While this approach is comprehensive, it may create additional clusters around unidentified libraries in the program. These library clusters are typically easy to identify and exclude from analysis.

The second clustering method is optionally available via the context menu and proves valuable for automatically filtering out library, runtime/compiler artifacts, and repetitive noisy functions. However, due to the inherent nature of LLMs, this approach can be inconsistent—artifacts might be missed, and results can vary between runs. While missed artifacts can usually be recovered through a quick re-run of the LLM analysis, this variability remains an inherent characteristic of this approach.

XRefer can also display these LLM-filtered artifacts in a dedicated view, separate from the clustering visualization. This view, shown in Figure 5, provides analysts with a streamlined overview of the binary’s most relevant artifacts while filtering out noise like library functions and runtime artifacts.

It’s important to note that clusters aren’t perfect boundaries. They may not capture every related function and can contain functions that are reused across different parts of the binary. However, any missed related functions will typically be found in the vicinity of their logical cluster, and reused functions are generally easy to identify at a glance. The goal of clustering is not to create strict divisions, but rather to establish general zones and subzones of related functionality.

Function Labeling: Prefixing Cluster Membership in Names

XRefer can optionally encode cluster information directly into IDA’s interface by prefixing function names. These labels provide architectural context directly inside the disassembly and pseudocode windows. Table 2 shows the classification system used to prefix functions based on their cluster relationships.

Down in the Trenches: Context-Aware Code Navigation

Having seen how XRefer’s cluster analysis provides a high-level view of binary architecture, let’s examine its second navigation paradigm: a context-aware view that updates automatically based on the function currently being analyzed.

The function context table (shown in Figure 6) organizes information into three main components:

1. Cluster Membership – At the top, displaying which clusters the current function belongs to. Functions appearing in multiple clusters often indicate utility or helper functions rather than specialized functionality.

2. Direct References – Listed under “DIRECT XREFS,” showing artifacts directly used or called by the current function.

3. Indirect References – Categorized tables prefixed with “INDIRECT,” showing artifacts used by all functions called through the current function’s execution paths. This provides a preview of downstream functionality without requiring manual traversal of each function.

Both direct and indirect references include:

• APIs and API traces

• Strings

• Libraries

• capa results

For direct references, each artifact is listed with its reference addresses. Double clicking these addresses jumps to their exact location in the current function. For indirect artifacts, the displayed addresses are different—they point to function calls within the current function that eventually lead to those artifacts through execution paths. This x-ray-like capability eliminates the tedious process of diving into nested function calls to discover artifacts and functionality, only to return to the primary function.

XRefer extends this visibility through its Peek View feature, accessible via the context menu. When enabled, clicking on any function dynamically filters the artifact view to display only those elements that lie along its execution paths. This instant preview allows analysts to quickly assess a function’s downstream behavior without manually tracing through its call graph, significantly streamlining the exploration of complex codebases. Figure 7 demonstrates how this functions in practice.

Beyond Peek View, XRefer offers on-demand artifact filtering through key bindings. A core design principle of XRefer is the uniform treatment of all artifact types—any operation available for one type of artifact is consistently available across all others. For instance, path analysis capabilities that work with API references can be similarly applied to capa results. Let’s examine the key functionalities available under this navigation paradigm.

Path Graphs

XRefer can generate and visualize all simple paths from the entry point to any type of artifact. These interactive graphs serve as powerful navigation tools, particularly when analysts need to trace specific functionality through the binary. Figure 8 demonstrates this capability by displaying all execution paths leading to GetComputerNameW in the ALPHV ransomware sample. Each function node in the graph provides a contextual pop-up showing its complete artifact inventory.

Path graphs include a simplification feature that can reduce visual complexity by omitting nodes that either contain no artifacts or contain only excluded artifacts (exclusions are discussed later). Figure 10 illustrates this simplification, where the graph is reduced from 15 to 12 nodes, representing a 20% reduction in complexity. While more complex graphs can achieve higher simplification ratios, their full visualization extends beyond the practical constraints of this blog post.

Search

XRefer provides unified search functionality across all artifact types directly within the function context table view. Figure 8 demonstrates this capability while searching for the API reference used in path graph generation.

Cross References++

XRefer implements its own cross-reference view that goes beyond IDA Pro’s traditional functionality (accessed via “X”). This view, similarly triggered by pressing “X” on any artifact, encompasses all artifact types, including elements that IDA Pro cannot typically track, such as capa results, APIs identified through dynamic traces, and strings extracted by language-specific modules like the Rust parser.

Trace Navigation

While API traces are integrated throughout XRefer’s interface—from function context tables (Figure 6) to information pop-ups (Figure 8)—the plugin also offers dedicated trace navigation modes. These three modes, illustrated in Figure 11, provide views of API calls with their arguments, each offering different levels of scope:

• Function Scope – Shows only the API calls made directly within the current function, providing a clean view of its immediate external interactions

• Path Scope – Reveals all API calls that occur downstream from the current function, following its execution paths. This helps analysts understand the complete chain of system interactions triggered by a particular function.

• Full Trace – Displays the complete API trace captured during dynamic analysis, regardless of static code paths. Useful when you may have calls associated with encrypted regions in the binary or generally from dynamically resolved APIs.

Artifact Exclusion

XRefer supports artifact exclusion to reduce noise when analyzing large, complex binaries. Excluded artifacts are omitted from multiple views and processes including the main interface, cluster analysis, and simplified path graphs.

Artifacts can be excluded in two ways:

• Directly through XRefer’s interface, where multiple artifacts can be selected and excluded using key bindings

• Via the settings dialog (shown in Figure 12), which supports wildcard-based exclusion patterns. For instance, noisy Rust standard library references can be filtered using patterns like std*, providing efficient bulk exclusion of known noise sources.

These exclusions persist across sessions, allowing analysts to maintain their preferred filtering setup.

Specialized Rust Support

XRefer has a specialized Rust module (discussed later) that extracts strings and library usage information. During Rust compilation, the compiler embeds library source paths that typically appear adjacent to their corresponding library code within functions. These compiler-inserted references serve two key purposes as function artifacts:

• Identifying library dependencies and their specific functionality within code sections

• Providing positional hints that help locate where library code is actually implemented within functions

This compiler-provided context feeds into both XRefer’s cluster analysis and enhances manual navigation, helping analysts quickly locate relevant code regions while understanding which Rust library implementations are being used. The module also includes a basic function renaming capability for Rust binaries, which will be covered in detail later.

Auxiliary Features

Before concluding this section, two standalone features warrant mention, though they operate independently of the clustering mechanism and exclusion system.

Boundary Method Scanning: XRefer allows multiple artifacts to be selected in the function context view for boundary scanning. This operation identifies the Lowest Common Ancestor (LCA) in the call graph for all selected artifacts. In niche scenarios, this can be used to isolate specific functionality based on a subset of artifacts and identify the most specific parent function that encompasses all selected artifacts without intermediate functions. While originally intended to serve a larger purpose in the clustering mechanism, the clustering system ultimately took a different direction, leaving this as a standalone feature.

String Lookups: During processing, XRefer can optionally query strings against public Git repositories through Grep App for categorization purposes. This is strictly a placeholder implementation; locally maintained databases by teams or individuals would be better suited for these queries. The feature operates independently of XRefer’s broader ecosystem, primarily serving to categorize known strings for noise reduction or occasional OSINT insights.

Under the Hood: XRefer’s Analysis Engine

Having explored both navigation paradigms, let’s examine the technical foundation that makes them possible. While a deep dive into XRefer’s internals would warrant its own blog post, understanding a high-level view of its analysis pipelines and extensibility will help set the context.

Ingestion Sources

XRefer builds its understanding of binaries through two primary data ingestion channels:

1. Internal – Whereby it extracts and processes all the imports, strings and any library data from the binary itself. This also involves language-specific modules. XRefer comes out of the box with a Rust-specific language module, and more can be added.

2. External – This includes API traces from third-party tooling and results from capa’s analysis. XRefer currently supports ingestion of API traces from VMRay’s sandbox and Cape Sandbox. Additional modules can be written, for instance, TTD traces would be a good candidate.

Note: Out of the mentioned traces, VMRay produces the best results. Cape sandbox, while good, hooks NT* APIs and is much noisier in terms of visual display as it loses 1:1 API to import mapping. Unfortunately, that’s one of the very few open-source solutions available right now.

XRefer feeds the available data along with cluster relationships in the form of call flows to the LLM, which generates semantic descriptions for each cluster, their relationships, and the overall binary. As demonstrated in Table 3, while external data sources enhance the analysis, XRefer can produce meaningful results even with just internal binary analysis.

Note: While LLM-generated labels and descriptions may vary in phrasing between runs, they tend to consistently convey similar semantic information.

Language Modules and Rust

XRefer supports language-specific analysis through dedicated modules and ships with a module for Rust binaries. This allows for specialized handling of language-specific characteristics. The Rust module provides:

• Identification of rust_main 

• Parsing of Rust thread objects and their indirect calls, improving path coverage

• Extraction of library/crate information for better code understanding in both cluster analysis and context tables

• Limited function renaming capabilities for a subset of functions where no inlining conflicts are detected, based on compiler strings and excluding runtime/std/internal libraries

Rust Function Renaming

The Rust module includes a limited function renaming capability for Rust binaries, though its approach is deliberately restrictive. While XRefer is not a function identification tool, it can leverage compiler strings embedded in Rust binaries for basic renaming. Due to the prevalence of inlining from compiler optimizations, the module only renames functions where no inlining is detected and excludes internal references (std*, core*, alloc*, etc.) which are particularly prone to inlining.

The renaming doesn’t provide the full function names but does include up to the submodule name. Again, this is not a substitute for proper function identification as only a very small handful of functions can be renamed this way. However, since this capability is not version-specific to the toolchain, platform, and/or crate, it represents a low-hanging fruit that would have been unwise to ignore. As an example, in the ALPHV binary, the Rust module was safely able to rename 362 out of ~2,700 functions.

LLM Analysis and Extensibility

XRefer ships with support for Gemini through a modular provider system. The architecture is designed for extensibility—new LLM providers can be added by implementing a simple interface that handles model-specific requirements like rate limiting and token management. It should be noted that cluster analysis, especially for large binaries, requires large context windows, an area in which Gemini models excel compared to other models.

Similarly, the prompt system is built to be extensible. New prompts can be easily added by creating prompt templates and corresponding response parsers. This allows XRefer’s analysis capabilities to grow as new use cases are identified or as LLM technology evolves.

It’s important to note that XRefer currently limits its LLM analysis to artifacts and function/cluster relationships (in the form of call flows) without submitting any actual code. For example, when analyzing a network communication module, XRefer provides the LLM with a rich set of artifacts like API calls (send, recv), strings (URLs, User-Agents), dynamic API traces, capa matches (network communication, socket operations), library/crate information, and their relationships in the call graph, rather than the underlying implementation code. This is just one simplified example. The actual analysis encompasses the full spectrum of artifacts XRefer collects. 

The effectiveness of this approach depends on the successful extraction of these artifacts, whether through specialized language modules, internal binary analysis, or ingestion of external data sources. When artifacts are available, the Gemini-powered analysis effectively breaks down binary functionality into distinct functional units, providing an explorable architectural view of the binary. Code-level analysis represents the next logical step in XRefer’s evolution.

Path Analysis: The Foundation of XRefer’s Navigation

XRefer’s architecture is fundamentally entrypoint-centric. It constructs execution paths between entry points and functions containing artifacts, forming the backbone of both its clustering algorithm and the context-aware navigation capabilities described earlier. While clustering can be implemented without path analysis, our current approach of path-based clustering consistently produces decent results. Standalone clustering may come as a feature later down the road.

While entry point selection for PE executables is straightforward and automatic, DLL analysis may require analysts to select specific exports as starting points, since the default entry point might not be the most interesting one. XRefer allows analysts to analyze multiple entry points, providing flexibility in how they approach the binary.

Path analysis introduces computational overhead, but the benefits it provides to both navigation capabilities and clustering accuracy make this trade-off worthwhile. It’s important to note that this overhead is entirely front-loaded into the initial processing phase. Once analysis is complete, the pre-processed results ensure fluid navigation and responsiveness during actual usage. Table 4 shows what analysis times look like for several large binaries.

Note: Times include LLM queries for artifact discovery and cluster analysis. Actual duration varies based on system capabilities and binary complexity—from seconds for simple binaries to longer for complex ones. This listing aims to provide a general sense of expected time scales.

A notable limitation of path analysis arises in binaries with numerous indirect calls, where complete coverage cannot be guaranteed without proper resolution of these indirect targets.

Configuration

All LLM configurations and paths for external data sources can be managed through XRefer’s settings dialog.

XRefer supports the ingestion of indirect cross-references that IDA cannot resolve statically. Examples can include dynamically resolved addresses for indirect calls, such as virtual function calls through vtables in C++ binaries and function pointer calls. These resolutions can be imported from sources like debugger scripts or binary instrumentation frameworks, enabling XRefer to construct more complete execution paths.

UI/UX and Compatibility

XRefer is implemented as an interactive TUI (Text-based User Interface) plugin. While IDA’s simplecustviewer_t wasn’t designed to be a proper TUI system, additional Qt implementations have been added to improve the interface. Users may encounter bugs, which will be addressed as they are reported.

Some rules of thumb when working with XRefer:

• All addresses, regardless of where they are displayed in the TUI, are navigable by double clicking.

• Cluster IDs (cluster.id.xxxx), regardless of where they show up in the TUI, are also explorable by single clicking.

• If an address is the start of a function, hovering over it will always display an information pop-up about that function.

• Hovering over cluster IDs will display a pop-up with details about that cluster.

• Press ESC to return to the previous location/state (unless a graph is explicitly “pinned”).

XRefer maintains state within the current function and resets upon navigation to a new function. For detailed usage instructions and key bindings, please refer to the XRefer repository.

It is recommended to enable auto-resizing, which automatically adjusts XRefer’s window dimensions when viewing graphs and restores default size upon exit. While XRefer is designed to remain open as a persistent companion view, it includes an expand/collapse feature for quick access when needed. Figure 17 demonstrates these interface elements.

XRefer is recommended for use with either IDA <= 8.3 or IDA >= 9.0. IDA 8.4 contains a simplecustviewer_t visual bug that causes washed-out colors in several areas. Due primarily to the author’s preference for dark mode, the plugin currently provides better color contrast in dark themes compared to the default theme, though this may be balanced in future releases.

XRefer has been primarily tested with Windows binaries. While there are no fundamental limitations preventing support for ELF or Mach-O formats and XRefer may just work out of the box with most of them, some tweaks or implementation fixes might be required to ensure proper support.

How to Get XRefer

XRefer is now available as an open-source tool in Mandiant’s GitHub repository. Installation requires installing Python dependencies from requirements.txt and copying the XRefer plugin to IDA’s plugin directory.

Note that one of the dependencies, asciinet, requires Java (JRE or OpenJDK) to be configured on your system. For detailed installation instructions, please refer to the XRefer repository.

Another alternative is to use FLARE-VM, which sets up a reverse engineering environment with a lot of useful tools, including XRefer.

Future Work

This is the initial release of XRefer and thus includes some implementations that are early in their maturity. While LLMs may eventually evolve to accurately interpret all forms of source and compiled code, the current approach focuses on systematic analysis rather than treating LLMs as a black box for binary summarization. This methodology not only provides high-level insights but actively supports analysts in their detailed investigation workflows.

Immediate areas for future development include, beyond bug fixes and UI/UX refinements:

• Extend cluster analysis to include code submissions, improving not just analysis at scale but also providing targeted insights for manual reverse engineering workflows

• Research and potentially implement path-independent clustering methodologies (the primary benefit here would be speed improvements if path analysis is not required)

• Implement LLM-based cluster merging (helps neatly tuck away similar clusters such as those belonging to a library)

• Ensure proper support for non-Windows file formats

• Add support for other language modules, particularly Golang

Currently, XRefer is tightly coupled with IDA due to its TUI implementation. As the core matures, the processing engine may be decoupled into a standalone package.

Acknowledgements

Special thanks to Genwei Jiang and Mustafa Nasser for their code contributions to XRefer and to Ana Martinez Gomez for including XRefer in the default FLARE-VM configuration. Additional thanks to the FLARE team members who provided valuable feedback through their use of XRefer.

The AI phase of industrial evolution is marked by a profound transformation in how humans and intelligent machines collaborate. The blurring of boundaries between physical and digital systems across the manufacturing landscape is accelerating, driven by advancements in automation, robotics, artificial intelligence, and the Internet of Things. 

This interconnectedness creates unprecedented opportunities for efficiency, innovation, and customized production. However, it also exposes manufacturers to a new generation of cyber threats targeting industrial operations, supply chains, and increasingly-sophisticated production processes. Safeguarding these critical assets requires a holistic approach that transcends traditional boundaries and embraces sector-wide collaboration. 

To enhance our commitment to the manufacturing and industry sector, today we are announcing a new partnership with the Global Resilience Federation (GRF) by joining four of its affiliate groups: the Business Resilience Council (BRC), the Manufacturing Information Sharing and Analysis Center (MFG-ISAC), the Operational Technology Information Sharing and Analysis Center (OT-ISAC), and the Energy Analytic Security Exchange (EASE). Google Cloud is proud to be the first cloud service provider to partner with the GRF Business Resilience Council and its affiliates.

Through this partnership, Google Cloud will strengthen its commitment to the manufacturing industry by providing critical expertise and advanced security solutions. Our collaboration with industry leaders will focus on fortifying the resilience of manufacturing systems and supply chains against evolving cyber threats. This partnership underscores our dedication to supporting the manufacturing sector’s digital transformation and modernization while ensuring the security and integrity of critical infrastructure. 

In today’s interconnected world, safeguarding your organization demands a comprehensive strategy that goes beyond traditional measures. Google Cloud will devote resources and experts to work alongside industry leaders to transform, secure, and defend the Manufacturing  sector and will contribute to the manufacturing companies through a network of resources and expertise spanning IT, OT, industrial operations technology, supply chain, logistics, engineering technology, and product security, specifically designed to navigate the complexities of Industry 4.0 and 5.0.

This collaboration among professionals in cyber and physical security, geopolitical risk, business continuity, disaster recovery, and third-party risk management is critical for organizations with regional, national, and international footprints. In an era where the severity of cyber threats is constantly increasing, resilience is key. Partnerships fostered by GRF provide the knowledge and support necessary to maintain vigilance, manage crises, and navigate response scenarios to enable continuity of your operations.

As a GRF partner and a member of these four groups, Google Cloud will bring experts and resources — including unique insights from Mandiant, our Threat Horizon reports, and the Google Cloud Office of the CISO— to help the manufacturing and industry sector protect against cyberattacks. Google will work with defenders and sector leaders to share knowledge we’ve learned building and deploying secure technology.

This partnership is a continuation of our August 2021 commitment to invest at least $10 billion over five years to advance cybersecurity. This same commitment has enabled us to join other organizations including Health ISAC, Financial Services ISAC, and Electricity ISAC, so we can continue to support the security and resilience of our critical infrastructure across key sectors.

“Partnering with GRF and becoming a member of its affiliated groups BRC, MFG-ISAC, OT-ISAC, and EASE is a critical step in our commitment to help the manufacturing and industrial sectors transform and secure their critical infrastructure,” said Phil Venables, VP and CISO, Google Cloud. “As a leading provider of cloud technologies and security solutions, we recognize the vital role these sectors play in driving economic growth and innovation. This partnership aligns with our dedication to supporting the modernization and resilience of manufacturing and industrial operations in the face of evolving cyber threats. By sharing our expertise and collaborating with industry leaders, we aim to raise awareness, develop innovative solutions, and strengthen the collective defense of these essential industries.”

“As a provider of innovative technology solutions, we recognize the vital role of the manufacturing and industrial sectors in driving our economy. This partnership reflects our commitment to supporting their transformation and strengthening their defenses against evolving cyber threats. Through collaboration and knowledge-sharing, we aim to foster a more secure and resilient future for these essential sectors,” said Nick Godfrey, senior director and global head, Office of the CISO, Google Cloud.

“Phil Venables and Google Cloud have long advocated for collaborative security and collective resilience, and their active role in the BRC and these communities brings invaluable expertise to help build a more secure ecosystem for businesses of all sizes — including their critical vendors and suppliers,” said Mark Orsi, CEO, GRF. “Google Cloud continues its leadership in advancing security and operational resilience across manufacturing, utilities, industrial, and critical infrastructure sectors — ultimately fostering a safer and more sustainable global supply chain.”

Your business data sets you apart from the competition. It fuels your innovations, your culture, and provides all your employees a foundation from which to build and explore. Since 2022, enterprises in all industries have turned to Looker Studio Pro to empower their businesses with self-service dashboards and AI-driven visualizations and insights, complete with advanced enterprise capabilities and Google Cloud technical support.

As the Looker community has grown, we’ve gotten more requests for guidance on how users can make their Looker Studio Pro environments even stronger, and tap into more sophisticated features. Those requests have only increased, accelerated by the debut of Studio in Looker, which brings Looker Studio Pro to the broader Looker platform. To help, today we are debuting a new on-demand training course: Looker Studio Pro Essentials.

Looker Studio Pro connects businesses’ need to govern data access with individual employees’ needs to explore, build and ask questions. This Google Cloud Skills Boost course helps users go beyond the basics of setting up reports and visualizations, and provides a deep dive into Looker Studio Pro’s more powerful features and capabilities.

Here’s what you can expect to get from this course:

• Gain a comprehensive understanding of Looker Studio Pro: Explore its key features and functionality, and discover how it elevates your data analysis capabilities.

• Enhance collaboration: Learn how to create and manage collaborative workspaces, streamline report sharing, and automate report delivery.

• Schedule and share reports: Learn how to customize scheduling options to your business, including delivery of reports to multiple recipients via Google Chat and email, based on your sharing preferences.

• Ensure data security and control: Become an expert in user management, audit log monitoring, and other essential administrative tasks that can help you maintain data integrity.

• Leverage Google Cloud customer care: Learn how to use Google Cloud Customer Care resources to find solutions, report issues, and provide feedback.

From your focus, to your employees, to your customers, your business is unique. That’s why we designed this course to bring value to everyone — from sales and marketing professionals, to data analysts, to product innovators — providing them with the knowledge and skills they need to fully leverage Looker Studio Pro in their own environments. Because in the gen AI era, how you leverage your data and invigorate your employees to do more is the true opportunity. Accelerate that opportunity with the new Looker Studio Pro Essentials course today.

For developers and businesses that run applications on Google Kubernetes Engine (GKE), scaling deployments down to zero when they are idle can offer significant financial savings. GKE’s Cluster Autoscaler efficiently manages node pool sizes, but for applications that require complete shutdown and startup (scaling the node pool all the way to and from zero), you need an alternative, as GKE doesn’t natively offer scale-to-zero functionality. This is important for applications with intermittent workloads or varying traffic patterns. 

In this blog post, we demonstrate how to integrate the open-source Kubernetes Event-driven Autoscaler (KEDA) to achieve this. With KEDA, you can align your costs directly with your needs, paying only for the resources consumed.

Why scale to zero?

Minimizing costs is a primary driver for scaling to zero, and applies to a wide variety of scenarios. For technical experts, this is particularly crucial when dealing with:

• GPU-intensive workloads: AI/ML workloads often require powerful GPUs, which can be expensive to keep running even when idle.

• Applications with predictable downtime: Internal tools with specific usage hours — scale down resources for applications used only during business hours or specific days of the week.

• Seasonal applications: Scale to zero during the off-season for applications with predictable periods of low activity.

• On-demand staging environments: Replicate production environments for testing and validation, scaling them to zero after testing is complete.

• Development, demo and proof-of-concept environments:

  • • Short-term demonstrations: Showcase applications or features to clients or stakeholders, scaling down resources after the demonstration.

    • Temporary proof-of-concept deployments: Test new ideas or technologies in a live environment, scaling to zero after evaluation.

    • Development environment: Spin up resources for testing, code reviews, or feature branches and scale them down to zero when not needed, optimizing costs for temporary workloads.

• Event-driven applications:

• Microservices with sporadic traffic: Scale individual services to zero when they are idle and automatically scale them up when requests arrive, optimizing resource utilization for unpredictable traffic patterns.

• Serverless functions: Execute code in response to events without managing servers, automatically scaling to zero when inactive.

Introducing KEDA for GKE

KEDA is an open-source, Kubernetes-native solution that enables you to scale deployments based on a variety of metrics and events. KEDA can trigger scaling actions based on external events such as message queue depth or incoming HTTP requests. And unlike the current implementation of Horizontal Pod Autoscaler (HPA), KEDA supports scaling workloads to zero, making it a strong choice for handling intermittent jobs or applications with fluctuating demand.

Use cases

Let’s explore two common scenarios where KEDA’s scale-to-zero capabilities are beneficial:

1. Scaling a Pub/Sub worker

• Scenario: A deployment processes messages from a Pub/Sub topic. When no messages are available, scaling down to zero saves resources and costs.

• Solution: KEDA’s Pub/Sub scaler monitors the message queue and triggers scaling actions accordingly. By configuring a ScaledObject resource, you can specify that the deployment scales down to zero replicas when the queue is empty.

2. Scaling a GPU-dependent workload, such as an Ollama deployment for LLM serving

• Scenario: An Ollama-based large language model (LLM) performs inference tasks. To minimize GPU usage and costs, the deployment needs to scale down to zero when there are no inference requests.

• Solution: Combining HTTP-KEDA (a beta feature of KEDA) with Ollama enables scale-to-zero functionality. HTTP-KEDA scales deployments based on HTTP request metrics, while Ollama serves the LLM.

Get started with KEDA on GKE

KEDA offers a powerful and flexible solution for achieving scale-to-zero functionality on GKE. By leveraging KEDA’s event-driven scaling capabilities, you can optimize resource utilization, minimize costs, and improve the efficiency of your Kubernetes deployments. Please remember to validate usage scenarios as scale to zero mechanism can influence workload performance. Scaling to zero can increase latency due to cold starts. When an application scales to zero, it means there are no running instances. When a request comes in, a new instance has to be started, increasing latency. 

There are also considerations about state management. When instances are terminated, any in-memory state is lost.

To help you get started quickly, we’ve published a guide for scaling GKE to zero with KEDA, with specific steps for scaling a Pub/Sub worker to zero with KEDA on GKE and scaling a Ollama to zero with KEDA on GKE. 

To learn more about KEDA and its various scalers, refer to the official KEDA documentation at https://keda.sh/docs/latest.

Dun & Bradstreet, a leading global provider of business data and analytics, is committed to maintaining its position at the forefront of innovation. For the past two years, this commitment has included the company’s deliberate approach to improving its software development lifecycle by infusing AI solutions. 

While development velocity and security were the company’s most pressing considerations, Dun & Bradstreet was also inundated with productivity and operational challenges common to many global enterprises which included: 

• Significant time onboarding new team members 

• Siloed knowledge of legacy codebases 

• Low test coverage

• Application modernization challenges

To achieve its goal of accelerating software development, Dun & Bradstreet knew it had to take a holistic “people, process, and tools” approach to solve the traditional development lifecycle issues that most enterprise engineering teams face. They looked to AI-assistance to anchor this new effort.

Finding a partner for the future of the software development lifecycle

As a provider of information that can move markets and drive economies, Dun & Bradstreet had a high bar for any technology tools, with demanding expectations as high as the financial professionals and government leaders they serve. 

Dun & Bradstreet executed a thorough evaluation process to identify the best partner and coding assistance tool, considering both open-source and commercial options. The company ultimately selected Gemini Code Assist due to the Gemini model’s performance, seamless integration with their existing development environment, and robust security features.

The implementation of Gemini Code Assist was a collaborative effort between Dun & Bradstreet’s development teams and the Google Cloud team. The developers who were part of the team were actively involved in the configuration and customization of the tool to ensure that it met their specific needs and workflows.

A key focus area for Dun & Bradstreet was Google’s security stance. Incorporating AI into the development process required both top-grade protection of private data and guardrails to ensure the safety of machine-generated code. Google’s security expertise and guidance allowed Dun & Bradstreet to move forward with confidence due to the following factors:

• Gemini models are built in-house, allowing Google to fully validate and filter all source code samples used in model training.

• Trust and verify: Integration into a company’s existing coding and review lifecycles allows developers to guide the model outputs with human oversight, without learning a whole new system.

• Google’s partnership with Snyk provides additional options for automated security scanning, covering both AI-generated and human-written code.

• Google’s AI Principles underpin the architecture and design decisions for Gemini Code Assist. Privacy and security protections include single-tenant storage of customer code references, encrypted logging, and fine-grained administrative controls to prevent accidental data leakage.

• Google’s indemnification policies.

“AI-assisted code creation is not just a leap forward in efficiency — it’s a testament to how innovation and security can work hand-in-hand to drive business success,” said Jay DePaul, chief cybersecurity technology risk officer at Dun & Bradstreet. “By embedding robust guardrails, we’re enabling our teams to build faster, safer, and smarter.”

Transformation in action

Dun & Bradstreet decided to move forward with Code Assist in October 2024. The solution is now starting to roll out to more teams across the organization. Adoption has been smooth, aided by Code Assist’s intuitive interface and comprehensive documentation.

Having a program for incubation at large organizations helps to iron out both technical and potential adoption blockers. For example, the Dun & Bradstreet team identified the need to educate teams on how coding assistants are there to help developers as a partner, not as replacements.

Now that the rollout is underway, Dun & Bradstreet is sharing the factors that drove their adoption of Gemini Code Assist.

• Increased developer productivity: Gemini Code Assist’s AI-powered code suggestions and completions have significantly reduced the time developers spend writing code. The tool’s ability to automate repetitive tasks has freed up time for the developers so they can focus on more complex and creative aspects of their work.

• Improved code quality: The automated code review and linting capabilities of Gemini Code Assist helped Dun & Bradstreet’s developers detect errors and potential issues early in the development process. This has led to a significant reduction in bugs and improved overall code quality.

• Easier application modernization: A significant amount of time was saved when converting Spring apps to Kotlin.

• Increased developer efficiency: Early internal indicators show a 30% increase in developer productivity.

• Developer onboarding: New developers at Dun & Bradstreet have been able to ramp up quicker due to the real-time guidance and support provided by Gemini Code Assist. 

• Enhanced knowledge sharing: Gemini Code Assist has fostered a culture of knowledge sharing within Dun & Bradstreet’s development teams. The tool’s ability to provide code examples and best practices made it easier for developers to learn from each other and collaborate effectively.

Leading the way with AI

Gemini Code Assist has proven to be a valuable solution for Dun & Bradstreet as it has empowered their developers with advanced AI capabilities and intelligent code assistance. 

“AI-assisted code creation is a game changer for everyone involved in the solution-delivery business,” said Adam Fayne, vice president for Enterprise Engineering at Dun & Bradstreet. “It enables our teams to innovate, test, and deploy faster, without having to risk security or quality.”

The company has been able to accelerate velocity, improve software quality, and maintain its competitive edge in the market. Companies like Dun & Bradstreet trust Google Cloud and Gemini to greatly enhance their software developer lifecycles. In fact, Google Cloud was recently named a Leader in the 2024 Gartner Magic Quadrant for AI Code Assistants for its Completeness of Vision and Ability to Execute.

Visit the Gemini Code Assist website for more information on use cases and how to start using it today.

Ford Pro Intelligence is a cloud-based platform that is used for managing and supporting fleet operations of its commercial customers. Ford commercial customers range from small businesses, large enterprises like United Postal Service and Pepsi where fleets can be thousands of vehicles, and government groups and municipalities like the City of Dallas. The Ford Pro Intelligence platform collects connected vehicle data from fleet vehicles to help fleet operators streamline operations, increase productivity, reduce cost of ownership, and improve their fleet’s performance and overall uptime through the alerts on vehicle health and maintenance. 

Telemetry data from vehicles provides a wealth of opportunity, but it also presents a challenge: planning for the future as cars and services evolve. We needed a platform that could support the volume, variety and velocity of vehicle data as automotive innovations emerge, including new types of car sensors, more advanced vehicles, and increased integration with augmented data sources like driver information, local weather, road conditions, maps, and more.

In this blog, we’ll discuss our technical requirements, the decision-making process, and how building our platform with Bigtable, Google Cloud’s flexible NoSQL database for high throughput and low-latency applications at scale, unlocked powerful features for our customers like real-time vehicle health notifications, AI-powered predictive maintenance, and in-depth fleet monitoring dashboards.

Scoping the problem

We wanted to set some goals for our platform based on our connected vehicle data. One of our primary goals is to provide real-time information for fleet managers. For example, we want to inform our fleet partners immediately if tire pressure is low, a vehicle requires brake maintenance, or there is an airbag activation, so they can take action. 

Connected vehicle data can be extremely complex and variable. When Ford Pro set out to build its vehicle telemetry platform, we knew we needed a database that could handle some unique challenges. Here’s what we had to consider:

• A diverse and growing vehicle ecosystem: We handle telemetry data from dozens of car and truck models, with new sensors added every year to support different requirements. Plus, we support non-Ford vehicles too!

• Connectivity isn’t a guarantee: A “connected” car isn’t always connected. Vehicles go offline due to spotty service or even just driving through a tunnel. Our platform needs to handle unpredictable or duplicated streams of time-series data.

• Vehicles are constantly evolving: Manufacturers frequently push over-the-air updates that change how vehicles operate and the telemetry data they generate. This means our data is highly dynamic, and our database needs to support a flexible, ever-evolving schema.

• Security is paramount: At Ford, we are committed to our customer’s data privacy and security. It’s imperative to our technology. We serve customers around the world and must ensure we can easily incorporate privacy and security measures while maintaining regulatory compliance, such as GDPR, in every country we operate.

These challenges, along with the application feature requirements, we knew that we needed an operational data store that can support low-latency access for both real-time and historical data with a flexible schema. 

Where we started

The Ford Pro Intelligence platform offers a diverse range of features and services that cater to the diverse needs of our customers. To ensure flexibility in data access, we prioritize real-time reporting of vehicle status, event-based notifications, location services, and historical journey reconstruction. These capabilities necessitate a variety of data access methods to support both real-time and historical data access — all while maintaining low latency and high throughput to meet the demands of Ford customers.

Our starting point was an Apache Druid-based data warehouse that contained valuable historical data. While Apache Druid could handle high-throughput write traffic and generate reports, it was not able to support our low-latency API requirements or high data volumes. As a result, we started working with Google Cloud to explore our options. 

We began our search with BigQuery. We already used BigQuery for reporting, so this option would have given us a serverless, managed version of what we already had. While BigQuery was able to perform the queries we wanted, our API team raised concerns about latency and scale — we required single-digit millisecond latency with high throughput. We discussed putting a cache layer in front of BigQuery for faster service of the latest data but soon discovered that it wouldn’t scale for the volume and variety of requests we wanted to offer our customers.

From there, we considered several alternative options, including Memorystore and PostgreSQL. While each of these solutions offered certain advantages, they did not meet some of our specific requirements in several key areas. We prioritized low-latency performance to ensure real-time processing of data and seamless user experiences. Flexibility, in terms of schema design, to accommodate our evolving data structures and wide column requirements was also a must. Scalability was another crucial factor as we anticipated significant growth in data volume and traffic over time. 

When we looked at Bigtable, its core features of scalable throughput and low latency made it a strong contender. NoSQL is an ideal option for creating a flexible schema, and Bigtable doesn’t store empty values, which is great for our sparse data and cost optimization. Time-series data is also inherent to Bigtable’s design; all data written is versioned with a timestamp, making it a naturally good fit for use cases with vehicle telemetry data. Bigtable also met our needs for an operational data store and analytics data source, allowing us to handle both of these workloads at scale on a single platform. In addition, Bigtable’s data lifecycle management features are specifically geared toward handling the time-oriented nature of vehicle telemetry data. The automated garbage collection policies use time and version as criteria for purging obsolete data effectively, enabling us to manage storage costs and reduce operational overhead. 

In the end, the choice was obvious, and we decided to use Bigtable as our central vehicle telemetry data repository.

Ford Pro Telematics and Bigtable

We receive vehicle telemetry data as a protocol buffer to a passthrough service hosted on Compute Engine. We then push that data to Pub/Sub for Google-scale processing by a streaming Dataflow job that writes to Bigtable. Ford Pro customers can access data through our dashboard or an API for both historical lookback for things like journey construction and real-time access to see fleet status, position, and activity.

With Bigtable helping to power Ford Pro Telematics, we have been able to provide a number of benefits for our customers, including:

• Enabling the API service to access telematics data 

• Improving data quality with Bigtable’s built-in time series data management features

• Reducing operational overhead with a fully managed service

• Delivering robust data regulation compliance tooling across regions 

The platform provides interactive dashboards that display relevant information, such as real-time vehicle locations, trip history, detailed trip information, live map tracking and EV charging status. Customers can also set up real-time notifications about vehicle health and other important events, such accidents, delays, or EV charging faults. For example, a fleet manager can use the dashboard to track the location of a vehicle and dispatch assistance if an accident occurs.

We leverage BigQuery alongside Bigtable to generate reports. BigQuery is used for long-running reports and analysis, while Bigtable is used for real-time reporting, and direct access to vehicle telemetry. Regular reports are available for fleet managers, including vehicle consumption, driver reimbursement reports, and monthly trip wrap ups. Our customers can also leverage and integrate this data into their own tooling using our APIs, which enable them to query vehicle status and access up to one year of historical data.

Looking to the future

The automotive industry is constantly evolving, and with the advent of connected vehicles, there are more opportunities than ever before to improve the Ford commercial customer experience. Adopting a fully managed service like Bigtable allows us to spend less time maintaining our own infrastructure and more time innovating and adding new features to our platform. Our company is excited to be at the forefront of this innovation, and we see many ways that we can use our platform to help our customers.

One of the most exciting possibilities is the use of machine learning to predict vehicle maintenance and create service schedules. By collecting data from vehicle diagnostics over time, we can feed this information into machine learning models that can identify patterns and trends. This will allow us to alert customers to potential problems before they even occur, and to schedule service appointments at the most convenient times.

Another area where we can help our customers is in improving efficiency. By providing insights about charging patterns, EV range, and fuel consumption, we can help fleet managers optimize their operations. For example, if a fleet manager knows that there are some shorter routes for their cars, they can let those cars hit the road without a full charge. This can save time and money, and it can also reduce emissions.

In addition to helping our customers save time and money, we are also committed to improving their safety and security. Our platform can provide alerts for warning lights, oil life, and model recalls. This information can help customers stay safe on the road, and it can also help them avoid costly repairs.

We are already getting great feedback from customers about our platform, and we are looking forward to further increasing their safety, security, and productivity. We believe that our platform has the potential to revolutionize the automotive industry, and we are excited to be a part of this journey.

Get started

• Learn more about Bigtable and why it is a great solution for automotive telemetry and time series data. 

• Read more on how others like Palo Alto Networks, Flipkart, and Globo are reducing cloud spend while improving service performance, scalability and reliability by moving to Bigtable.

• Ready to unlock the power of Bigtable? Start a free trial today!

Tis the season for learning new skills! Get ready for 12 Days of Learning, a festive digital advent calendar packed with courses, hands-on labs, videos, and community opportunities—all designed to boost your generative AI expertise. Discover a new learning resource on Google Cloud’s social channels every day for twelve days this December. 

Before you start: Get no-cost access to generative AI courses and labs

Join the Innovators community to activate 35 monthly learning credits in Google Cloud Skills Boost at no cost. Use these credits to access courses and labs throughout the month of December—and beyond! 

Ready to get started? Review all of the resources below.

Get festive with generative AI foundations

Learn how to use gen AI in your day-to-day work. These resources are designed for developers looking to gain foundational knowledge in gen AI.

1. A Developer’s Guide to LLMs: In this 10-minute video, explore the exciting world of large language models (LLMs). Discover different AI model options, analyze pricing structures, and delve into essential features.

2. Responsible AI: Fairness & Bias: This course introduces the concepts of responsible AI and shares practical methods to help you implement best practices using Google Cloud products and open source tools.

3. Gemini for end-to-end SDLC: This course explores how Google Cloud’s Gemini AI can assist in all stages of the software development lifecycle, from building and debugging web applications to testing and data querying. The course ends with a hands-on lab where you can build practical experience with Gemini.

4. Responsible AI for Developers: Interpretability & Transparency: This course introduces AI interpretability and transparency concepts. Learn how to train a classification model on image data and deploy it to Vertex AI to serve predictions with explanations.

5. Introduction to Security in the World of AI: This course equips security and data protection leaders with strategies to securely manage AI within their organizations. Bring these concepts to life with real-world scenarios from four different industries.

Cozy up with gen AI use cases

Launch these courses and labs to get more in-depth, hands-on experience with generative AI, from working with Gemini models to building agents and applications. 

1. Build Generative AI Agents with Vertex AI and Flutter: Learn how to develop an app using Flutter and then integrate the app with Gemini. Then use Vertex AI Agent Builder to build and manage AI agents and applications.

2. Machine Learning Ops (MLOps) for Generative AI: In this course, learn how to overcome MLOps challenges and use generative AI to streamline processes.

3. Boost productivity with Gemini in BigQuery: This course provides an overview of features to assist in the data-to-AI workflow, including data exploration and preparation, code generation, workflow discovery, and visualization. 

4. Build Generative AI Apps with Firebase Genkit: Learn how to integrate gen AI features into your applications using Firebase Genkit—from prototyping to production. 

5. Website Modernization with Generative AI on Google Cloud: Transform your website experiences with gen AI with this hands-on course. Learn how to build a generative search experience in an interactive lab.

6. Work with Gemini models in BigQuery: Through a practical use case involving customer relationship management, learn how to solve a real-world business problem with Gemini models. Plus, receive step-by-step guidance through coding solutions using SQL queries and Python notebooks.

Get a jump-start on your New Year’s resolutions with AI Skills Quest

Get an early start on your learning goals by signing up for AI Skills Quest, a monthly learning challenge that puts gen AI on your resume with verifiable skill badges. When you sign up, choose your path based on your level of knowledge:

• Beginner/Intermediate cohort: Learn fundamental AI concepts, prompt design, and Gemini app development techniques in Vertex AI, plus other Gemini integrations in various technical workflows.

• Advanced cohort: Already know the basics, but want to add breadth and depth to your AI skills? Sign up for the Advanced path to learn advanced AI concepts like RAG and MLOps.

Ready to ring in the new year with new skills? Find more generative AI learning content on Google Cloud Skills Boost.

2024 has been a landmark year for Google Earth Engine, marked by significant advancements in platform management, cloud integration, and core functionality. With increased interoperability between Google Cloud tools and services, and Earth Engine, we’ve unlocked powerful new workflows and use cases for our users.  Here’s a round up of this year’s top Earth Engine launches, many of which were highlighted in our Geo for Good 2024 summit.

Management: Streamlining Workflows

Earlier this year, we launched the new Earth Engine Overview page in the Cloud Console, serving as a centralized hub for Earth Engine resources, allowing you to manage Earth Engine from the same console used to manage and monitor other Cloud services. 

In this console, we also introduced a new Tasks page, allowing you to view and monitor Earth Engine export and import tasks alongside usage management and billing. The Tasks page provides a useful set of fields for each task, including state, runtime, and priority. Task cancellation is also easier than ever with single or bulk task cancellation in this new interface. 

As we deepen Earth Engine’s interoperability across Google Cloud, we’ll be adding more information and controls to the Cloud Console so that you can further centralize the management of Earth Engine alongside other services.

Integrations: deepening cloud interoperability

Earth Engine users can integrate with a number of cloud services and tools to enable advanced solutions requiring custom machine learning and robust data analytics. This year, we launched a set of features that improved existing interoperability, making it easier to both enable and deploy these solutions.  

Vertex AI integration
Using Earth Engine with Vertex AI enables use cases that require deep learning, such as crop classification. You can host a model in Vertex AI and get predictions from within the Earth Engine Code Editor. This year, we announced a major performance improvement to our Vertex Preview connector, which will give you more reliability and more throughput than the current Vertex connector.

Earth Engine access
To ensure all Earth Engine users can take advantage of these new integration improvements and management features, we’ve also transitioned all Earth Engine users to Cloud projects. With this change, all Earth Engine users can now leverage the power and flexibility of Google Cloud’s infrastructure, security, and growing ecosystem of tools to drive forward the science, research, and operational decision making required to make the world a better place.

Security: enhancing control

This year we launched Earth Engine support for VPC Service Controls – a key security feature that allows organizations to define a security perimeter around their Google Cloud resources. This new integration, available to customers with professional and premium plans, provides enhanced control over data, and helps prevent unauthorized access and data exfiltration. With VPC-SC, customers can now set granular access policies, restrict data access to authorized networks and users, and monitor and audit data flows, ensuring compliance with internal security policies and external regulations.

Platform: improving performance

Zonal Statistics
Computing statistics about regions of an image is a core Earth Engine capability. We recently launched a significant performance improvement to batch zonal statistics exports in Earth Engine. We’ve optimized the way we parallelize zonal statistics exports, such as exports that generate statistics for all regions in a large collection. This means that you will get substantially more concurrent compute power per batch task when you use ReduceRegions().

With this launch, large-scale zonal statistics exports are running several times faster than this time last year, meaning you get your results faster, and that Earth Engine can complete even larger analyses than ever. For example, you can now calculate the average tree canopy coverage of every census tract in the continental United States at 1 meter scale in 7 hours. Learn more about how we sped up large-scale zonal statistics computations in our technical blog post.

Python v1
Over the last year, we’ve focused on ease-of-use, reliability, and transparency for Earth Engine Python. The client library has moved into an open-source repository at Google which means we can sync changes to GitHub immediately, keeping you up-to-date on changes between releases. We are also sharing  pre-releases, so you can see and work with Python library candidate releases before they come out. We have a static loaded client library, which makes it easier to build on our Python library and better testing and error messaging. We’ve also continued making progress on improving geemap and integrations like xee.

With all of these changes, we’re excited to announce that the Python Client library is now ‘v1’, representing the maturity of Earth Engine Python. Check out this blog post to read more about these improvements and see how you can take full advantage of Python and integrate it into Google’s Cloud tooling.

COG-backed asset improvements
If you have data stored in Google Cloud Storage (GCS), in Cloud-Optimized GeoTIFF (COG) format, you can easily use it in Earth Engine via Cloud Geotiff Backed Earth Engine Assets, improving the previous experience requiring a single file GeoTIFF, where all bands have the same projection and type.

Now you can create an Earth Engine asset backed by multiple GeoTiffs, which may have different projections, different resolutions, and different band types–and Earth Engine will take care of these complexities for you. There are also major performance improvements to the previous feature: Cloud GeoTiff backed assets now have similar performance to native Earth Engine assets. In addition, If you want to use your GCS COGs elsewhere, like open source pipelines or other tools, the data is stored once and you can use it seamlessly across products.

Looking forward to 2025

We’re excited to see Earth Engine users leverage more advanced tools, stronger security, and seamless integrations to improve sustainability and climate resilience. In the coming year, we’re looking forward to further deepening cloud interoperability, making it easier to develop actionable insights and inform sustainability decision-making through geospatial data.

Conventional fraud detection methods have a hard time keeping up with increasingly sophisticated criminal tactics. Existing systems often rely on the limited data of individual institutions, and this hinders the detection of intricate schemes that span multiple banks and jurisdictions. 

To better combat fraud in cross-border payments, Swift, the global provider of secure financial messaging services, is working with Google Cloud to develop anti-fraud technologies that use advanced AI and federated learning.  

In the first half of 2025, Swift plans to roll out a sandbox with synthetic data to prototype learning from historic fraud, working with 12 global financial institutions, with Google Cloud as a strategic partner. This initiative builds on Swift’s existing Payment Controls Service (PCS), and follows a successful pilot with financial institutions across Europe, North America, Asia and the Middle East.

The partnership: Google Cloud and Swift 

Google Cloud is collaborating with Swift — along with technology partners including Rhino Health and Capgemini — to develop a secure, privacy-preserving solution for financial institutions to combat fraud. This innovative approach uses federated learning techniques, combined with privacy-enhancing technologies (PETs), to enable collaborative intelligence without compromising proprietary data.

Rhino Health will develop and deliver the core federated learning platform, and Capgemini will manage the implementation and integration of the solution.

“Swift is in a unique position in the financial industry – a trusted and cooperative network that is integral to the functioning of the global economy. As such, we are ideally placed to lead collaborative, industry-wide efforts to fight fraud. This exploration will help the community validate whether federated learning technology can help financial institutions stay one step ahead of bad actors through sharing of fraud labels, and in turn enabling them to provide an enhanced cross-border payments experience to their customers,” said Rachel Levi, head of artificial intelligence, Swift. 

“At Google Cloud, we are committed to empowering financial institutions with cutting-edge technology to combat the evolving threat of fraud. Our collaboration with Swift exemplifies the transformative potential of federated learning and confidential computing. By enabling secure collaboration and knowledge sharing without compromising data privacy, we are fostering a safer and more resilient financial ecosystem for everyone,” said Andrea Gallego, Managing Director, global GTM incubation, Google Cloud,  

The challenge: Traditional fraud detection is falling behind

The lack of visibility across the payment lifecycle creates vulnerabilities that can be exploited by criminals. A collaborative approach to fraud modeling offers significant advantages over traditional methods in combating financial crimes. To be effective, this approach requires data sharing across institutions, which is often restricted because of privacy concerns, regulatory requirements, and intellectual property considerations.

The solution: Federated learning

Federated learning offers a powerful solution for collaborative AI model training without compromising privacy and confidentiality. Instead of requiring financial institutions to pool their sensitive data, the model training occurs within financial institutions on decentralized data.

Here’s how it works for Swift: 

1. A copy of Swift’s anomaly detection model is sent to each participating bank. 

2. Each financial institution trains this model locally on their own data.

3. Only the learnings from this training — not the data itself — are transmitted back to a central server for aggregation, managed by Swift. 

4. The central server aggregates these learnings to enhance Swift’s global model. 

This approach significantly minimizes data movement and ensures that sensitive information remains within each financial institution’s secure environment. 

Core benefits of the federated learning solution

By using federated learning solutions, financial institutions can achieve substantial benefits, including:  

• Shared intelligence: Financial institutions work together by sharing information on fraudulent activities, patterns, and trends, which creates a much larger and richer decentralized data pool than any single institution could gather alone.

• Enhanced detection: The collaborative global model can identify complex fraud schemes that might go unnoticed by individual institutions, leading to improved detection and prevention.

• Reduced false positives: Sharing information helps refine fraud models, leading to more accurate identification of genuine threats and fewer false alarms that disrupt both legitimate activity and the customer experience.

• Faster adaptation: The collaborative approach allows for faster adaptation to new fraud trends and criminal tactics. As new threats emerge, the shared knowledge pool helps all participants quickly adjust their models and their fraud prevention tools.

• Network effects: The more institutions participate, the more comprehensive the data pool becomes, creating a powerful network effect that strengthens fraud prevention for everyone involved.

For widespread adoption, federated learning must seamlessly integrate with existing financial systems and infrastructure. This allows financial institutions to easily participate and benefit from the collective intelligence without disrupting their operations.

Architecting the global fraud AI solution

The initial scope remains a synthetic data sandbox centered on prototyping learning from historic payments fraud. The platform allows multiple financial institutions to train a robust fraud detection model while preserving the confidentiality of their sensitive transaction data. It uses federated learning and confidential computing techniques, such as Trusted Execution Environments (TEEs), to enable secure, multi-party machine learning without training data movement.

There are several key components to this solution:

• Federated server in TEE execution environment: A secure, isolated environment where a federated learning (FL) server orchestrates the collaboration of multiple clients by first sending an initial model to the FL clients. The clients perform training on their local datasets, then send the model updates back to the FL server for aggregation to form a global model.

• Federated client: Executes tasks, performs local computation and learning with local dataset (such as  data from an individual financial institution), then submits results back to FL server for secure aggregation.

• Bank-specific encrypted data: Each bank holds its own private, encrypted transaction data that includes historic fraud labels. This data remains encrypted throughout the entire process, including computation, ensuring end-to-end data privacy.

• Global fraud-based model: A pre-trained anomaly detection model from Swift that serves as the starting point for federated learning.

• Secure aggregation: Using a Secure Aggregation protocol to compute these weighted averages would ensure that the server learns only the historic fraud labels from participating financial institutions, but not exactly which  financial institution, thereby preserving the privacy of each participant in the federated learning process.

• Global anomaly detection trained model and aggregated weights: The improved anomaly detection model, along with its learned weights, is securely exchanged back to the participating financial institutions. They can then deploy this enhanced model locally for fraud detection monitoring on their own transactions.

We’re seeing more enterprises adopt federated learning to combat global fraud, including global consulting firm Capgemini. 

“Payment fraud stands as one of the greatest threats that undermines the integrity and stability of the financial ecosystem, with its impact acutely felt upon some of the most vulnerable segments of our society,” said Sudhir Pai, chief technology and innovation officer, Financial Services, Capgemini. 

“This is a global epidemic that demands a collaborative effort to achieve meaningful change. Our application of federated learning is grounded with privacy-by-design principles, leveraging AI to pioneer secure aggregation and anonymization of data which is of primary concern to large financial institutions. The potential to apply our learnings within a singular global trained model across other industries will ensure we break down any siloes and combat fraud at scale,” he said. 

“We are proud to support Swift’s program in partnership with Google Cloud and Capgemini,” said Chris Laws, chief operating officer, Rhino. “Fighting financial crime is an excellent example of the value created from the complex multi-party data collaborations enabled by federated computing, as all parties can have confidence in the security and confidentiality of their data.”

Building a safer financial ecosystem, together

This effort to fight fraud collaboratively will help build a safer and more secure financial ecosystem. By harnessing the power of federated learning and adhering to strong principles of data privacy, security, platform interoperability, confidentiality, and scalability, this solution has the potential to redefine how we combat fraud in the age of fragmented globalized finance and demonstrates a commitment to building a more resilient and trustworthy financial world.