Cloud

AI is rapidly transforming industries and redefining the future of work. However, many organizations face a significant hurdle: bridging the knowledge gap and acquiring the necessary skills to effectively harness the power of AI. 

Recognizing this challenge, Google Cloud is set to launch the AI Playground in Shoreditch, Central London, in the first quarter of 2025. This innovative space will serve as a dynamic hub for businesses and individuals to demystify AI, explore its potential, and develop practical expertise.

More than a showcase — dive deep into AI

Google’s powerful Gemini model family — now in its supercharged second generation — takes center stage at the AI Playground, featuring several interactive demos that put its multimodal and agentic capabilities on display. Among the capabilities guests can experience and experiment with first hand are Gemini’s ability to analyze complex data, generate creative formats, and power innovative solutions.

The AI Playground is much more than a technology showcase. 

We’ve built the space to serve as an immersive learning environment where visitors can actively engage with AI, participate in hands-on workshops and hackathons, and connect with Google Cloud AI experts. This unique approach fosters a deeper understanding of AI concepts and encourages experimentation with cutting-edge tools and techniques.

Addressing the AI skills gap

The AI Playground directly addresses the growing need for AI skills development in today’s rapidly evolving technological landscape, providing a dedicated space for:

• Hands-on experimentation: Gain practical experience with cutting-edge AI tools and techniques, moving beyond theoretical knowledge to real-world application.

• Skills development: Build on-demand AI skills through interactive workshops and hackathons led by Google Cloud experts, equipping individuals and teams with the expertise needed to thrive in the AI era.

• Community building: Connect with fellow AI enthusiasts, share knowledge, and facilitate collaboration, creating a vibrant ecosystem for learning and innovation.

• Real-world inspiration: Explore AI applications across diverse industries and discover new possibilities for your organization, sparking creativity and driving the development of novel AI solutions.

Empowering the future of AI

By providing accessible learning opportunities and fostering a thriving AI community, the AI Playground empowers individuals and businesses to embrace the transformative power of AI and contribute to shaping a better future. It’s a place where curiosity meets innovation, learning is hands-on, and  the potential of AI is unlocked. Mark your calendars for Q1 2025 and prepare to embark on your AI journey at Google Cloud’s AI Playground.

Enterprise workloads like SAP S/4HANA present unique challenges when migrating to a public cloud, making the choice of a cloud provider critically important. As an in-memory database for large SAP deployments, SAP HANA can have massive memory and CPU processing requirements, and that can just be the beginning of the challenges. Businesses are generating, processing, and making business decisions on more data than ever before, and they require increasingly complex data analysis, algorithms, and reporting. SAP provides the real-time operational and historical analytics and insights businesses need to make informed decisions quickly. However, this requires machines that can process and analyze large volumes of data in real time, demanding high levels of memory and compute power. 

To meet these demands, we recently introduced X4 machine types. The Compute Engine X4 machine family is purpose-built to handle the demanding requirements of SAP HANA Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) workloads. These machines deliver strong performance, scalability, and reliability, empowering businesses to unlock the full potential of their SAP S/4HANA, SAP Business Suite (ECC) on SAP HANA deployments, and SAP’s Industry Solutions. X4 is also built to support OLAP workloads such as SAP BW/4HANA and SAP BW on HANA.

Uncompromising performance in a scalable solution

As businesses grow, whether organically or via acquisitions, they often risk hitting CPU and memory limitations of their SAP HANA system. This forces them to upgrade to larger scale-up machines earlier or using multiple machines in a scale-out configuration. Scaling out is a complex custom process that requires application data redesign and approvals from SAP, as well as being costly from an operational and resource perspective. In addition, the planned downtime required for these migration operations can be unacceptable for organizations that rely on SAP for mission-critical operations.

X4 solves these challenges by allowing businesses to grow further in scale-up mode before hitting performance bottlenecks. X4 is available in 16TB, 24TB, and 32TB memory configurations and 960, 1440, 1920 vCPU cores respectively with “standard sizing” SAP certification for both SAP HANA OLTP and OLAP capable of running the most demanding enterprise SAP HANA workloads. X4 machines also boast scale-out OLTP certifications for up to four nodes the size of the servers, for a total of 128 terabytes for S/4HANA use cases — larger than any other cloud provider’s. The X4 16TB machines achieved an SAP Benchmark IaaS SAPS result that is over 8% higher than the closest IaaS cloud’s. (Source: SAP note 2456432 – Logon required). 

As the only cloud IaaS provider offering a 32TB SAP-certified machine, Google Cloud helps ensure that your SAP HANA environment grows seamlessly with your business needs while providing a cost-effective and simple cloud-native experience. Here’s how:

• Industry-leading compute and storage: The Google Cloud X4 machine family gains its power and scale through a combination of powerful hardware and software optimizations. Integration with Google Cloud Hyperdisk provides high performance, scalability, and resilient block storage. Hyperdisk provides dedicated, dynamically tunable read/write operations per second (IOPS) and throughput with each volume up to 5GB per second, delivering substantially higher performance compared to the previous generation of Persistent Disk. This level of performance is crucial for applications that require rapid data access, like databases, real-time analytics, and reduces SAP HANA rehydration times. Additionally, X4 machines run on one of the world’s largest privately managed networks, offering low-latency performance. 

• High availability and reliability: Minimizing downtime for mission-critical SAP applications is a top priority for customers. X4 machines have an industry-leading 99.95% Compute Engine Memory-Optimized Single Instance SLA. Paired with Hyperdisk, X4 enables rapid SAP HANA rehydration capabilities (<45 min full rehydration for 32TB) that minimize application business process downtime, so your mission-critical SAP applications stay up and running. 

• Performance-boosting innovation: Titanium is a hardware- and software-based technology designed to significantly enhance workload performance, efficiency, and security while also simplifying management by automating many infrastructure-level tasks. Titanium shifts networking and storage functions away from the machine’s CPU to specialized hardware, allowing the machine’s CPU to focus on running SAP HANA applications. Titanium’s tight integration with Google Cloud’s Hyperdisk block storage service allows for best-in-class storage I/O performance (up to 400k IOPS per instance), surpassing what’s typically possible with traditional architectures. 

• Enhanced security and compliance: X4 machines inherit Google Cloud’s robust security features, including encryption of data in transit and at rest as well as compliance with industry standards and regulations with support for Google Cloud’s Assured Workloads. Titanium enhances security feature performance by offloading security-sensitive tasks like encryption and key management to dedicated hardware with enhanced protections. This helps improve the security posture of workloads, reduces the risk of vulnerabilities, and enables customers to meet their compliance and regulatory requirements.

• Simplified cloud-native experience: X4 machines offer the benefits of a cloud-native architecture for SAP S/4 HANA, integrating seamlessly with Google Cloud’s ecosystem and providing access to a wide range of cloud services. Compute Engine also offers a seamless migration experience for X4. You can easily migrate your SAP instance from M2 servers (up to 12 terabytes) to X4. The migration process from other servers or on-prem environments to X4 is the same as the standard process for migrating to any other Google Cloud server. 

• A choice of service model: When migrating to Google Cloud, SAP customers have two choices. You can choose a direct (BYOL) deployment model, or you can take advantage of RISE with SAP, which brings solutions and expertise from SAP and technology ecosystem partners into one single-contract subscription offering.

  “In the past few years, our SAP HANA systems have seen significant data growth with an increasing need for higher performance. With the 24TB X4 machines and Hyperdisk storage, we have been able to raise the ceiling for our future data growth and are also looking to see improvements in our performance. Added to this, Google’s X4 machines are cloud native, giving us opportunities to automate system management and operations.” – Shawn Lund, US Chief Technology Officer, Deloitte

Google Cloud’s X4 machines represent a significant advancement in SAP HANA infrastructure, offering high performance, scalability, and cost-efficiency. By integrating with Google Cloud’s powerful ecosystem and providing a cloud-native experience, X4 machines empower businesses to unlock the full potential of their SAP S/4HANA deployments and drive digital transformation. 

Learn more about X4 machines and RISE with SAP on Google Cloud. View SAP Datasphere for Google BigQuery on Google Cloud Marketplace.

As cybersecurity threats have grown more sophisticated and prevalent, we’ve seen organizations develop robust cyber threat intelligence (CTI) programs to help bolster defenses. However, creating and maturing a CTI program remains a challenge because it requires people, processes, technologies, and metrics to validate success.

To help organizations better operationalize threat intelligence, we’ve published the CTI Program Design Playbook. This training track is the inaugural release from Mandiant Academy’s newest on-demand training, developed for professionals who actively defend networks. It’s designed to provide you with the essential knowledge and skills to design, build, operate, and optimize a CTI program from the ground up.

What you’ll learn: An overview

This track includes three two-hour courses addressing different aspects of CTI program development.

In the first course, Designing and Building a CTI Program, you will learn how to establish the foundations of a CTI program, including a CTI program charter and roadmap.

In the second course, Operating an Effective CTI Program, you will learn how to effectively identify and maintain a CTI program’s stakeholders. This course also describes the resources necessary to collect, analyze, and disseminate threat intelligence while accommodating stakeholders’ unique requirements.

Students who are part of an active CTI program but would like to mature their team’s capabilities may benefit from the third course, Optimizing an Efficient CTI Program. This course teaches you helpful ways to accurately measure a CTI program’s effectiveness and, where necessary, make improvements.

Who should take these courses?

This training track is the inaugural release from Mandiant Academy’s newest approach to on-demand training, which breaks the core tenets of effective cybersecurity strategies into discrete two-hour courses. This new style aims to address the needs of busy practitioners who already possess the knowledge and skills needed to defend networks but have limited time to enhance their capabilities. 

The CTI Program Design Playbook was developed for practitioners:

• Intelligence analysts supporting network defenders at the tactical, operational, or strategic levels

• Managers of active intelligence programs who would like to improve their team’s capabilities

• Cybersecurity practitioners who desire an intelligence capability

• Practitioners tasked with founding their organization’s first intelligence capability

Start learning today

To learn more about the CTI Program Design Playbook and to sign up for the course, please visit our website. You can access a wealth of knowledge through Mandiant Academy’s on-demand, instructor-led, and experiential training options. We hope this course proves helpful in your efforts to defend your organization against cyber threats.

As a Kubernetes platform engineer, you’ve probably followed the buzz around eBPF and its revolutionary impact on Kubernetes networking. Perhaps you’ve explored Cilium, a popular solution leveraging eBPF, and wondered how Google Kubernetes Engine (GKE) harnesses this power. That curiosity may have led you to GKE Dataplane V2. And you’ve undoubtedly noticed that where eBPF goes, enhanced observability follows.

In this article, we’ll shine a light on a tool that you can leverage with GKE Dataplane V2: Hubble. This powerful observability tool provides deep visibility with added Kubernetes context into your network packet flow.

GKE Dataplane V2 observability provides a Managed Hubble solution. This includes the following components:

• Hubble Relay: a service that collects network telemetry data about your pods and nodes

• Hubble CLI: a command-line interface tool providing live traffic information within the cluster

Enabling Dataplane V2 observability in GKE, either through the Google Cloud console or the gcloud command, triggers the deployment of a dedicated hubble-relay deployment with a single pod. This pod houses both the Hubble Relay and CLI components as dedicated containers, providing immediate access to network telemetry data and command-line tools for analysis.

While the Hubble UI is not included in the initial deployment, you can easily enhance your observability setup by adding it. This web-based interface offers a visual and interactive way to explore and analyze the data collected by Hubble Relay, making it easier to identify trends, spot anomalies, and troubleshoot issues.

With Hubble CLI and UI at our disposal, let’s explore how these tools can be used to effectively troubleshoot your Kubernetes environment.

Before you start

• Make sure you’re comfortable using the Hubble CLI (link to documentation).

• Know how to deploy the Hubble UI binary distribution (link to documentation).

Setting the stage

For this example, we’ll use two pods, each deployed in a separate namespace and exposed as a service.

Scenario 1: Tracing a simple request

In this scenario, we observe a basic interaction between a pod and a service.

1. Initiate a request: Execute a curl request from the store-frontend pod to backend-svc service:

$ kubectl exec store-frontend -n fe -it -- curl “backend-svc.be.svc.cluster.local.”

2. Observe with Hubble: Use the Hubble CLI to examine the traffic:

$ hubble observe --pod fe/store-frontend --follow

Hint: For a full list of Hubble filters and flags, run hubble observe –help

Hubble captures the entire flow: the store-frontend pod contacting kube-dns for service name resolution, followed by the TCP connection to the store-backend pod. See how Hubble adds the namespace and pod names to the packets? This Kubernetes-aware insight makes it much easier to understand and analyze your network traffic.

Scenario 2: When DNS goes wrong

Let’s simulate a scenario where our DNS server is overwhelmed. We deploy an application that floods the cluster with DNS requests, putting significant strain on the kube-dns pods. Then, we replicate the request from Scenario 1.

By examining the timestamps of the DNS request and response in the Hubble CLI output, we can clearly observe a delay of approximately 5 seconds for kube-dns to respond. This highlights the impact of the overloaded DNS server and explains the latency observed in our system.

Let’s intensify the load on our DNS server even further, pushing it to the point where DNS requests fail outright due to an inability to resolve service names.

$ kubectl exec -it store-frontend -n fe -- curl "backend-svc.be.svc.cluster.local."
curl: (6) Could not resolve host: backend-svc.be.svc.cluster.local.

In this extreme scenario, the Hubble drop reveals that our DNS queries are simply not being answered, indicating a complete breakdown in name resolution.

We finally will make the kube-dns pods unavailable by taking them down.

Hubble’s flow logs provide a clear picture of what’s happening. You can see that the store-frontend pod attempts to reach the kube-dns service, but the request fails because there are no healthy kube-dns pods to handle it.

Hubble CLI isn’t just a troubleshooting tool; it’s also powerful for optimization. For example, while running the first scenario, you may have noticed a surprisingly high number of DNS requests generated from a single curl command:

A quick investigation reveals the root cause: a missing trailing dot in the service name, and the pod’s resolv.conf being configured with ndots:5. This combination meant the query wasn’t treated as absolute and was instead expanded with all five search domains listed in resolv.conf, resulting in six DNS requests for every single curl call.

Scenario 3: Network policy mishap!

It appears there was a slight oversight with a new Ingress NetworkPolicy. Let’s just say Monday mornings and network policies don’t always mix…! The front-end application was accidentally left out of the allowlist.

Communications between the front-end and the back-end applications hang and timeout. The front-end engineer rushes to Hubble CLI. Hubble shows that the front-end application traffic is being denied and dropped by a network policy.

Thankfully, the Hubble CLI allows you to filter events by type using the –type flag. For instance, --type policy-verdict shows only events related to network policies. You can further refine this by using --verdict DROPPED to see only events where traffic was dropped due to a policy.

Hubble UI

While the Hubble CLI is undeniably powerful, the Hubble UI offers a complementary and equally valuable perspective. Beyond replicating the CLI’s functionality, the UI provides a visual map of your cluster’s network traffic. This allows you to easily grasp how pods and Services connect, both internally and to the outside world. With filtering options like namespaces and labels, you can quickly isolate specific traffic flows and troubleshoot potential problems.

Explore the depths of GKE Dataplane V2 with Hubble

This exploration of Hubble, while introductory, provides a solid foundation for understanding its capabilities. Remember that far more intricate queries and filters can be crafted to gain deeper insights into specific network issues. Simply knowing about Hubble and how to access it is a significant first step towards effectively operating and troubleshooting Cilium-based Dataplane V2 clusters. As you dive deeper into Hubble’s capabilities, you’ll discover its full potential for observability and optimization within your GKE environment.

When it comes to AI, large language models (LLMs) and machine learning (ML) are taking entire industries to the next level. But with larger models and datasets, developers need distributed environments that span multiple AI accelerators (e.g. GPUs and TPUs) across multiple compute hosts to train their models efficiently. This can lead to orchestration, resource management, and scalability challenges.

We’re here to help. At Google Cloud, we provide a robust suite of GPU and TPU resources alongside advanced orchestration tools as part of AI Hypercomputer architecture to simplify distributed, large-scale training. In this blog, we’ll guide you through the orchestration tools available for GPU accelerators on Google Cloud that can help you streamline and scale your machine learning workflows.

Choose the right accelerator family

A key element of distributed training lies in selecting the right GPU. Google Cloud’s specialized machine families offer tailored solutions for varying needs of performance and cost efficiency. The A3 machine series, featuring NVIDIA H100 and NVIDIA H200 (upcoming) GPUs, delivers strong GPU-to-GPU bandwidth that’s a great fit for large-scale training workloads. In contrast, the A2 machine series with NVIDIA A100 GPUs is designed for scenarios that require minimal inter-node communication such as streamlined, single-node training. Additionally, the versatile G2 machine family, equipped with NVIDIA L4 GPUs, provides the flexibility necessary for inference and testing workloads.

We also offer multiple GPU consumption models to meet the needs of large-scale training:

• Committed Use Discounts (CUDs) provide significant cost savings and guaranteed capacity in return for a long-term commitment.

• Dynamic Workload Scheduler (DWS) comes in two modes, which are designed for various workloads that need assurance or can be flexible about start time; the capacity is available for a defined duration and offered at a lower list price.

• On-demand consumption is the most flexible, with no upfront commitments, although the capacity availability is not guaranteed.

• Spot VMs provide drastically lower costs but are preemptible, requiring resilient and disruption-tolerant job designs.

To further accelerate your distributed training, we’ll explore three powerful orchestration strategies on Google Cloud: Google Kubernetes Engine (GKE), Cluster Toolkit, and Vertex AI custom training pipeline. Each approach brings its unique strengths, enabling you to leverage Google Cloud’s powerful infrastructure to drive your machine learning projects forward quickly and scalably.

Let’s walk through each of the options to better understand how Google Cloud’s advanced orchestration tools can help you optimize resources, reduce complexity, and achieve strong performance in your ML initiatives.

Option 1: GKE for unified workload management

Enterprises with robust platform teams often want a unified environment on which to run all their workloads, including custom training, for simpler management. GKE is a good choice in this context, providing the flexibility and scalability to handle diverse workloads on a single platform. With GKE, platform teams gain centralized control and visibility, while optimizing resource utilization and streamlining management.

Here’s how to orchestrate ML workloads running on GKE:

1. GKE cluster and nodepool provisioning
If you have reservation (CUD or DWS calendar) and prefer to use Terraform, follow the instructions from cluster provisioning templates, and specify the parameter file (terraform.tfvars):

Then execute the following command to provision the GKE cluster and nodepool:

In addition, Cluster Toolkit includes terraform based example blueprints to provision A3 or A3 Mega GKE clusters and nodepool. 

If you prefer to use the gcloud command, follow the step-by-step instructions from this tutorial to create a GKE cluster and nodepool with A3/A3 Mega VMs. 

For DWS Flex, you can create a DWS enabled node-pool with these gcloud commands:

2. Enable GPU direct communication with A3 TCPX/A3 Mega TCPXO and perform an initial benchmark test
Follow these steps to install GPUDirect for TCPX/TCPXO libraries, configure NCCL, and deploy a test workload to perform your initial benchmark tests.

Validate the output of allgather benchmark tests for two A3 Mega nodes:

In the above benchmark output table, the first column is message size, while the algbw and busbw columns on the right indicate per GPU bandwidth. Usually, we use the in/out-of-place busbw column with the biggest message size (highlighted row) to determine cross-node bandwidth. For A3 Mega nodes, we expect a range of 185-190GB/s per GPU; this may indicate near cross-node 1600gbps network bandwidth for A3 Mega nodes with 8 NVIDIA H100 GPUs and 8 NICs.

You may expand the NCCL tests from two nodes to 8, 16, 32, etc. to ensure your cross-node network performance is within a decent range and that all the nodes are healthy.

3. Configure distributed training batch workload
You can use JobSet, a Kubernetes-native API for managing a group of k8s Jobs as a unit using a unified API, to deploy distributed HPC (e.g., MPI) and AI/ML training workloads (PyTorch, Jax, Tensorflow etc.) on Kubernetes.

The following example illustrates a JobSet yaml manifest for A3 with GPUDirect-TCPX, which includes:

1. Key JobSet configuration elements

b. Training job settings, including pytorch main container
c. gcsfuse, tcpx (A3 high), tcpxo (A3 Mega) RxDM container
d. NCCL environment variables

For DWS batch workloads, please refer to the following A3 Mega-based example, integrating Kueue and JobSet settings.

Lastly, refer to this Helmchart example to see how to perform Megatron LM (Llama2) training on A3 Mega.

Option 2: Slurm via Cluster Toolkit

Slurm is one of the most popular high-performance computing (HPC) job schedulers. Used by researchers in both academia and industry, it offers a robust solution for LLM training orchestration with familiar semantics. Support for Slurm on Google Cloud is provided by Cluster Toolkit, formerly known as Cloud HPC Toolkit, open-source software that simplifies the process of deploying HPC, AI, and ML workloads on Google Cloud. It is designed to be highly customizable and extensible, and to address the deployment needs of a broad range of use cases, including deploying infrastructure for large-scale LLM training.

1. Provisioning A3-high and A3-mega clusters
Install Cluster Toolkit using the configuration instructions in the public documentation. Be sure to note some of the prerequisites including supported versions of Go, Terraform, and Packer.

Once you have a working Cluster Toolkit installation including the downloaded github repository, navigate to the examples/machine-learning blueprints directory. Here, you will have two folders for deploying H100 clusters based on the A3-series machine shapes, a3-highgpu-8g and a3-megagpu-8g. In this example, we’ll explore the blueprint in the a3-megagpu-8g folder.

Google Cloud Cluster Toolkit blueprints are Infrastructure as Code (IaC) documents that describe the infrastructure you would like to deploy, and are conceptually similar to Terraform or other IaC tooling. For the a3-megagpu-8g blueprint, there are three main files that control the deployment: 

1. slurm-a3mega-base.yaml – includes creating the necessary VPC networks along with the filestore instance used for a common home filesystem on the cluster nodes.
2. slurm-a3mega-image.yaml – creates the Compute Engine image instance that is used by Slurm to provision nodes based on the cluster’s definitio
3. slurm-a3mega-cluster.yaml – sets up the main cluster components, including the Slurm controller (the main orchestrator for Slurm jobs), the Slurm login node (a host used for job submission) and the a3mega partition (the working nodes in the cluster)

While you can customize each of the blueprint components if needed, you can easily get started by simply specifying the details for your working environment in the deployment-base.yaml and the deployment-image-cluster.yaml.

2. Enable GPUDirect-TCPXO optimized NCCL communication
Once the Slurm cluster is created, follow this tutorial to enable GPUDirect-TCPXO for optimized NCCL communication on the GPU networks. To validate the environment and ensure the TCPXO plugin is being properly loaded, build and compile the NCCL tests. Then, run sbatch run-nccl-tests.sh from the login node, being sure to change the number of nodes in the script to match those in your cluster. This runs a distributed all_gather test across the GPUs and nodes indicated in the script.

When working as intended, the NCCL tests should show output results indicating high-speed bandwidth throughput at various message sizes. A common measure of performance is to use the busbw value in GB/s from the second or last row of the output table, which shows the 4Gb and 8Gb message size values. A cluster with TCPXO active should report around 190 GB/s busbw throughput. See the performance page in the NVIDIA NCCL-tests repository for more details around these metrics.

3. Run an NeMo training workload
Follow this NeMo training tutorial to run an example NeMo Framework Pre-Training job using the following steps:

Step 1:

• Creates a NeMo Framework-derived container with the necessary TCPXO environment variables

• Submits a Slurm job to copy the framework launcher scripts and a few other auxiliary files into your working directory

Step 2:

• Establishes a Python virtual environment and installs NeMo Framework python package dependencies

Step 3:

• This command runs distributed training of a 5B parameter GPT3 model across eight nodes for 10 steps using mock data as the input.

Option 3: Vertex AI

For teams seeking managed infrastructure experience as well as access to leading open models such as Llama 3.1, Mistral, etc., Vertex AI Model Garden and Custom Training Job service presents an attractive option. This fully managed solution removes most of the orchestration burden and provides end-to-end ML platform operations, allowing you to focus on model development and experimentation. Vertex AI’s end-to-end training support further simplifies the process, offering an integrated workflow from data preparation to deployment.

Let’s look at how to perform  single-node or multi-node fine-tuning/training workload on Vertex.   

Single-node multi-GPU fine-tuning/training on Vertex
This notebook demonstrates fine-tuning and deploying Llama 3.1 models with the Vertex AI SDK. All of the examples in this notebook use parameter-efficient finetuning (PEFT) methods with Low-Rank Adaptation (LoRA) to reduce training and storage costs. LoRA is one approach of PEFT, where pretrained model weights are frozen and rank decomposition matrices representing the change in model weights are trained during fine-tuning.

Multi-node distributed fine-tuning/training on Vertex AI

This Vertex sample training repo provides examples on how to launch multi-node distributed training on A3 Mega (8 x NVIDIA H100) on Vertex. 

The NeMo example illustrates how to perform pre-training, continued pre-training and supervised fine-tuning (SFT). In addition, NeMo allows optimized training as a popular approach to evaluate the AI accelerator (A3 Mega in this case). To benchmark, you can rely on the reported metrics such as epoch time, step-time, etc. Since NeMo runs on most NVIDIA GPU types, it can be helpful for comparing different AI chips for a given task. Read on to learn how to run the example on Vertex with A3 Mega node types.

launch.sh is the main entry point to launch NeMo distributed training with command parameters:

Example:

At the end of launch.sh script, we use curl command to call Vertex customJobs API to launch NeMo training job in Vertex:

Job configurations in vertex-payload.json are part of curl command to launch Nemo training, it includes job specifications on resource requirements as showed:

The job configuration arguments “${TRANSFER_MODEL_CMD} ${LAUNCH_CMD}” in turn embed full content from the job training script, which also includes all the NCCL environments required by A3 Mega, while other pytorch launch commands are executed by Vertex CustomJob.

Optionally, build your own custom job container image as an “imageUri” parameter in vertex-payload.json, using this Dockerfile as your reference.

DIY enthusiasts: Building custom training environments

Lastly, we recognize many organizations prefer a more hands-on approach and have specific orchestration tools or frameworks that they wish to use. If that describes you, Google Compute Engine provides the foundation to build your own tailored training environments, letting you create and configure virtual machines (VMs) with your desired specifications, including the type and number of GPUs, CPU, memory, and storage. This granular control lets you optimize your infrastructure for your specific training workloads and integrate your preferred orchestration tools.

To facilitate this process, we provide example code snippets demonstrating how to use the gcloud compute instance create and gcloud compute instance bulk create API calls to create and manage your vanilla A3 Mega instances. Whether you need to create a single VM or provision a large-scale cluster, these resources can help streamline your infrastructure setup.

Conclusion

With the right orchestration strategy and Google Cloud’s robust and leading AI infrastructure, you can achieve your training goals and transform your business objectives into reality.

To learn more about distributed training, please review GKE example, Cluster Toolkit example, and Vertex AI example.

Search is at the heart of how we interact with the digital ecosystem, from online shopping to finding critical information. Enter generative AI, and user expectations are higher than ever. For applications to meet diverse user needs, they need to deliver fast, accurate and contextually relevant results, regardless of how queries are framed. For example:

• Online shoppers expect to find “waterproof hiking boots with ankle support” just as easily as a specific “SummitEdge Pro” model.

• Legal professionals need to pinpoint precise case citations or explore nuanced legal concepts with varied search terms.

• Doctors require precision when searching for critical patient information. A doctor looking for “allergy to penicillin” must locate the record accurately, whether the information is labeled as “drug sensitivities” or misspelled as “peniciln”. 

Spanner, Google’s always-on multi-model database with virtually unlimited scale, addresses these challenges with AI-powered hybrid search capabilities. Spanner allows developers to combine vector search, full-text search, and machine learning (ML) model reranking capabilities in a unified platform directly integrated with the operational data store, using a familiar SQL interface. 

In this post, we will explore how you can build a customized search engine for an ecommerce marketplace using Spanner.

Building a tailored search engine on Spanner

For ecommerce – along with many other industries – a single search method often falls short, resulting in dissatisfied users, incomplete information, or lost revenue. Keyword search excels at precision but struggles with alternate phrasing or natural language; vector search captures semantics but may overlook specific terms. Combining the strengths of both would enable organizations to deliver a more effective search experience.

SpanMart, a hypothetical ecommerce marketplace, allows users to search for products using keywords or natural language. Its products table supports multiple search methods with two specialized columns and associated indexes:

• A description_tokens column: This is a tokenized version of the description column, breaking down the text into individual terms. A search index (products_by_description) on this column accelerates full-text search, acting like an inverted index in information retrieval.

• An embedding column: This stores vector representations of the product descriptions, capturing semantic meaning rather than individual words. Similar descriptions are mapped close together in the “embedding space”. These embeddings are generated using models like Vertex AI Embeddings. A vector index (products_by_embedding) organizes these embeddings using a ScaNN tree structure for efficient semantic searches.

Here’s how the products table and its indexes are defined Spanner:

With these components in place, SpanMart can build an intelligent search pipeline that integrates: 

• Vector search for semantic relevance.
• Full-text search for precise keyword matching.
• Result fusion for combining the results from different retrieval methods. 
• ML model reranking for advanced result refinement.

This pipeline operates entirely within Spanner, where the operational data is stored. By avoiding integration with separate search engines or vector databases, Spanner eliminates the need for multiple technical stacks, complex ETL pipelines, and intricate application logic for inter-system communication. This reduces architectural and operational overhead and avoids potential performance inefficiencies.

The diagram below illustrates a high-level overview of how these components work together in Spanner.

Combining the power of vector and full-text search

When a user searches for products on SpanMart, the system first uses the embedding model to convert the user query into a vector that captures its semantic meaning. Then, SpanMart can build two queries:

• Approximate nearest neighbors (ANN) vector search:

• Full-text search (FTS):

These two queries excel in different scenarios and complement each other. For instance, when a user searches for a specific product model number, such as “Supercar T-6468”, the full-text search query can accurately find the exact model, while the vector search query suggests similar items. Conversely, for more complex natural language queries, such as “gift for an 8-year-old who enjoys logical reasoning but not a toy”, full-text search may struggle to yield useful results, whereas vector search can provide a relevant list of recommendations. Combining both queries would produce robust results for both styles of searches.

Reciprocal rank fusion (RRF)

RRF is a simple yet effective technique for combining results from multiple search queries. It calculates a relevance score for each record based on its position in all result sets, rewarding records ranked highly in individual searches. This method is particularly useful when the relevance scores from the individual searches are calculated in different spaces, making them difficult to compare directly. RRF addresses this by focusing on the relative rankings instead of scores within each result set.

Here’s how RRF works in our example:

• Calculate rank reciprocals: For each product, calculate its rank reciprocal in each result set by taking the inverse of its rank after adding a constant (e.g., 60). This constant prevents top-ranked products from dominating the final score and allows lower-ranked products to contribute meaningfully. For instance, a product ranked 5th in one result set would have a rank reciprocal of 1/(5 + 60) = 1/65 in that result set.

• Sum rank reciprocals: Sum the rank reciprocals from all result sets to get the final RRF score of a product.

The formula for RRF is –

 – where:

• d is a product description

• R is the set of retrievers (in this case, the two search queries)

• rank_r (?) is the rank of product description d in the results of retriever r.

• k is a constant

Implementing RRF within Spanner’s SQL interface is relatively straightforward. Here’s how:

Explanations:

• Common table expressions (CTEs): These are the WITH clauses, which are used in this query to improve readability. However, due to a current limitation, they may cause the query optimizer to default to an older version that lacks full-text search support. For now, the query uses the @{optimizer_version=7} hint to suggest a more recent optimizer version. 

• ANN CTE: This is the same as the previous ANN query, but with a twist. We assign a rank to each product in the results. While Spanner doesn’t support a direct way to assign ranks, there’s a workaround. By converting the results into an array of structs, we can use the offset of each element within the array as its rank. Since array offsets start at zero, we use offset + 1 to represent the actual rank. Note that this is purely a SQL language workaround without performance impact. The query planner effectively optimizes away the array conversion and directly assigns an offset to each row in the result set.

• FTS CTE: Similarly, this part mirrors the previous full-text search query, with the rank assigned using the array offset.

• Combining and ranking: The results from both CTEs are unioned, and grouped by the product id. For each product, we calculate the rrf_score and then select the top 50 products.

While RRF is an effective technique, Spanner’s versatile SQL interface empowers application developers to explore and implement various other result fusion methods. For instance, developers can normalize scores across different searches to a common range and then combine them using a weighted sum, assigning different importance to each search method. This flexibility allows for fine-grained control over the search experience and enables developers to tailor it to specific application requirements.

Using an ML model to rerank search results 

ML model-based reranking is a powerful way of refining search results to deliver improved results to the users. It applies an advanced yet computationally expensive model to a narrowed set of initial candidates, retrieved using methods like vector search, full-text search, or their combination, as discussed earlier. Due to its high computational cost, ML model-based reranking is applied after the initial retrieval reduces the result set to a small set of promising candidates.

Spanner’s integration with Vertex AI makes it possible to perform ML model-based reranking directly within Spanner. You can use a model deployed to your Vertex AI endpoint, including those available from the Vertex AI Model Garden. Once the model is deployed, you can create a corresponding reranker MODEL in Spanner.

In this example, SpanMart employs a Cross-Encoder model for reranking. This model takes two text inputs –  text and text_pair – and outputs a relevance score indicating how well the two texts align. Unlike vector search, which uses an embedding model to independently map each text into a fixed-dimensional space before measuring their similarity, a Cross-Encoder directly evaluates the two texts together. This allows the Cross-Encoder to capture richer contextual and semantic nuances in complex queries, such as “gift for an 8-year-old who enjoys logical reasoning but not a toy”. In a more advanced setup, the reranker could leverage a custom-trained model that incorporates additional signals such as product reviews, promotions, and user-specific data like browsing and purchase history, to offer an even more comprehensive search experience.

Once this model is defined in Spanner, we can proceed to add reranking on the initial search results using the following query:

Explanations:

• ANN, FTS and RRF CTEs: These are the same previously defined approximate nearest neighbors, full-text search and reciprocal rank fusion queries, respectively.

• ML.PREDICT Ranking: This step applies the reranker model to each product description as text from the RRF results, along with the search query as text_pair. The model assigns a relevance score to each product. The products are then sorted by these scores, and the top 10 are selected.

Get started

In this post, we demonstrated one approach to combine full-text search and vector search in Spanner, but developers are encouraged to explore other approaches, such as refining full-text search results with vector search, or combining multiple search results with customized fusion methods.

Learn more about Spanner and try it out today. For additional information, check out:

• Spanner vector search documentation and codelab

• Spanner full-text search overview

• Spanner vertex AI integration overview and tutorial on generating ML predictions using SQL

For enterprises, brilliance isn’t just about individual genius – it’s about the collective intelligence within an organization. But this brilliance is often hidden in silos, inaccessible to those who need it most, when they need it most. Our studies show that enterprise workers use an average of four to six tools just to ask and answer a question¹.

Generative AI holds immense promise for employee productivity. That’s why today, we’re introducing Google Agentspace. It unlocks enterprise expertise for employees with agents that bring together Gemini’s advanced reasoning, Google-quality search, and enterprise data, regardless of where it’s hosted. Google Agentspace makes your employees highly productive by helping them accomplish complex tasks that require planning, research, content generation, and actions – all with a single prompt.   

Three ways in which Google Agentspace unlocks enterprise expertise:

1. New ways to interact and engage with your enterprise data using NotebookLM: We built NotebookLM to help users make sense of complex information, and now we’re bringing this capability to enterprises. With NotebookLM Plus, your employees can upload information to synthesize, uncover insights, and enjoy new ways of engaging with data, such as podcast-like audio summaries and more. It’s the same experience millions of users love, now enhanced with security and privacy features for work. We’re also starting to roll out the experimental version of Gemini 2.0 Flash in NotebookLM. You can sign up for the early access program for NotebookLM for enterprise today. 

2. Information discovery across the enterprise: Google Agentspace gives employees a single, company-branded multimodal search agent that acts as a central source of enterprise truth for your entire organization. Building on the best of Google’s search capabilities, Agentspace can provide conversational assistance, answer complex questions, make proactive suggestions, and take actions based on your company’s unique information. Google Agentspace can do this across unstructured data – such as documents and emails – and structured data such as tables. We’ve also built translation in, so that you can understand all information, even if it originates in another language. With pre-built connectors for the most commonly used third-party applications, such as Confluence, Google Drive, Jira, Microsoft SharePoint, ServiceNow, and more, your employees can now easily access and query relevant data sources, and make better decisions.

3. Expert agents to automate your business functions: Google Agentspace is the launch point for custom AI agents that apply generative AI contextually. Now, enterprises can empower their employees in marketing, finance, legal, engineering, and more to conduct better research, draft content quickly, and automate repetitive tasks, including multi-step workflows. With Google Agentspace, enterprises can scale AI by providing a single location where employees can easily discover and access agents for their organization. Soon, employees will be able to use a low-code visual tool to build and tune their own expert agents with Google Agentspace.

With Google Agentspace, business analysts can effortlessly uncover industry trends and create compelling, data-driven presentations fueled by AI-generated insights. HR teams can revolutionize the employee experience with streamlined onboarding, even for complex tasks like 401k selection. Software engineers can proactively identify and resolve bugs, enabling them to build and iterate with greater efficiency, and accelerate deployment cycles. Marketers can unlock deeper performance analysis, optimize content recommendations, and fine-tune campaigns to achieve better results. And with multi-language support, all employees – no matter their region – can unlock enterprise expertise with agents. Sign up for the early access program. 

Finally, security is always top of mind. Google Agentspace is built on Google Cloud’s secure by design infrastructure, giving you the peace of mind to confidently deploy AI agents across your organization. Plus, we provide granular IT controls, including role-based access control (RBAC), VPC service controls, and IAM integration, ensuring your data remains protected and compliant at all times. 

Hear from our customers 

Our enterprise customers are already seeing great promise around connecting their employees with the wealth of their enterprise’s information. 

“At Deloitte, knowledge fuels our work. Our knowledge management teams are impressed with Google Agentspace’s ability to unify information scattered across various data silos. With Agentspace, our professionals can find the information they need instantly, boosting productivity and efficiency. Beyond search, the AI agentic capabilities within Google Agentspace can empower teams to take action on this knowledge, enabling us to deliver solutions and achieve client outcomes faster. In the long term, we see Agentspace as a key driver in transforming how we serve clients, fostering deeper collaboration, and unlocking new levels of insight and innovation across our organization.” – Kevin Laughridge, Alphabet Google US Lead Alliance Partner, Deloitte Consulting LLP 

“At Nokia, we create technology that helps the world act together. to connect and improve the world. Google Agentspace has the potential to revolutionize how our teams across Nokia find and leverage critical insights. We’re particularly excited by Google Agentspace’s ability to blend various data sources quotes and deliver personalized, contextually relevant answers. By unifying our knowledge resources, providing AI powered assistance and automating workflows, we strive towards reduced time spent searching for information, faster decision-making, and improved collaboration and productivity.” – Alan Triggs, Chief Digital Officer, Nokia 

“At Decathlon, we’re driven by a passion for sports and a commitment to innovation. Google Agentspace’s ability to connect teams with the right information from across organizations and provide self-service assistance is incredibly promising. Google Agentspace could become an essential enabler for product designers, marketers, and researchers, enabling them to make faster, more informed decisions and ultimately delivering even better experiences for customers. Our cross-functional teams — from data to workplace tools — see it as a great promise to meet our needs,” – Youssef BAKKALI, Group Product Manager, Decathlon

“At Onix we help businesses harness the power of technology to drive innovation and growth. Google Agentspace has immense potential to transform how enterprises deploy agents and access enterprise data at scale. By connecting disparate data sources and leveraging Google’s advanced AI for agentic workflows, Google Agentspace empowers organizations to unlock new levels of creativity. These last few months Onix has been partnering with Google to onboard early customers, and we are excited to see our clients take their first steps on this transformative journey.” – Ramnish Singh, SVP Solutions Engineering, Onix 

“Banco BV is committed to providing our team with the most innovative tools and technologies for collaboration. Google Agentspace is helping us realize the vision of enabling our employees to leverage some of the most advanced generative AI technologies for search, assistance and actions across all our critical systems in a secure and compliant way. We are particularly excited to see how our early users are leveraging Google Agentspace for faster and more comprehensive analysis and engaging with content in new multimodal ways that reflect our vision of work as more relaxed and collaborative.” – Fabio Jabur, Head of Data and AI, Banco BV 

“Google Agentspace has the potential to redefine how enterprises leverage knowledge, particularly through the power of AI agents. We’re excited to partner with Google Cloud to bring this to our customers, enabling them to securely use generative AI to enhance productivity, streamline workflows, and drive innovation. Google Agentspace’s unique ability to personalize search, provide AI assistance, and leverage innovations like NotebookLM will revolutionize how knowledge workers discover, generate, and utilize information. We believe that agents represent the future of knowledge work, and Google Agentspace is at the forefront of this exciting evolution.” – Gaurav Johar, Head of Applied AI, Quantiphi

“Our team of over 12,000 dedicated staff is united by a shared purpose: to make every day a little better for our customers by keeping daily essentials within reach for all. As Singapore’s largest food retailer, we understand that our people are the key to delivering on our social mission. To empower them with the tools they need to put their best foot forward every day, we are building an organization-wide research and assistance platform with Google Agentspace, with the aim of reducing manual effort. Google Agentspace will equip our employees with both a seamless research assistant that searches across all our documents, internal systems, and even third-party applications, and a task assistant using natural language. Our ultimate goal is to improve knowledge discovery and streamline processes across our enterprise. By enabling our people to access and share knowledge effortlessly, we can foster improved customer experiences and enhanced operational efficiency.” – Dennis Seah, Chief Digital & Technology Officer, FairPrice 

Hear from users at Deloitte Consulting LLP

“My work involves various market analyses as a strategy professional. It normally takes us a couple of weeks to read through all the research material. NotebookLM allowed us to get the initial insights in minutes and spend the time going deeper. It helps my team collaborate from a single source, pooling notes and links, freeing up more of our research time for strategic thinking and brainstorming allowing us to deliver compelling insights to our clients.” – Parinda Gandevikar, Senior Consultant, Deloitte Consulting LLP  

“NotebookLM’s podcast uncovered hidden insights in our GenAI report, revealing differences between industry sectors. Despite having read it multiple times, I had not caught this nuance. It was a real ‘aha’ moment.” – Laura Shact, Human Capital AI Lead, Deloitte Consulting LLP

“It’s a game changer for drawing insights from meetings. We took the recordings and uploaded it to NotebookLM and it did an amazing job of synthesizing the key findings.” – Tom Cahill, Senior Consultant, Deloitte Consulting LLP

“Before a long drive, I upload earnings call transcripts of recent tech companies and listen to the audio overview on my drive. It’s great how the key insights are distilled and interwoven with broader context that’s not even in those documents.” – Vivek Kulkarni, GenAI Transformation Lead, Deloitte USA LLP 

“I run a practice which involves creating and sharing a lot of different types of enablement content for the practice. It was delightful for our team to go from paging through dozens of documents to quickly finding the information we need, with the source document just one click away in NotebookLM.” – Gopal Srinivasan, Alphabet Google AI Alliance Lead, Deloitte Consulting LLP

Get started

Unlocking your organization’s collective brilliance is an investment in its future success. By connecting employees to enterprise expertise, you can create a more informed, innovative, and agile organization. 

Learn more about Google Agentspace and sign up for early access today.

^{1. Source: Survey of Knowledge Worker Communication Needs, Google Inc. (2024)}

Written by: Muhammad Umair

Here at Mandiant FLARE, malware reverse engineering is a regular part of our day jobs. At times we are required to perform basic triages on binaries, where every hour saved is critical to incident response timelines. At other times we examine complicated samples for days developing comprehensive analysis reports. As we face larger and more complex malware, often written in modern languages like Rust, knowing where to go, what to look at, and developing a “map” of the malware forms a significant effort that directly impacts our response times and triage effectiveness.

Today we introduce a new tool, XRefer (pronounced eks-reffer), which aims to shoulder some of this burden for anyone who endeavors to go down these rabbit holes like us, helping analysts get to the important parts faster while maintaining the context of their investigation.

Introduction

XRefer provides a persistent companion view to assist analysts in navigating and understanding binaries. It’s a modular and extensible tool that comes in the form of an IDA Pro plugin. Figure 1 shows the XRefer interface.

At its core, XRefer offers two complementary navigation paradigms:

1. Gemini-powered cluster analysis, which decomposes the binary into functional units and leverages the large language model (LLM) to describe their purpose and relationships. Think of this like viewing a city from Google Maps: you can quickly identify the business districts, residential areas, and green spaces. In binary terms, this feature helps identify functional groupings like command-and-control communication, persistence mechanisms, or information-gathering routines, giving you a strategic view of the malware’s architecture.

2. A context-aware view, which dynamically updates based on your current location in the code. This view presents both immediate artifacts of the current function and those from related functions along the same execution path. It’s similar to standing outside a shopping mall with X-ray vision: without entering each store, you can see the restaurant menus, shop inventories, and services offered on each floor. This allows you to make informed decisions about which areas deserve deeper investigation. Just as a mall directory helps you efficiently plan your shopping route, XRefer’s context-aware view helps analysts quickly identify relevant code paths by surfacing APIs, strings, capa matches, library information, and other artifacts that might otherwise require manual exploration of multiple functions. 

Let’s take a closer look at each of these paradigms, beginning with cluster-based navigation.

The Birds-Eye View: Cluster-Based Binary Navigation

One of XRefer’s key features is its ability to break down a binary into functional units, providing an immediate high-level understanding of its architecture. To demonstrate this capability, let’s examine an ALPHV ransomware sample written in Rust. Despite containing over 2,700 functions, XRefer’s analysis organizes key functionality of this complex binary into clear functional clusters, as shown in the Cluster Relationship graph in Figure 2.

These functional clusters are descriptively labelled as follows:

• Ransomware Main Module

• Configuration Parsing Module    

• User Profile and Process Information Module

• Privilege Escalation, System Information, and AntiAnalysis Module

• File Processing Pipeline Module

• Network Communication and Cluster Management Module

• Thread Synchronization and Keyed Events Module

• File Path and Encryption Key Generation Module

• Console Clearing Module

• UI Rendering and Console Output Module

• Image Generation and Encoding Module

• Data Encoding and Hashing Module

• File Discovery and Dispatch Module

• Thread Synchronization and Time Management Module

While each cluster contains deeper sub-clusters that analysts can explore, we’ll focus on the high-level view for now. The clustering and relationship identification is performed through static analysis. XRefer then leverages Gemini to provide natural language descriptions of each cluster and how they relate to one another. Figure 3 illustrates key components in the graph navigation interface.

At the top, the view provides a brief description of the binary’s functionality and its category. Next, it describes the currently selected cluster and its relationships to other clusters. For convenient navigation, the cross-references of that cluster are listed, followed by a visual graph representation. For readability, these details are transcribed in Table 1.

The clusters can also be viewed in a linear format within XRefer’s interface, as shown in Figure 1.

To better demonstrate cluster navigation visually, we’ve used a lightweight backdoor that displays more clearly on screen. Figure 4 provides a quick glimpse of the cluster navigation workflow, showing how analysts can quickly browse clusters and navigate to their respective functions in the disassembly or pseudocode views.

The navigation can automatically sync with clusters—when you navigate to a function that belongs to a known cluster, XRefer can automatically open that cluster’s view and highlight the current function within it. XRefer offers two approaches to clustering:

1. Cluster all paths that are part of XRefer’s analysis (XRefer’s analysis is discussed later)

2. Cluster a focused subset of functions, pre-filtered by Gemini based on their artifacts

Note: Throughout this blog post, we use the term “artifacts” to refer to binary elements like strings, API calls, library references, and other extractable information that help understand program behavior.

By default, XRefer employs the first method. While this approach is comprehensive, it may create additional clusters around unidentified libraries in the program. These library clusters are typically easy to identify and exclude from analysis.

The second clustering method is optionally available via the context menu and proves valuable for automatically filtering out library, runtime/compiler artifacts, and repetitive noisy functions. However, due to the inherent nature of LLMs, this approach can be inconsistent—artifacts might be missed, and results can vary between runs. While missed artifacts can usually be recovered through a quick re-run of the LLM analysis, this variability remains an inherent characteristic of this approach.

XRefer can also display these LLM-filtered artifacts in a dedicated view, separate from the clustering visualization. This view, shown in Figure 5, provides analysts with a streamlined overview of the binary’s most relevant artifacts while filtering out noise like library functions and runtime artifacts.

It’s important to note that clusters aren’t perfect boundaries. They may not capture every related function and can contain functions that are reused across different parts of the binary. However, any missed related functions will typically be found in the vicinity of their logical cluster, and reused functions are generally easy to identify at a glance. The goal of clustering is not to create strict divisions, but rather to establish general zones and subzones of related functionality.

Function Labeling: Prefixing Cluster Membership in Names

XRefer can optionally encode cluster information directly into IDA’s interface by prefixing function names. These labels provide architectural context directly inside the disassembly and pseudocode windows. Table 2 shows the classification system used to prefix functions based on their cluster relationships.

Down in the Trenches: Context-Aware Code Navigation

Having seen how XRefer’s cluster analysis provides a high-level view of binary architecture, let’s examine its second navigation paradigm: a context-aware view that updates automatically based on the function currently being analyzed.

The function context table (shown in Figure 6) organizes information into three main components:

1. Cluster Membership – At the top, displaying which clusters the current function belongs to. Functions appearing in multiple clusters often indicate utility or helper functions rather than specialized functionality.

2. Direct References – Listed under “DIRECT XREFS,” showing artifacts directly used or called by the current function.

3. Indirect References – Categorized tables prefixed with “INDIRECT,” showing artifacts used by all functions called through the current function’s execution paths. This provides a preview of downstream functionality without requiring manual traversal of each function.

Both direct and indirect references include:

• APIs and API traces

• Strings

• Libraries

• capa results

For direct references, each artifact is listed with its reference addresses. Double clicking these addresses jumps to their exact location in the current function. For indirect artifacts, the displayed addresses are different—they point to function calls within the current function that eventually lead to those artifacts through execution paths. This x-ray-like capability eliminates the tedious process of diving into nested function calls to discover artifacts and functionality, only to return to the primary function.

XRefer extends this visibility through its Peek View feature, accessible via the context menu. When enabled, clicking on any function dynamically filters the artifact view to display only those elements that lie along its execution paths. This instant preview allows analysts to quickly assess a function’s downstream behavior without manually tracing through its call graph, significantly streamlining the exploration of complex codebases. Figure 7 demonstrates how this functions in practice.

Beyond Peek View, XRefer offers on-demand artifact filtering through key bindings. A core design principle of XRefer is the uniform treatment of all artifact types—any operation available for one type of artifact is consistently available across all others. For instance, path analysis capabilities that work with API references can be similarly applied to capa results. Let’s examine the key functionalities available under this navigation paradigm.

Path Graphs

XRefer can generate and visualize all simple paths from the entry point to any type of artifact. These interactive graphs serve as powerful navigation tools, particularly when analysts need to trace specific functionality through the binary. Figure 8 demonstrates this capability by displaying all execution paths leading to GetComputerNameW in the ALPHV ransomware sample. Each function node in the graph provides a contextual pop-up showing its complete artifact inventory.

Path graphs include a simplification feature that can reduce visual complexity by omitting nodes that either contain no artifacts or contain only excluded artifacts (exclusions are discussed later). Figure 10 illustrates this simplification, where the graph is reduced from 15 to 12 nodes, representing a 20% reduction in complexity. While more complex graphs can achieve higher simplification ratios, their full visualization extends beyond the practical constraints of this blog post.

Search

XRefer provides unified search functionality across all artifact types directly within the function context table view. Figure 8 demonstrates this capability while searching for the API reference used in path graph generation.

Cross References++

XRefer implements its own cross-reference view that goes beyond IDA Pro’s traditional functionality (accessed via “X”). This view, similarly triggered by pressing “X” on any artifact, encompasses all artifact types, including elements that IDA Pro cannot typically track, such as capa results, APIs identified through dynamic traces, and strings extracted by language-specific modules like the Rust parser.

Trace Navigation

While API traces are integrated throughout XRefer’s interface—from function context tables (Figure 6) to information pop-ups (Figure 8)—the plugin also offers dedicated trace navigation modes. These three modes, illustrated in Figure 11, provide views of API calls with their arguments, each offering different levels of scope:

• Function Scope – Shows only the API calls made directly within the current function, providing a clean view of its immediate external interactions

• Path Scope – Reveals all API calls that occur downstream from the current function, following its execution paths. This helps analysts understand the complete chain of system interactions triggered by a particular function.

• Full Trace – Displays the complete API trace captured during dynamic analysis, regardless of static code paths. Useful when you may have calls associated with encrypted regions in the binary or generally from dynamically resolved APIs.

Artifact Exclusion

XRefer supports artifact exclusion to reduce noise when analyzing large, complex binaries. Excluded artifacts are omitted from multiple views and processes including the main interface, cluster analysis, and simplified path graphs.

Artifacts can be excluded in two ways:

• Directly through XRefer’s interface, where multiple artifacts can be selected and excluded using key bindings

• Via the settings dialog (shown in Figure 12), which supports wildcard-based exclusion patterns. For instance, noisy Rust standard library references can be filtered using patterns like std*, providing efficient bulk exclusion of known noise sources.

These exclusions persist across sessions, allowing analysts to maintain their preferred filtering setup.

Specialized Rust Support

XRefer has a specialized Rust module (discussed later) that extracts strings and library usage information. During Rust compilation, the compiler embeds library source paths that typically appear adjacent to their corresponding library code within functions. These compiler-inserted references serve two key purposes as function artifacts:

• Identifying library dependencies and their specific functionality within code sections

• Providing positional hints that help locate where library code is actually implemented within functions

This compiler-provided context feeds into both XRefer’s cluster analysis and enhances manual navigation, helping analysts quickly locate relevant code regions while understanding which Rust library implementations are being used. The module also includes a basic function renaming capability for Rust binaries, which will be covered in detail later.

Auxiliary Features

Before concluding this section, two standalone features warrant mention, though they operate independently of the clustering mechanism and exclusion system.

Boundary Method Scanning: XRefer allows multiple artifacts to be selected in the function context view for boundary scanning. This operation identifies the Lowest Common Ancestor (LCA) in the call graph for all selected artifacts. In niche scenarios, this can be used to isolate specific functionality based on a subset of artifacts and identify the most specific parent function that encompasses all selected artifacts without intermediate functions. While originally intended to serve a larger purpose in the clustering mechanism, the clustering system ultimately took a different direction, leaving this as a standalone feature.

String Lookups: During processing, XRefer can optionally query strings against public Git repositories through Grep App for categorization purposes. This is strictly a placeholder implementation; locally maintained databases by teams or individuals would be better suited for these queries. The feature operates independently of XRefer’s broader ecosystem, primarily serving to categorize known strings for noise reduction or occasional OSINT insights.

Under the Hood: XRefer’s Analysis Engine

Having explored both navigation paradigms, let’s examine the technical foundation that makes them possible. While a deep dive into XRefer’s internals would warrant its own blog post, understanding a high-level view of its analysis pipelines and extensibility will help set the context.

Ingestion Sources

XRefer builds its understanding of binaries through two primary data ingestion channels:

1. Internal – Whereby it extracts and processes all the imports, strings and any library data from the binary itself. This also involves language-specific modules. XRefer comes out of the box with a Rust-specific language module, and more can be added.

2. External – This includes API traces from third-party tooling and results from capa’s analysis. XRefer currently supports ingestion of API traces from VMRay’s sandbox and Cape Sandbox. Additional modules can be written, for instance, TTD traces would be a good candidate.

Note: Out of the mentioned traces, VMRay produces the best results. Cape sandbox, while good, hooks NT* APIs and is much noisier in terms of visual display as it loses 1:1 API to import mapping. Unfortunately, that’s one of the very few open-source solutions available right now.

XRefer feeds the available data along with cluster relationships in the form of call flows to the LLM, which generates semantic descriptions for each cluster, their relationships, and the overall binary. As demonstrated in Table 3, while external data sources enhance the analysis, XRefer can produce meaningful results even with just internal binary analysis.

Note: While LLM-generated labels and descriptions may vary in phrasing between runs, they tend to consistently convey similar semantic information.

Language Modules and Rust

XRefer supports language-specific analysis through dedicated modules and ships with a module for Rust binaries. This allows for specialized handling of language-specific characteristics. The Rust module provides:

• Identification of rust_main 

• Parsing of Rust thread objects and their indirect calls, improving path coverage

• Extraction of library/crate information for better code understanding in both cluster analysis and context tables

• Limited function renaming capabilities for a subset of functions where no inlining conflicts are detected, based on compiler strings and excluding runtime/std/internal libraries

Rust Function Renaming

The Rust module includes a limited function renaming capability for Rust binaries, though its approach is deliberately restrictive. While XRefer is not a function identification tool, it can leverage compiler strings embedded in Rust binaries for basic renaming. Due to the prevalence of inlining from compiler optimizations, the module only renames functions where no inlining is detected and excludes internal references (std*, core*, alloc*, etc.) which are particularly prone to inlining.

The renaming doesn’t provide the full function names but does include up to the submodule name. Again, this is not a substitute for proper function identification as only a very small handful of functions can be renamed this way. However, since this capability is not version-specific to the toolchain, platform, and/or crate, it represents a low-hanging fruit that would have been unwise to ignore. As an example, in the ALPHV binary, the Rust module was safely able to rename 362 out of ~2,700 functions.

LLM Analysis and Extensibility

XRefer ships with support for Gemini through a modular provider system. The architecture is designed for extensibility—new LLM providers can be added by implementing a simple interface that handles model-specific requirements like rate limiting and token management. It should be noted that cluster analysis, especially for large binaries, requires large context windows, an area in which Gemini models excel compared to other models.

Similarly, the prompt system is built to be extensible. New prompts can be easily added by creating prompt templates and corresponding response parsers. This allows XRefer’s analysis capabilities to grow as new use cases are identified or as LLM technology evolves.

It’s important to note that XRefer currently limits its LLM analysis to artifacts and function/cluster relationships (in the form of call flows) without submitting any actual code. For example, when analyzing a network communication module, XRefer provides the LLM with a rich set of artifacts like API calls (send, recv), strings (URLs, User-Agents), dynamic API traces, capa matches (network communication, socket operations), library/crate information, and their relationships in the call graph, rather than the underlying implementation code. This is just one simplified example. The actual analysis encompasses the full spectrum of artifacts XRefer collects. 

The effectiveness of this approach depends on the successful extraction of these artifacts, whether through specialized language modules, internal binary analysis, or ingestion of external data sources. When artifacts are available, the Gemini-powered analysis effectively breaks down binary functionality into distinct functional units, providing an explorable architectural view of the binary. Code-level analysis represents the next logical step in XRefer’s evolution.

Path Analysis: The Foundation of XRefer’s Navigation

XRefer’s architecture is fundamentally entrypoint-centric. It constructs execution paths between entry points and functions containing artifacts, forming the backbone of both its clustering algorithm and the context-aware navigation capabilities described earlier. While clustering can be implemented without path analysis, our current approach of path-based clustering consistently produces decent results. Standalone clustering may come as a feature later down the road.

While entry point selection for PE executables is straightforward and automatic, DLL analysis may require analysts to select specific exports as starting points, since the default entry point might not be the most interesting one. XRefer allows analysts to analyze multiple entry points, providing flexibility in how they approach the binary.

Path analysis introduces computational overhead, but the benefits it provides to both navigation capabilities and clustering accuracy make this trade-off worthwhile. It’s important to note that this overhead is entirely front-loaded into the initial processing phase. Once analysis is complete, the pre-processed results ensure fluid navigation and responsiveness during actual usage. Table 4 shows what analysis times look like for several large binaries.

Note: Times include LLM queries for artifact discovery and cluster analysis. Actual duration varies based on system capabilities and binary complexity—from seconds for simple binaries to longer for complex ones. This listing aims to provide a general sense of expected time scales.

A notable limitation of path analysis arises in binaries with numerous indirect calls, where complete coverage cannot be guaranteed without proper resolution of these indirect targets.

Configuration

All LLM configurations and paths for external data sources can be managed through XRefer’s settings dialog.

XRefer supports the ingestion of indirect cross-references that IDA cannot resolve statically. Examples can include dynamically resolved addresses for indirect calls, such as virtual function calls through vtables in C++ binaries and function pointer calls. These resolutions can be imported from sources like debugger scripts or binary instrumentation frameworks, enabling XRefer to construct more complete execution paths.

UI/UX and Compatibility

XRefer is implemented as an interactive TUI (Text-based User Interface) plugin. While IDA’s simplecustviewer_t wasn’t designed to be a proper TUI system, additional Qt implementations have been added to improve the interface. Users may encounter bugs, which will be addressed as they are reported.

Some rules of thumb when working with XRefer:

• All addresses, regardless of where they are displayed in the TUI, are navigable by double clicking.

• Cluster IDs (cluster.id.xxxx), regardless of where they show up in the TUI, are also explorable by single clicking.

• If an address is the start of a function, hovering over it will always display an information pop-up about that function.

• Hovering over cluster IDs will display a pop-up with details about that cluster.

• Press ESC to return to the previous location/state (unless a graph is explicitly “pinned”).

XRefer maintains state within the current function and resets upon navigation to a new function. For detailed usage instructions and key bindings, please refer to the XRefer repository.

It is recommended to enable auto-resizing, which automatically adjusts XRefer’s window dimensions when viewing graphs and restores default size upon exit. While XRefer is designed to remain open as a persistent companion view, it includes an expand/collapse feature for quick access when needed. Figure 17 demonstrates these interface elements.

XRefer is recommended for use with either IDA <= 8.3 or IDA >= 9.0. IDA 8.4 contains a simplecustviewer_t visual bug that causes washed-out colors in several areas. Due primarily to the author’s preference for dark mode, the plugin currently provides better color contrast in dark themes compared to the default theme, though this may be balanced in future releases.

XRefer has been primarily tested with Windows binaries. While there are no fundamental limitations preventing support for ELF or Mach-O formats and XRefer may just work out of the box with most of them, some tweaks or implementation fixes might be required to ensure proper support.

How to Get XRefer

XRefer is now available as an open-source tool in Mandiant’s GitHub repository. Installation requires installing Python dependencies from requirements.txt and copying the XRefer plugin to IDA’s plugin directory.

Note that one of the dependencies, asciinet, requires Java (JRE or OpenJDK) to be configured on your system. For detailed installation instructions, please refer to the XRefer repository.

Another alternative is to use FLARE-VM, which sets up a reverse engineering environment with a lot of useful tools, including XRefer.

Future Work

This is the initial release of XRefer and thus includes some implementations that are early in their maturity. While LLMs may eventually evolve to accurately interpret all forms of source and compiled code, the current approach focuses on systematic analysis rather than treating LLMs as a black box for binary summarization. This methodology not only provides high-level insights but actively supports analysts in their detailed investigation workflows.

Immediate areas for future development include, beyond bug fixes and UI/UX refinements:

• Extend cluster analysis to include code submissions, improving not just analysis at scale but also providing targeted insights for manual reverse engineering workflows

• Research and potentially implement path-independent clustering methodologies (the primary benefit here would be speed improvements if path analysis is not required)

• Implement LLM-based cluster merging (helps neatly tuck away similar clusters such as those belonging to a library)

• Ensure proper support for non-Windows file formats

• Add support for other language modules, particularly Golang

Currently, XRefer is tightly coupled with IDA due to its TUI implementation. As the core matures, the processing engine may be decoupled into a standalone package.

Acknowledgements

Special thanks to Genwei Jiang and Mustafa Nasser for their code contributions to XRefer and to Ana Martinez Gomez for including XRefer in the default FLARE-VM configuration. Additional thanks to the FLARE team members who provided valuable feedback through their use of XRefer.