GCP – Cloud CISO Perspectives: Recent advances in how threat actors use AI tools

Welcome to the first Cloud CISO Perspectives for November 2025. Today, Sandra Joyce, vice-president, Google Threat Intelligence, updates us on the state of the adversarial misuse of AI.

As with all Cloud CISO Perspectives, the contents of this newsletter are posted to the Google Cloud blog. If you’re reading this on the website and you’d like to receive the email version, you can subscribe here.

Recent advances in how threat actors use AI tools

By Sandra Joyce, vice-president, Google Threat Intelligence

Based on GTIG’s unique visibility into the misuse of AI tools and the broader threat landscape, the new report details four key findings on how government-backed threat actors and cybercriminals are integrating AI across their entire attack lifecycle. By understanding how adversaries are innovating with AI, security leaders can get ahead of threats and take proactive measures to update their security posture against a changing threat landscape.

1. AI generating commands to steal documents and data

For the first time, GTIG has identified malware families that use large language models (LLMs) during execution. These tools can dynamically generate malicious scripts, use self-modification to obfuscate their own code to evade detection, and receive commands from AI models rather than traditional command-and-control (C2) servers.

One such new malware detailed in the full report is a data miner we track as PROMPTSTEAL. In June, GTIG identified the Russian government-backed actor APT28 (also known as FROZENLAKE) using PROMPTSTEAL, which masquerades as an image generation program that guides the user through a series of prompts to generate images.

In the background, PROMPSTEAL queries the API for Hugging Face, a platform for open-source machine learning including LLMs, to generate commands for execution, rather than hard-coding commands in the malware. The prompt specifically asks the LLM to output commands to gather system information, to copy documents to a specified directory, and to exfiltrate data.

Our analysis indicates continued development of this malware, with new samples adding obfuscation and changing the C2 method.

FROZENLAKE’s use of PROMPTSTEAL constitutes our first observation of malware querying a LLM deployed in live operations. Combined with other recent experimental implementations of novel AI techniques, this campaign provides an early indicator of how threats are evolving and how adversaries can potentially integrate AI capabilities into future intrusion activity.

What Google is doing: Google has taken action against this actor by disabling the assets associated with their activity. Google DeepMind has also used these insights to further strengthen our protections against misuse by strengthening both Google’s classifiers and the model itself. This enables the model to refuse to assist with these types of attacks moving forward.

2. Social engineering to bypass safeguards

Threat actors have been adopting social engineering pretexts in their prompts to bypass AI safeguards. We observed actors posing as cybersecurity researchers and as students in capture-the-flag (CTF) competitions to persuade Gemini to provide information that would otherwise receive a safety response from Gemini.

In one interaction, a threat actor asked Gemini to identify vulnerabilities on a compromised system, but received a safety response from Gemini that a detailed response would not be safe. They reframed the prompt by depicting themselves as a participant in a CTF exercise, and in response Gemini returned helpful information that could be misused to exploit the system.

The threat actor appeared to learn from this interaction and continued to use the CTF pretext over several weeks in support of phishing, exploitation, and webshell development.

What Google is doing: We took action against the CTF threat actor by disabling the assets associated with the actor’s activity. Google DeepMind was able to use these insights to further strengthen our protections against misuse. Observations have been used to strengthen both classifiers and the model itself, enabling it to refuse to assist with these types of attacks moving forward.

3. Maturing cybercrime marketplace for AI tooling

In addition to misusing mainstream AI-enabled tools and services, there is a growing interest and marketplace for purpose-built AI tools and services that can enable illicit activities. To identify evolving threats, GTIG tracks posts and advertisements on underground forums related to AI tools and services as well as discussions surrounding the technology.

Many underground forum advertisements mirror language comparable to marketing for legitimate AI models, citing the need to improve the efficiency of workflows and effort while simultaneously offering guidance for prospective customers interested in their offerings.

The underground marketplace for illicit AI tools has matured in 2025. GTIG has identified multiple offerings of multifunctional tools designed to support phishing, malware development, vulnerability research, and other capabilities. This development has lowered the barrier to entry for less sophisticated, poorly-resourced threat actors.

What Google is doing: While there are no direct mitigations to prevent threat actors from developing their own AI tools, at Google we use threat intelligence to disrupt adversary operations — including monitoring the cybercrime AI tool marketplace.

4. Continued augmentation of the full attack lifecycle

State-sponsored actors from North Korea, Iran, and the People’s Republic of China (PRC) continue to misuse AI to enhance all stages of their operations, from reconnaissance and phishing lure creation to C2 development and data exfiltration.

In one example, GTIG observed a suspected PRC-nexus actor using Gemini to support multiple stages of an intrusion campaign, including conducting initial reconnaissance on targets, researching phishing techniques to deliver payloads, soliciting assistance from Gemini related to lateral movement, seeking technical support for C2 efforts once inside a victim’s system, and helping with data exfiltration.

What Google is doing: GTIG takes a holistic, intelligence-driven approach to detecting and disrupting threat activity. Our understanding of government-backed threat actors and their campaigns can help provide the needed context to identify threat-enabling activity. By tracking this activity, we’re able to leverage our insights to counter threats across Google platforms, including disrupting the activity of threat actors who have misused Gemini.

Our learnings from countering malicious activities are fed back into our product development to improve safety and security for our AI models. Google DeepMind was able to use these insights to further strengthen our protections against misuse. Observations have been used to strengthen both classifiers and the model itself, enabling it to refuse to assist with these types of attacks moving forward.

Building AI safely and responsibly

At Google, we are committed to developing AI responsibly and are taking proactive steps to disrupt malicious activity, disabling the projects and accounts associated with these threat actors. In addition to taking action against accounts, we have proactively fed the intelligence back into our teams and products to better protect Google and its users. We continuously improve our models to make them less susceptible to misuse, and share our findings to arm defenders and enable stronger protections across the ecosystem.

We believe our approach to AI must be both bold and responsible. That means developing AI in a way that maximizes the positive benefits to society while addressing the challenges. Guided by our AI Principles, Google designs AI systems with robust security measures and strong safety guardrails, and we continuously test the security and safety of our models to improve them.

For more on these shifting behaviors, along with the steps we’ve taken to thwart these efforts, you can read GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools here.

In case you missed it

Here are the latest updates, products, services, and resources from our security teams so far this month:

• How Google Does It: Threat modeling, from basics to AI: Threat modeling plays a critical role at Google in how we detect and respond to threats — and secure our use of the public cloud. Read more.
• How rapid threat models inject more reality into tabletops: Using rapid threat models in tabletop exercises can help you better understand how defense should adapt to the dynamic threat environment. Read more.
• How we’re helping customers prepare for a quantum-safe future: Google has been working on quantum-safe computing for nearly a decade. Here’s our latest on protecting data in transit, digital signatures, and public key infrastructure. Read more.
• HTTPS by default coming to Chrome: One year from now, with the release of Chrome 154 in October 2026, we will change the default settings of Chrome to enable “Always Use Secure Connections”. This means Chrome will ask for the user’s permission before the first access to any public site without HTTPS. Read more.
• How AI helps Android keep you safe from mobile scams: For years, Android has been on the frontlines in the battle against scammers, using the best of Google AI to build proactive, layered protections that can anticipate and block scams before they reach you. Read more.

Please visit the Google Cloud blog for more security stories published this month.

Threat Intelligence news

• A defender’s guide to privileged account monitoring: Privileged access stands as the most critical pathway for adversaries seeking to compromise sensitive systems and data. This guide can help you protect the proverbial keys to your kingdom with recommendations and insights to prevent, detect, and respond to intrusions targeting privileged accounts. Read more.
• Pro-Russia information operations leverage Russian drone incursions into Polish airspace: GTIG has observed multiple instances of pro-Russia information operations (IO) actors promoting narratives related to the reported incursion of Russian drones into Polish airspace that occurred in September. The IO activity appeared consistent with previously-observed instances of pro-Russia IO targeting Poland — and more broadly the NATO Alliance and the West. Read more.
• Vietnamese actors using fake job posting campaigns to deliver malware and steal credentials: GTIG is tracking a cluster of financially-motivated threat actors operating from Vietnam that use fake job postings on legitimate platforms to target individuals in the digital advertising and marketing sectors. Read more.

Please visit the Google Cloud blog for more threat intelligence stories published this month.

Now hear this: Podcasts from Google Cloud

• The end of ‘collect everything’: Moving from centralization to data access: Will the next big SIEM and SOC cost-savings come from managing security data access? Balazs Scheidler, CEO, Axoflow, and founder of syslog-ng, debates the future of security data with hosts Anton Chuvakin and Tim Peacock. Listen here.
• Cyber Savvy Boardroom: Valuing investment beyond the balance sheet: Andreas Wuchner, cybersecurity and risk expert, and board advisor, shares his perspective on how smart investments can transform risk management into a brand promise. Listen here.
• Behind the Binary: Building a robust network at Black Hat: Host Josh Stroschein is joined by Mark Overholser, a technical marketing engineer, Corelight, who also helps run the Black Hat Network Operations Center (NOC). He gives us an insider’s look at the philosophy and challenges behind building a robust network for a security conference. Listen here.

To have our Cloud CISO Perspectives post delivered twice a month to your inbox, sign up for our newsletter. We’ll be back in a few weeks with more security-related updates from Google Cloud.