Are you a cloud architect or IT admin tasked with ensuring deployments are following best practices and generating configuration validation reports? The struggle of adopting best practices is real. And not just the first time: ensuring that a config doesn’t drift from org-wide best practices over time is notoriously difficult.
Workload Manager provides a rule-based validation service for evaluating your workloads running on Google Cloud. Workload Manager scans your workloads, including SAP and Microsoft SQL Server, to detect deviations from standards, rules, and best practices to improve system quality, reliability, and performance. .
Introducing custom rules in Workload Manager
Today, we’re excited to extend Workload Manager with custom rules (GA), a detective-based service that helps ensure your validations are not blocking any deployments, but that allows you to easily detect compliance issues across different architectural intents. Now, you can flexibly and consistently validate your Google Cloud deployments across Projects, Folders and Orgs against best practices and custom standards to help ensure that they remain compliant.
Here’s how to get started with Workload Manager custom rules in a matter of minutes.
1) Codify best practices and validate resources Identify best practices relevant to your deployments from the Google Cloud Architecture Framework, codify them in Rego, a declarative policy language that’s used to define rules and express policies over complex data structures, and run or schedule evaluation scans across your deployments.
You can create new Rego rules based on your preferences, or reach out to your account team to get more help crafting new rules.
2) Export findings to BigQuery dataset and visualize them using Looker You can configure your own BigQuery dataset to export each validation scan and easily integrate it with your existing reporting systems, build a new Looker dashboard, or export results to Google Sheets to plan remediation steps.
Additionally, you can configure Pub/Sub-based notifications to send email, Google Chat messages, or integrate with your third-party systems based on different evaluation success criteria.
A flexible system to do more than typical config validation
With custom rules you can build rules with complex logic and validation requirements across multiple domains. You can delegate build and management to your subject matter experts, reducing development time and accelerating the time to release new policies.
And with central BigQuery table export, you can combine violation findings from multiple evaluations and easily integrate with your reporting system to build a central compliance program.
Get started today with custom rules in Workload Manager by referring to the documentation and testing sample policies against your deployments.
Need more help? Engage with your account teams to get more help in crafting, curating and adopting best practices.
Rapid advancements in artificial intelligence (AI) are unlocking new possibilities for the way we work and accelerating innovation in science, technology, and beyond. In cybersecurity, AI is poised to transform digital defense, empowering defenders and enhancing our collective security. Large language models (LLMs) open new possibilities for defenders, from sifting through complex telemetry to secure coding, vulnerabilitydiscovery, and streamlining operations. However, some of these same AI capabilities are also available to attackers, leading to understandable anxieties about the potential for AI to be misused for malicious purposes.
Much of the current discourse around cyber threat actors’ misuse of AI is confined to theoretical research. While these studies demonstrate the potential for malicious exploitation of AI, they don’t necessarily reflect the reality of how AI is currently being used by threat actors in the wild. To bridge this gap, we are sharing a comprehensive analysis of how threat actors interacted with Google’s AI-powered assistant, Gemini. Our analysis was grounded by the expertise of Google’s Threat Intelligence Group (GTIG), which combines decades of experience tracking threat actors on the front lines and protecting Google, our users, and our customers from government-backed attackers, targeted 0-day exploits, coordinated information operations (IO), and serious cyber crime networks.
We believe the private sector, governments, educational institutions, and other stakeholders must work together to maximize AI’s benefits while also reducing the risks of abuse. At Google, we are committed to developing responsible AI guided by our principles, and we share resources and best practices to enable responsible AI development across the industry. We continuously improve our AI models to make them less susceptible to misuse, and we apply our intelligence to improve Google’s defenses and protect users from cyber threat activity. We also proactively disrupt malicious activity to protect our users and help make the internet safer. We share our findings with the security community to raise awareness and enable stronger protections for all.
aside_block
<ListValue: [StructValue([(‘title’, ‘Adversarial Misuse of Generative AI’), (‘body’, <wagtail.rich_text.RichText object at 0x3e4a6bb3fa00>), (‘btn_text’, ‘Download now’), (‘href’, ‘https://services.google.com/fh/files/misc/adversarial-misuse-generative-ai.pdf’), (‘image’, None)])]>
Executive Summary
Google Threat Intelligence Group (GTIG) is committed to tracking and protecting against cyber threat activity. We relentlessly defend Google, our users, and our customers by building the most complete threat picture to disrupt adversaries. As part of that effort, we investigate activity associated with threat actors to protect against malicious activity, including the misuse of generative AI or LLMs.
This report shares our findings on government-backed threat actor use of the Gemini web application. The report encompasses new findings across advanced persistent threat (APT) and coordinated information operations (IO) actors tracked by GTIG. By using a mix of analyst review and LLM-assisted analysis, we investigated prompts by APT and IO threat actors who attempted to misuse Gemini.
Advanced Persistent Threat (APT) refers to government-backed hacking activity, including cyber espionage and destructive computer network attacks.
Information Operations (IO) attempt to influence online audiences in a deceptive, coordinated manner. Examples include sockpuppet accounts and comment brigading.
GTIG takes a holistic, intelligence-driven approach to detecting and disrupting threat activity, and our understanding of government-backed threat actors and their campaigns provides the needed context to identify threat enabling activity. We use a wide variety of technical signals to track government-backed threat actors and their infrastructure, and we are able to correlate those signals with activity on our platforms to protect Google and our users. By tracking this activity, we’re able to leverage our insights to counter threats across Google platforms, including disrupting the activity of threat actors who have misused Gemini. We also actively share our insights with the public to raise awareness and enable stronger protections across the wider ecosystem.
Our analysis of government-backed threat actor use of Gemini focused on understanding how threat actors are using AI in their operations and if any of this activity represents novel or unique AI-enabled attack or abuse techniques. Our findings, which are consistent with those of our industry peers, reveal that while AI can be a useful tool for threat actors, it is not yet the game-changer it is sometimes portrayed to be. While we do see threat actors using generative AI to perform common tasks like troubleshooting, research, and content generation, we do not see indications of them developing novel capabilities.
Our key findings include:
We did not observe any original or persistent attempts by threat actors to use prompt attacks or other machine learning (ML)-focused threats as outlined in the Secure AI Framework (SAIF) risk taxonomy. Rather than engineering tailored prompts, threat actors used more basic measures or publicly available jailbreak prompts in unsuccessful attempts to bypass Gemini’s safety controls.
Threat actors are experimenting with Gemini to enable their operations, finding productivity gains but not yet developing novel capabilities. At present, they primarily use AI for research, troubleshooting code, and creating and localizing content.
APT actors used Gemini to support several phases of the attack lifecycle, including researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques. Iranian APT actors were the heaviest users of Gemini, using it for a wide range of purposes. Of note, we observed limited use of Gemini by Russian APT actors during the period of analysis.
IO actors used Gemini for research; content generation including developing personas and messaging; translation and localization; and to find ways to increase their reach. Again, Iranian IO actors were the heaviest users of Gemini, accounting for three quarters of all use by IO actors. We also observed Chinese and Russian IO actors using Gemini primarily for general research and content creation.
Gemini’s safety and security measures restricted content that would enhance adversary capabilities as observed in this dataset. Gemini provided assistance with common tasks like creating content, summarizing, explaining complex concepts, and even simple coding tasks. Assisting with more elaborate or explicitly malicious tasks generated safety responses from Gemini.
Threat actors attempted unsuccessfully to use Gemini to enable abuse of Google products, including researching techniques for Gmail phishing, stealing data, coding a Chrome infostealer, and bypassing Google’s account verification methods.
Rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume. For skilled actors, generative AI tools provide a helpful framework, similar to the use of Metasploit or Cobalt Strike in cyber threat activity. For less skilled actors, they also provide a learning and productivity tool, enabling them to more quickly develop tools and incorporate existing techniques. However, current LLMs on their own are unlikely to enable breakthrough capabilities for threat actors.We note that the AI landscape is in constant flux, with new AI models and agentic systems emerging daily. As this evolution unfolds, GTIG anticipates the threat landscape to evolve in stride as threat actors adopt new AI technologies in their operations.
AI-Focused Threats
Attackers can use LLMs in two ways. One way is attempting to leverage LLMs to accelerate their campaigns (e.g., by generating code for malware or content for phishing emails). The overwhelming majority of activity we observed falls into this category. The second way attackers can use LLMs is to instruct a model or AI agent to take a malicious action (e.g., finding sensitive user data and exfiltrating it). These risks are outlined in Google’s Secure AI Framework (SAIF) risk taxonomy.
We did not observe any original or persistent attempts by threat actors to use prompt attacks or other AI-specific threats. Rather than engineering tailored prompts, threat actors used more basic measures, such as rephrasing a prompt or sending the same prompt multiple times. These attempts were unsuccessful.
Jailbreak Attempts: Basic and Based on Publicly Available Prompts
We observed a handful of cases of low-effort experimentation using publicly available jailbreak prompts in unsuccessful attempts to bypass Gemini’s safety controls. Threat actors copied and pasted publicly available prompts and appended small variations in the final instruction (e.g., basic instructions to create ransomware or malware). Gemini responded with safety fallback responses and declined to follow the threat actor’s instructions.
In one example of a failed jailbreak attempt, an APT actor copied publicly available prompts into Gemini and appended basic instructions to perform coding tasks. These tasks included encoding text from a file and writing it to an executable and writing Python code for a distributed denial-of-service (DDoS) tool. In the former case, Gemini provided Python code to convert Base64 to hex, but provided a safety filtered response when the user entered a follow-up prompt that requested the same code as a VBScript.
The same group used a different publicly available jailbreak prompt to request Python code for DDoS. Gemini provided a safety filtered response stating that it could not assist, and the threat actor abandoned the session and did not attempt further interaction.
What is an AI jailbreak?
Jailbreaks are one type of Prompt Injection attack, causing an AI model to behave in ways that they’ve been trained to avoid (e.g., outputting unsafe content or leaking sensitive information). Prompt Injections generally cause the LLM to execute malicious “injected” instructions as part of data that were not meant to be executed by the LLM.
Controls against prompt injection include input/output validation and sanitization as well as adversarial training and testing. Training, tuning, and evaluation processes also help fortify models against prompt injection.
Example of a jailbreak prompt publicly available on GitHub
Some malicious actors unsuccessfully attempted to prompt Gemini for guidance on abusing Google products, such as advanced phishing techniques for Gmail, assistance coding a Chrome infostealer, and methods to bypass Google’s account creation verification methods. These attempts were unsuccessful. Gemini did not produce malware or other content that could plausibly be used in a successful malicious campaign. Instead, the responses consisted of safety-guided content and generally helpful, neutral advice about coding and cybersecurity. In our continuous work to protect Google and our users, we have not seen threat actors either expand their capabilities or better succeed in their efforts to bypass Google’s defenses.
Government-backed attackers attempted to use Gemini for coding and scripting tasks, gathering information about potential targets, researching publicly known vulnerabilities, and enabling post-compromise activities, such as defense evasion in a target environment.
Iran: Iranian APT actors were the heaviest users of Gemini, using it for a wide range of purposes, including research on defense organizations, vulnerability research, and creating content for campaigns. APT42 focused on crafting phishing campaigns, conducting reconnaissance on defense experts and organizations, and generating content with cybersecurity themes.
China: Chinese APT actors used Gemini to conduct reconnaissance, for scripting and development, to troubleshoot code, and to research how to obtain deeper access to target networks. They focused on topics such as lateral movement, privilege escalation, data exfiltration, and detection evasion.
North Korea: North Korean APT actors used Gemini to support several phases of the attack lifecycle, including researching potential infrastructure and free hosting providers, reconnaissance on target organizations, payload development, and assistance with malicious scripting and evasion techniques. They also used Gemini to research topics of strategic interest to the North Korean government, such as the South Korean military and cryptocurrency. Of note, North Korean actors also used Gemini to draft cover letters and research jobs—activities that would likely support North Korea’s efforts to place clandestine IT workers at Western companies.
Russia: With Russian APT actors, we observed limited use of Gemini during the period of analysis. Their Gemini use focused on coding tasks, including converting publicly available malware into another coding language and adding encryption functions to existing code.
Google analyzed Gemini activity associated with known APT actors and identified APT groups from more than 20 countries that used Gemini. The highest volume of usage was from Iran and China. APT actors used Gemini to support several phases of the attack lifecycle, including researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques. The top use cases by APT actors focused on:
Assistance with coding tasks, including troubleshooting, tool and script development and converting or rewriting existing code
Vulnerability research focused on publicly reported vulnerabilities and specific CVEs
General research on various technologies, translations and technical explanations
Reconnaissance about likely targets, including details about specific organizations
Enabling post-compromise activity, such seeking advice on techniques to evade detection, escalate privileges or conduct internal reconnaissance in a target environment
We observed APT actors use Gemini attempting to support all phases of the attack lifecycle.
Attack Lifecycle
Topics of Gemini Usage
Reconnaissance
Attacker gathers information about the target
Iran
Recon on experts, international defense organizations, government organizations
Topics related to Iran-Israel proxy conflict
North Korea
Research on companies across multiple sectors and geos
Recon on US military and operations in South Korea
Research free hosting providers
China
Research on US military, US-based IT service providers
Understand public database of US intelligence personnel
Research on target network ranges; determine domain names of targets
Weaponization
Attacker develops or acquires tools to exploit target
Develop webcam recording code in C++
Convert Chrome infostealer function from Python to Node.js
Rewrite publicly available malware into another language
Add AES encryption functionality to provided code
Delivery
Attacker delivers weaponized exploit or payload to the target system
Better understanding advanced phishing techniques
Generating content for targeting a US defense organization
Generating content with cybersecurity and AI themes
Exploitation
Attacker exploits vulnerability to gain access
Reverse engineer endpoint detection and response (EDR) server components for health check and authentication
Access Microsoft Exchange using password hash
Research vulnerabilities in WinRM protocol
Understand publicly reported vulnerabilities, including Internet of Things (IoT) bugs
Installation
Attacker installs tools or malware to maintain access
Sign an Outlook Visual Studio Tools for Office (VSTO) plug-in and deploy it silently to all computers
Add a self-signed certificate to Active Directory
Research Mimikatz for Windows 11
Research Chrome extensions that provide parental controls and monitoring
Command and control (C2)
Attacker establishes communication channel with the compromised system
Generate code to remotely access Windows Event Log
Active Directory management commands
JSON Web Token (JWT) security and routing rules in Ruby on Rails
Character encoding issues in smbclient
Command to check IPs of admins on the domain controller
Actions on objectives
Attacker achieves their intended goal such as data theft or disruption
Automate workflows with Selenium (e.g. logging into compromised account)
Generate a PHP script to extract emails from Gmail into electronic mail (EML) files
Upload large files to OneDrive
Solution to TLS 1.3 visibility challenges
Iranian Government-Backed Actors
Iranian government-backed actors accounted for the largest Gemini use linked to APT actors. Across Iranian government-backed actors, we observed a broad scope of research and use cases, including to enable reconnaissance on targets, for research into publicly reported vulnerabilities, to request translation and technical explanations, and to create content for possible use in future campaigns. Their use reflected strategic Iranian interests including research focused on defense organizations and experts, defense systems, foreign governments, individual dissidents, the Israel-Hamas conflict, and social issues in Iran.
At a Glance: Iranian APT Actors Using Gemini
Over 10 Iran-backed groups observed using Gemini
Google abuse-focused use cases:
Researching methods for extracting data from Android devices, including SMS messages, accounts, contacts, and social media accounts
Example use cases:
Coding and scripting
PowerShell and Linux commands
Python code for website scraping
Debugging and improving a Ghidra script
Developing PHP scripts to collect and store user IP addresses and browser information in a MySQL database
Assistance with C# programming
Modifying assembly code
Help understanding error messages
Vulnerability research
Research on specific CVEs and technologies, such as WinRM and IoT devices
Exploitation techniques and proof-of-concept code
Research on server-side request forgery (SSRF) exploitation techniques
Research on the open-source router exploitation tool RomBuster
Research about organizations
International defense organizations
Military and government organizations
Cybersecurity companies
International organizations that monitor development of advanced weapons
Research about warfare defenses
Information on the Iran-Israel proxy conflict
Unmanned aerial vehicles (UAV)
Anti-drone systems
Satellite technology
Remote sensing technology
Israel defense systems
Generating content
Generating content with cybersecurity and AI themes
Tailoring content to target a defense organization
Translating various texts into Farsi, Hebrew, and English
Crafting Phishing Campaigns
Over 30% of Iranian APT actors’ Gemini use was linked to APT42. APT42’s Gemini activity reflected the group’s focus on crafting successful phishing campaigns. We observed the group using Gemini to conduct reconnaissance into individual policy and defense experts, as well as organizations of interest for the group.
In addition to reconnaissance, APT42 used the text generation and editing capabilities of Gemini to craft material for phishing campaigns, including generating content with cybersecurity themes and tailoring the output to a US defense organization. APT42 also utilized Gemini for translation including localization, or tailoring content for a local audience. This includes content tailored to local culture and local language, such as asking for translations to be in fluent English.
Vulnerability Research
The majority of APT42’s efforts focused on research into publicly known vulnerabilities, such as a request to generate a list of critical vulnerabilities from 2023. They also focused on vulnerabilities in specific products such as Mikrotik, Apereo, and Atlassian.
Of note, APT42 appeared to be researching how to use generative AI tools for offensive purposes, asking Gemini for help preparing training content for a red team focused on how offensive teams can use AI tools in their operations.
Research Into Military and Weapons Systems
APT42 also appears to have used Gemini’s translation and explanation functions to better understand publicly available information on defense systems. Their efforts included general research into the Israel-Hamas conflict, as well as strategic trends in China’s defense industry. The threat actor also used Gemini for technical explanations about US-made aerospace systems.
Another Iranian APT group also focused on understanding warfare defenses including specific research into satellite signal jamming and anti-drone systems. Other Iranian APT actors researched specific defense systems, including researching information about specific unmanned aerial vehicle (UAV) models, jamming F-35 fighter jets, anti-drone systems, and Israel’s missile defense systems.
People’s Republic of China (PRC) Government-Backed Actors
Government-backed actors linked to the People’s Republic of China (PRC) attempted to use Gemini to enable reconnaissance on targets, for scripting and development, to request translation and explanation of technical concepts, and attempting to enable deeper access to a network following initial compromise.PRC threat actors’ usage resembled an IT admin seeking to streamline, troubleshoot, or automate their tasks. In a malicious context, however, this activity could be used to enable lateral movement, privilege escalation, data exfiltration, and detection evasion.
At a Glance: People’s Republic of China APT Actors Using Gemini
Over 20 China-backed groups observed using Gemini
Notable use cases:
Reconnaissance
Research US military and US-based IT organizations
Gather US government network ranges
Understand publicly available information about US intelligence community personnel
Determine domain names of targets spanning eight countries, mostly government agencies
Access Microsoft Exchange using password hash
Vulnerability research
Reverse engineer Carbon Black EDR’s server components for health check and authentication
Scripting and development
Generate code to remotely access Windows Event Log
Active Directory management commands
Translation and explanation
Understand graph databases (Nebula Graph)
Solutions to TLS 1.3 visibility challenges
Understand a malicious PHP script
Web JWT security and routing rules in Ruby on Rails
Deeper system access and post-compromise actions
Sign an Outlook VSTO plug-in and deploy it silently to all computers
Add a self-signed certificate to Active Directory
Upload large files to OneDrive
Character encoding issues in smbclient
Command to check IPs of admins on the Domain Controller
Record passwords on the VMware vCenter
Impacket troubleshooting
Enabling Deeper Access in a Target Network
PRC-backed APT actors also used Gemini to work through scripting and development tasks, many of which appeared intended to enable deeper access in a target network after threat actors obtained initial access. For example, one PRC-backed group asked Gemini for assistance figuring out how to sign a plugin for Microsoft Outlook and silently deploy it to all computers. The same actor also asked Gemini to generate code to remotely access Windows Event Log; sought instructions on how to add a self-signed certificate to Active Directory; and asked Gemini for a command to identify the IP addresses of administrators on the domain controller. Other actors used Gemini for help troubleshooting Chinese character encoding issues in smbclient and how to record passwords on the VMware vCenter.
In another example, PRC-backed APT actors asked Gemini for assistance with Active Directory management commands and requested help troubleshooting impacket, a Python-based tool for working with network protocols. While impacket is commonly used for benign purposes, the context of the threat actor made it clear that the actor was using the tool for malicious purposes.
Explaining Tools, Concepts, and Code
PRC actors utilized Gemini to learn about specific tools and technologies and develop solutions to technical challenges. For example, a PRC APT actor used Gemini to break down how to use the graph database Nebula Graph. In another instance, the same actor used Gemini to offer possible solutions to TLS 1.3 visibility challenges. Another PRC-backed APT group sought to understand a malicious PHP script.
Vulnerability Research and Reverse Engineering
In one case, a PRC-backed APT actor attempted unsuccessfully to get Gemini’s help reverse engineering the endpoint detection and response (EDR) tool Carbon Black. The same threat actor copied disassembled Python bytecode into Gemini to convert the bytecode into Python code. It’s not clear what their objective was.
Unsuccessful Attempts to Elicit Internal System Information From Gemini
In one case, the PRC-backed APT actor APT41 attempted unsuccessfully to use Gemini to learn about Gemini’s underlying infrastructure and systems. The actor asked Gemini to share details such as its IP address, kernel version, and network configuration. Gemini responded but did not disclose sensitive information. In a helpful tone, the responses provided publicly available details that would be widely known about the topic, while also indicating that the requested information is kept secret to prevent unauthorized access.
North Korean Government-Backed Actors
North Korean APT actors used Gemini to support several phases of the attack lifecycle, including researching potential infrastructure and free hosting providers, reconnaissance on target organizations, payload development, and assistance with malicious scripting and evasion techniques. They also used Gemini to research topics of strategic interest to the North Korean government, such as South Korean nuclear technology and cryptocurrency. We also observed that North Korean actors were using LLMs in likely attempts to enable North Korea’s efforts to place clandestine IT workers at Western companies.
At a Glance: North Korean APT Actors Using Gemini
Nine North Korea-backed groups observed using Gemini
Google-focused use cases:
Research advanced techniques for phishing Gmail
Scripting to steal data from compromised Gmail accounts
Understanding a Chrome extension that provides parental controls (capable of taking screenshots, keylogging)
Convert Chrome infostealer function from Python to Node.js
Bypassing restrictions on Google Voice
Generate code snippets for a Chrome extension
Notable use cases:
Enabling clandestine IT worker scheme
Best Discord servers for freelancers
Exchange with overseas employees
Jobs on LinkedIn
Average salary
Drafting work proposals
Generate cover letters from job postings
Research on topics
Free hosting providers
Cryptocurrency
Operational technology (OT) and industrial networks
Nuclear technology and power plants in South Korea
Historic cyber events (e.g., major worms and DDoS attacks; Russia-Ukraine conflict) and cyber forces of foreign militaries
Research about organizations
Companies across 11 sectors and 13 countries
South Korean military
US military
German defense organizations
Malware development
Evasion techniques
Automating workflows for logging into compromised accounts
Understanding Mimikatz for Windows 11
Scripting and troubleshooting
Clandestine IT Worker Threat
North Korean APT actors used Gemini to draft cover letters and research jobs—activities that would likely support efforts by North Korean nationals to use fake identities and obtain freelance and full-time jobs at foreign companies while concealing their true identities and locations. One North Korea-backed group utilized Gemini to draft cover letters and proposals for job descriptions, researched average salaries for specific jobs, and asked about jobs on LinkedIn. The group also used Gemini for information about overseas employee exchanges. Many of the topics would be common for anyone researching and applying for jobs.
While normally employment-related research would be typical for any job seeker, we assess the usage is likely related to North Korea’s ongoing efforts to place clandestine workers in freelance gigs or full-time jobs at Western firms. The scheme, which involves thousands of North Korean workers and has affected hundreds of US-based companies, uses IT workers with false identities to complete freelance work and send wages back to the North Korean regime.
North Korea’s AI toolkit
Outside of their use of Gemini, North Korean cyber threat actors have shown a long-standing interest in AI tools. They likely use AI applications to augment malicious operations and improve efficiency and capabilities, and for producing content to support their campaigns, such as phishing lures and profile photos for fake personas. We assess with high confidence that North Korean cyber threat actors will continue to demonstrate an interest in these emerging technologies for the foreseeable future.
DPRK IT Workers
We have observed DPRK IT Workers leverage accounts on assistive writing tools, Monica (monica.im) and Ahrefs (ahrefs.com), which could potentially aid the group’s work despite a lack of language fluency. Additionally, the group has maintained accounts on Data Annotation Tech, a company hiring individuals to train AI models. Notably, a profile photo used by a suspected IT worker bore a noticeable resemblance to multiple different images on the internet, suggesting that a manipulation tool was used to generate the threat actor’s profile photo.
APT43
Google Threat Intelligence Group (GTIG) has detected evidence of APT43 actors accessing multiple publicly available LLM tools; however, the intended purpose is not clear. Based on the capabilities of these platforms and historical APT43 activities, it is possible these applications could be used in the creation of rapport-building emails, lure content, and malicious PowerShell and scripting efforts.
GTIG has detected APT43 actors reference publicly available AI chatbot tools alongside the topic “북핵 해결” (translation: “North Korean nuclear issue solution”), indicating the group is using AI applications to conduct technical research as well as open-source analysis on South Korean foreign and military affairs and nuclear issues.
GTIG has identified APT43 actors accessing multiple publicly available AI image generation tools, including tools used for image manipulation and creating realistic-looking human portraits.
Target Research and Reconnaissance
North Korean actors also engaged with Gemini with several questions that appeared focused on conducting initial research and reconnaissance into prospective targets. They also researched organizations and industries that are typical targets for North Korean actors, including the US and South Korean militaries and defense contractors. One North Korean APT group asked Gemini for information about companies and organizations across a variety of industry sectors and regions. Some of this Gemini usage related directly to organizations that the same group had attempted to target in phishing and malware campaigns that Google previously detected and disrupted.
In addition to research into companies, North Korean APT actors researched nuclear technology and power plants in South Korea, such as site locations, recent news articles, and the security status of the plants. Gemini responded with widely available, public information and facts that would be easily discoverable in an online search.
Help with Scripting, Payload Development, Defense Evasion
North Korean actors also tried to use Gemini to assist with development and scripting tasks. One North Korea-backed group attempted to use Gemini to help develop webcam recording code in C++. Gemini provided multiple versions of code, and repeated efforts by the actor potentially suggested their frustration by Gemini’s answers. The same group also asked Gemini to generate a robots.txt file to block crawlers and an .htaccess file to redirect all URLs except CSS extensions.
One North Korean APT actor used Gemini for assistance developing code for sandbox evasion. For example, the threat actor utilized Gemini to write code in C++ to detect VM environments and Hyper-V virtual machines. Gemini provided responses with short code snippets to perform simple sandbox checks. The same group also sought help troubleshooting Java errors when implementing AES encryption, and separately asked Gemini if it is possible to acquire a system password on Windows 11 using Mimikatz.
Russian Government-Backed Actors
During the period of analysis, we observed limited use of Gemini by Russia-backed APT actors. Of this limited use, the majority of usage appeared benign, rather than threat-enabling. The reasons for this low engagement are unclear. It is possible Russian actors avoided Gemini out of operational security considerations, staying off Western-controlled platforms to avoid monitoring of their activities. They may be using AI tools produced by Russian firms or locally hosting LLMs, which would ensure full control of their infrastructure. Alternatively, they may have favored other Western LLMs.
One Russian government-backed group used Gemini to request help with a handful of tasks, including help rewriting publicly available malware into another language, adding encryption functionality to code, and explanations for how a specific block of publicly available malicious code functions.
At a Glance: Russian APT Actors Using Gemini
Three Russia-backed groups observed using Gemini
Notable use cases:
Scripting
Help rewriting public malware into another language
Payload crafting
Add AES encryption functionality to provided code
Translation and explanation
Understand how some public malicious code works
Financially Motivated Actors Using LLMs
Threat actors in underground marketplaces are advertising ways to bypass security guardrails to help LLMs with malware development, phishing, and other malicious tasks. The offerings include jailbroken LLMs that are ready-made for malicious use.
Throughout 2023 and 2024, Google Threat Intelligence Group (GTIG) observed underground forum posts related to LLMs, indicating there is a burgeoning market for nefarious versions of LLMs. Some advertisements boast customized and jailbroken LLMs that don’t have restrictions for malware development purposes, or they tout a lack of security measures typically found on legitimate services, allowing the user to prompt the LLM about any topic or task without incurring security guardrails or limits on their queries. Examplesinclude FraudGPT, which has been advertised on Telegram as having no limitations, and WormGPT, a privacy focused, “uncensored” LLM capable of developing malware.
Financially motivated actors are using LLMs to help augment business email compromise (BEC) operations. GTIG has noted evidence of financially motivated actors using manipulated video and voice content in business email compromise (BEC) scams. Media reports indicate that financially motivated actors have reportedly used WormGPT to create more persuasive BEC messages.
Findings: Information Operations (IO) Actors Misusing Gemini
At a Glance: Information Operations Actors
IO actors attempted to use Gemini for research, content generation, translation and localization, and to find ways to increase their reach.
Iran: Iranian IO actors used Gemini for a wide range of tasks, accounting for three quarters of all IO prompts. They used Gemini for content creation and manipulation, including generating articles, rewriting text with a specific tone, and optimizing it for better reach. Their activity also focused on translation and localization, adapting content for different audiences, and for general research into news, current events, and political issues.
China: Pro-China IO actors used Gemini primarily for general research on various topics, including a variety of topics of strategic interest to the Chinese government. The most prolific IO actor we track, DRAGONBRIDGE, was responsible for the majority of this activity. They also used Gemini to research current events and politics, and in a few cases, they used Gemini to generate articles or content on specific topics.
Russia: Russian IO actors used Gemini primarily for general research, content creation, and translation. For example, their use involved assistance drafting content, rewriting article titles, and planning social media campaigns. Some activity demonstrated an interest in developing AI capabilities, asking for information on tools for creating online AI chatbots, developer tools for interacting with LLMs, and options for textual content analysis.
IO actors used Gemini for research, content generation including developing personas and messaging, translation and localization, and to find ways to increase their reach.Common use cases include general research into news and current events as well as specific research into individuals and organizations. In addition to creating content for campaigns, including personas and content, the actors researched increasing the efficacy of campaigns, including automating distribution, using search engine optimization (SEO) to optimize the reach of campaigns, and increasing operational security. As with government-backed groups, IO actors also used Gemini for translation and localization and for understanding the meanings or context of content.
Iran-Linked Information Operations Actors
Iran-based information operations (IO) groups used Gemini for a wide range of tasks, including general research, translation and localization, content creation and manipulation, and generating content with a specific bias or tone. We also observed Iran-based IO actors engage with Gemini about news events and ask Gemini to provide details on economic and political issues in Iran, the US, the Middle East, and Europe.
In line with their practice of mixing original and borrowed content, Iranian IO actors translated existing material, including news-like articles. They then used Gemini to explain the context and meaning of particular phrases within the given text.
Iran-based IO actors also used Gemini to localize the content, seeking human-like translation and asking Gemini for help with tasks like making the text sound like a native English speaker. They used Gemini to manipulate text (e.g., asking for help rewriting existing text on immigration and crime in a specific style or tone).
Iran’s activity also included research about improving the reach of their campaigns. For example, they attempted to generate SEO-optimized content, likely in an effort to reach a larger audience. Some actors also used Gemini to suggest strategies for increasing engagement on social media.
At a Glance: Iran-Linked IO Actors Using Gemini
Eight Iran-linked IO groups observed using Gemini
Example use cases:
Content creation – text
Generate article titles
Generate SEO-optimized content and titles
Draft a report critical of Bahrain
Draft titles and hashtags in English and Farsi for videos that are catchy or create urgency to watch the content
Draft titles and descriptions promoting Islam
Translation – content in / out of native language
Translate into Farsi-provided texts about a variety of topics, including the Iranian election, human rights, international law, Islam, and other topics
Translate Farsi-language idioms and proverbs to other languages
Translate news about the US economy, US government, and politics into Farsi, using a specified tone
Draft a French-language headline to get viewers to engage with specific content
Content manipulation – copy editing to refine content
Reformulate specific text about Sharia law
Paraphrase content describing specific improvements to Iran’s export economy
Rewrite a provided text about diplomacy and economic challenges with countries like China and Germany
Provide synonyms for specific words or phrases
Rewrite provided text about Islam and Iraq in different styles or tones
Proofread provided content
Content creation – biased text
Generate or reformulate text to criticize a government minister and other individuals for failures or other actions
Describe how a popular American TV show perpetuates harmful stereotypes
Generate Islam-themed titles for thumbnail previews on social media
General research – news and events
Provide an overview of current events in specific regions
Research about the Iran-Iraq war
Define specific terms
Suggest social media channels for information about Islam and the Quran
Provide information on countries’ policies toward the Middle East
Create persona – photo generation
Create a logo
PRC-Linked Information Operations Actors
IO actors linked to the People’s Republic of China (PRC) used Gemini primarily for general research on a wide variety of topics. The most prolific IO actor we track, the pro-China group DRAGONBRIDGE, was responsible for approximately three quarters of this activity. Of their activity, the majority use was general research about a wide variety of topics, ranging from details about the features of various social media platforms to questions about various topics of strategic interest to the PRC government. Actors researched information on current events and politics in other regions, with a focus on the US and Taiwan. They also showed interest in assessing the impact and risk of certain events. In a handful of cases, DRAGONBRIDGE used Gemini to generate articles or content on specific topics.
At a Glance: PRC-Linked IO Actors Using Gemini
Three PRC-linked IO groups observed using Gemini
Example use cases:
General research – political and social topics
Research about specific countries, organizations, and individuals
Research relations between specific countries and China
Research on topics sensitive to the the Chinese government (e.g., five poisons)
Research on Taiwanese politicians and their actions toward China
Research on US politics and political figures and their attitudes on China
Research foreign press coverage about China
Summarize key takeaways from a video
General research – technology
Compare functionality and features of different social media platforms
Explain technical concepts and suggestions for useful tools
Translation – content in / out of native language
Translate and summarize text between Chinese and other languages
Content creation – text
Draft articles on topics such as the use of AI and social movements in specific regions
Generate a summary of a movie trailer about a Chinese dissident
Create persona – text generation
Generate a company profile for a media company
DRAGONBRIDGE has experimented with other generative AI tools to create synthetic content in support of their IO campaigns. As early as 2022, the group used a commercial AI service in videos on YouTube to depict AI-generated news presenters. Their use of AI-generated video continued through 2024 but has not resulted in significantly higher engagement from real viewers. Google detected and terminated the channels distributing this content immediately upon discovery. DRAGONBRIDGE’s use of AI-generated videos or images has not resulted in significantly higher engagement from real viewers.
Russia-Linked Information Operations Actors
Russian IO actors used Gemini for general research, content creation, and translation. Half of this activity was associated with the Russian IO actor we track as KRYMSKYBRIDGE, which is linked to a Russian consulting firm that works with the Russian government. Approximately 40% of activity was linked to actors associated with Russian state sponsored entities formerly controlled by the late Russian oligarch Yevgeny Prigozhin. We also observed usage by actors tracked publicly as Doppelganger.
The majority of Russian IO actor usage was related to general research tasks, ranging from the Russia-Ukraine war to details about various tools and online services. Russian IO actors also used Gemini for content creation, rewriting article titles and planning social media campaigns. Translation to and from Russian was also a common task.
Russian IO actors focused on the generative AI landscape, which may indicate an interest in developing native capabilities in AI on infrastructure they control. They researched tools that can be used to create an online AI chatbot and developer tools for interacting with LLMs. One Russian IO actor used Gemini to suggest options for textual content analysis.
Pro-Russia IO actors have used AI in their influence campaigns in the past. In 2024, the actor known as CopyCop likely used LLMs to generate content, and some stories on their sites included metadata indicating an LLM was prompted to rewrite articles from genuine news sources with a particular political perspective or tone. CopyCop’s inauthentic news sites pose as US- and Europe-based news outlets and post Kremlin-aligned views on Western policy, the war in Ukraine, and domestic politics in the US and Europe.
At a Glance: Russia-Linked IO Actors Using Gemini
Four Russia-linked IO groups observed using Gemini
Example use cases:
General research
Research into the Russia-Ukraine war
Explain subscription plans and API details for online services
Research on different generative AI platforms, software, and systems for interacting with LLMs
Research on tools and methods for creating an online chatbot
Research tools for content analysis
Translation – content in / out of native language
Translate technical and business terminology into Russian
Translate text to/from Russian
Content creation – text
Draft a proposal for a social media agency
Rewrite article titles to garner more attention
Plan and strategize campaigns
Develop content strategy for different social media platforms and regions
Brainstorm ideas for a PR campaign and accompanying visual designs
Building AI Safely and Responsibly
We believe our approach to AI must be both bold and responsible. To us, that means developing AI in a way that maximizes the positive benefits to society while addressing the challenges. Guided by ourAI Principles, Google designs AI systems with robust security measures and strong safety guardrails, and we continuously test the security and safety of our models to improve them. Our policy guidelines and prohibited use policies prioritize safety and responsible use of Google’s generative AI tools. Google’s policy development process includes identifying emerging trends, thinking end-to-end, and designing for safety. We continuously enhance safeguards in our products to offer scaled protections to users across the globe.
At Google, we leverage threat intelligence to disrupt adversary operations. We investigate abuse of our products, services, users and platforms, including malicious cyber activities by government-backed threat actors, and work with law enforcement when appropriate. Moreover, our learnings from countering malicious activities are fed back into our product development to improve safety and security for our AI models. Google DeepMind also develops threat models for generative AI to identify potential vulnerabilities, and creates new evaluation and training techniques to address misuse caused by them. In conjunction with this research, DeepMind has shared how they’re actively deploying defenses within AI systems along with measurement and monitoring tools, one of which is a robust evaluation framework used to automatically red team an AI system’s vulnerability to indirect prompt injection attacks. Our AI development and Trust & Safety teams also work closely with our threat intelligence, security, and modelling teams to stem misuse.
The potential of AI, especially generative AI, is immense. As innovation moves forward, the industry needs security standards for building and deploying AI responsibly. That’s why we introduced the Secure AI Framework (SAIF), a conceptual framework to secure AI systems. We’ve shared a comprehensive toolkit for developers with resources and guidance for designing, building, and evaluating AI models responsibly. We’ve also shared best practices for implementing safeguards, evaluating model safety, and red teaming to test and secure AI systems.
About the Authors
Google Threat Intelligence Group brings together the Mandiant Intelligence and Threat Analysis Group (TAG) teams, and focuses on identifying, analyzing, mitigating, and eliminating entire classes of cyber threats against Alphabet, our users, and our customers. Our work includes countering threats from government-backed attackers, targeted 0-day exploits, coordinated information operations (IO), and serious cyber crime networks. We apply our intelligence to improve Google’s defenses and protect our users and customers.
Editor’s note: Today’s post is by Travis Naraine, IT Infrastructure Engineer, and Harel Shaked, Director of IT Services and Support, both for Outbrain, a leading technology platform that drives business results by engaging people across the open internet. Outbrain adopted Chrome Enterprise and integrations from Spin.AI to create policies for secure app and extension use and manage automatic updates for its dispersed workforce.
With a workforce as dispersed as ours, security is always a challenge. We standardized on Chrome Enterprise browser two years ago, and it’s become the linchpin of our cloud-first strategy, giving us a way to manage all of our users and stay secure. But we had concerns about browser extensions and we felt it was time to find a solution.
The value of extension management
We know people like to use browser extensions to improve their productivity and to access the tools and features they need to do their jobs. We also know there are malicious extensions available online. But vetting, testing, and blocking extensions manually was time-consuming and not 100% effective because it didn’t give us visibility into which extensions and apps were already in our environment.
Our process was reactive instead of proactive, raising concerns over missed opportunities to detect and block risky extensions. We needed a more automated way to enable employees to safely install Chrome Enterprise extensions.
Tools for extension risk assessment
As we explored solutions for another security project, we came across Spin.AI’s SpinOne platform, which includes the SaaS Security Posture Management (SSPM) solution for third-party application security. SSPM had several points in its favor, including features for continuous app assessment for browser extensions and the ability to easily integrate with Chrome Enterprise. The SpinOne platform met several of our SaaS security needs, and we like to stay with one vendor whenever possible.
Now we use Chrome Enterprise extension risk assessment, powered by Spin.AI, to generate risk scores and comprehensive risk assessment reports that assist in decisions about allowing or blocking extensions. In addition, with Chrome Enterprise Core‘s extension workflow, Outbrain employees can easily submit extension requests for IT and security teams to review and allow or deny use of the extensions.
The automated process through Chrome Enterprise saves significant time compared with manual reviews. The new policies and the Chrome Enterprise and Spin.AI solution has created an environment that nudges users to think more about anything they were installing—extensions, and other apps as well.
Using extensions securely and safely
Chrome Enterprise makes management and control easy, enforcing policies for the browser and extensions with less complexity. We even develop our own in-house extensions for Chrome Enterprise for tasks like inspecting widgets within the company.
In addition to setting browser policies through the Google Admin console, we can manage automatic updates to ensure our employees are using the newest version of Chrome with the latest security patches, further reducing our exposure to vulnerabilities.
We definitely have fewer worries about browser security today. We know that Spin.AI and Chrome Enterprise are doing their job in the background, so we’re not constantly concerned that a user is installing something malicious. We can set it and forget it.
Since 2022, Google Threat Intelligence Group (GTIG) has been tracking multiple cyber espionage operations conducted by China-nexus actors utilizing POISONPLUG.SHADOW. These operations employ a custom obfuscating compiler that we refer to as “ScatterBrain,” facilitating attacks against various entities across Europe and the Asia Pacific (APAC) region. ScatterBrain appears to be a substantial evolution of ScatterBee, an obfuscating compiler previously analyzed by PWC.
GTIG assesses that POISONPLUG is an advanced modular backdoor used by multiple distinct, but likely related threat groups based in the PRC, however we assess that POISONPLUG.SHADOW usage appears to be further restricted to clusters associated with APT41.
GTIG currently tracks three known POISONPLUG variants:
POISONPLUG
POISONPLUG.DEED
POISONPLUG.SHADOW
POISONPLUG.SHADOW—often referred to as “Shadowpad,” a malware family name first introduced by Kaspersky—stands out due to its use of a custom obfuscating compiler specifically designed to evade detection and analysis. Its complexity is compounded by not only the extensive obfuscation mechanisms employed but also by the attackers’ highly sophisticated threat tactics. These elements collectively make analysis exceptionally challenging and complicate efforts to identify, understand, and mitigate the associated threats it poses.
In addressing these challenges, GTIG collaborates closely with the FLARE team to dissect and analyze POISONPLUG.SHADOW. This partnership utilizes state-of-the-art reverse engineering techniques and comprehensive threat intelligence capabilities required to mitigate the sophisticated threats posed by this threat actor. We remain dedicated to advancing methodologies and fostering innovation to adapt to and counteract the ever-evolving tactics of threat actors, ensuring the security of Google and our customers against sophisticated cyber espionage operations.
Overview
In this blog post, we present our in-depth analysis of the ScatterBrain obfuscator, which has led to the development of a complete stand-alone static deobfuscator library independent of any binary analysis frameworks. Our analysis is based solely on the obfuscated samples we have successfully recovered, as we do not possess the obfuscating compiler itself. Despite this limitation, we have been able to comprehensively infer every aspect of the obfuscator and the necessary requirements to break it. Our analysis further reveals that ScatterBrain is continuously evolving, with incremental changes identified over time, highlighting its ongoing development.
This publication begins by exploring the fundamental primitives of ScatterBrain, outlining all its components and the challenges they present for analysis. We then detail the steps required to subvert and remove each protection mechanism, culminating in our deobfuscator. Our library takes protected binaries generated by ScatterBrain as input and produces fully functional deobfuscated binaries as output.
By detailing the inner workings of ScatterBrain and sharing our deobfuscator, we hope to provide valuable insights into developing effective countermeasures. Our blog post is intentionally exhaustive, drawing from our experience in dealing with obfuscation for clients, where we observed a significant lack of clarity in understanding modern obfuscation techniques. Similarly, analysts often struggle with understanding even relatively simplistic obfuscation methods primarily because standard binary analysis tooling is not designed to account for them. Therefore, our goal is to alleviate this burden and help enhance the collective understanding against commonly seen protection mechanisms.
For general questions about obfuscating compilers, we refer to our previous work on the topic, which provides an introduction and overview.
ScatterBrain Obfuscator
Introduction
ScatterBrain is a sophisticated obfuscating compiler that integrates multiple operational modes and protection components to significantly complicate the analysis of the binaries it generates. Designed to render modern binary analysis frameworks and defender tools ineffective, ScatterBrain disrupts both static and dynamic analyses.
Protection Modes: ScatterBrain operates in three distinct modes, each determining the overall structure and intensity of the applied protections. These modes allow the compiler to adapt its obfuscation strategies based on the specific requirements of the attack.
Protection Components: The compiler employs key protection components that include the following:
Selective or Full Control Flow Graph (CFG) Obfuscation: This technique restructures the program’s control flow, making it very difficult to analyze and create detection rules for.
Instruction Mutations: ScatterBrain alters instructions to obscure their true functionality without changing the program’s behavior.
Complete Import Protection: ScatterBrain employs a complete protection of a binary’s import table, making it extremely difficult to understand how the binary interacts with the underlying operating system.
These protection mechanisms collectively make it extremely challenging for analysts to deconstruct and understand the functionality of the obfuscated binaries. As a result, ScatterBrain poses a formidable obstacle for cybersecurity professionals attempting to dissect and mitigate the threats it generates.
Modes of Operation
A mode refers to how ScatterBrain will transform a given binary into its obfuscated representation. It is distinct from the actual core obfuscation mechanisms themselves and is more about the overall strategy of applying protections. Our analysis further revealed a consistent pattern in applying various protection modes at specific stages of an attack chain:
Selective: A group of individually selected functions are protected, leaving the remainder of the binary in its original state. Any import references within the selected functions are also obfuscated. This mode was observed to be used strictly for dropper samples of an attack chain.
Complete: The entirety of the code section and all imports are protected. This mode was applied solely to the plugins embedded within the main backdoor payload.
Complete “headerless”: This is an extension of the Complete mode with added data protections and the removal of the PE header. This mode was exclusively reserved for the final backdoor payload.
Selective
The selective mode of protection allows users of the obfuscator to selectively target individual functions within the binary for protection. Protecting an individual function involves keeping the function at its original starting address (produced by the original compiler and linker) and substituting the first instruction with a jump to the obfuscated code. The generated obfuscations are stored linearly from this starting point up to a designated “end marker” that signifies the ending boundary of the applied protection. This entire range constitutes a protected function.
The disassembly of a call site to a protected function can take the following from:
Figure 1: Disassembly of a call to a protected function
The start of the protected function:
.text:180001039 PROTECTED_FUNCTION
.text:180001039 jmp loc_18000DF97 ; jmp into obfuscated code
.text:180001039 sub_180001039 endp
.text:000000018000103E db 48h ; H. ; garbage data
.text:000000018000103F db 0FFh
.text:0000000180001040 db 0C1h
Figure 2: Disassembly inside of a protected function
The “end marker” consists of two sets of padding instructions, an int3 instruction and a single multi-nop instruction:
END_MARKER:
.text:18001A95C CC CC CC CC CC CC CC CC CC CC 66
66 0F 1F 84 00 00 00 00 00
.text:18001A95C int 3
.text:18001A95D int 3
.text:18001A95E int 3
.text:18001A95F int 3
.text:18001A960 int 3
.text:18001A961 int 3
.text:18001A962 int 3
.text:18001A963 int 3
.text:18001A964 int 3
.text:18001A965 int 3
.text:18001A966 db 66h, 66h ; @NOTE: IDA doesn't disassemble properly
.text:18001A966 nop word ptr [rax+rax+00000000h]
; -------------------------------------------------------------------------
; next, original function
.text:18001A970 ; [0000001F BYTES: COLLAPSED FUNCTION
__security_check_cookie. PRESS CTRL-NUMPAD+ TO EXPAND]
Figure 3: Disassembly listing of an end marker
Complete
The complete mode protects every function within the .text section of the binary, with all protections integrated directly into a single code section. There are no end markers to signify protected regions; instead, every function is uniformly protected, ensuring comprehensive coverage without additional sectioning.
This mode forces the need for some kind of deobfuscation tooling. Whereas selective mode only protects the selected functions and leaves everything else in its original state, this mode makes the output binary extremely difficult to analyze without accounting for the obfuscation.
Complete Headerless
This complete mode extends the complete approach to add further data obfuscations alongside the code protections. It is the most comprehensive mode of protection and was observed to be exclusively limited to the final payloads of an attack chain. It incorporates the following properties:
Full PE header of the protected binary is removed.
Custom loading logic (a loader) is introduced.
Becomes the entry point of the protected binary
Responsible of ensuring the protected binary is functional
Includes the option of mapping the final payload within a separate memory region distinct from the initial memory region it was loaded in
Metadata is protected via hash-like integrity checks.
The metadata is utilized by the loader as part of its initialization sequence.
Import protection will require relocation adjustments.
Done through an “import fixup table”
The loader’s entry routine crudely merges with the original entry of the binary by inserting multiple jmpinstructions to bridge the two together. The following is what the entry point looks like after running our deobfuscator against a binary protected in headerless mode.
Figure 4: Deobfuscated loader entry
The loader’s metadata is stored in the .data section of the protected binary. It is found via a memory scan that applies bitwise XOR operations against predefined constants. The use of these not only locates the metadata but also serves a dual purpose of verifying its integrity. By checking that the data matches expected patterns when XORed with these constants, the loader ensures that the metadata has not been altered or tampered with.
Figure 5: Memory scan to identify the loader’s metadata inside the .data section
The metadata contains the following (in order):
Import fixup table(fully explained in the Import Protection section)
Integrity-hash constants
Relative virtual address (RVA) of the.datasection
Offset to the import fixup table from the start of the.datasection
Size, in bytes, of the fixup table
Global pointer to the memory address that the backdoor is at
Encrypted and compressed data specific to the backdoor
Backdoor config and plugins
Figure 6: Loader’s metadata
Core Protection Components
Instruction Dispatcher
The instruction dispatcher is the central protection component that transforms the natural control flow of a binary (or individual function) into scattered basic blocks that end with a unique dispatcher routine that dynamically guides the execution of the protected binary.
Figure 7: Illustration of the control flow instruction dispatchers induce
Each call to a dispatcher is immediately followed by a 32-bit encoded displacement positioned at what would normally be the return address for the call. The dispatcher decodes this displacement to calculate the destination target for the next group of instructions to execute. A protected binary can easily contain thousands or even tens of thousands of these dispatchers making manual analysis of them practically infeasible. Additionally, the dynamic dispatching and decoding logic employed by each dispatcher effectively disrupts CFG reconstruction methods used by all binary analysis frameworks.
The decoding logic is unique for each dispatcher and is carried out using a combination of add, sub, xor, and, or, and lea instructions. The decoded offset value is then either subtracted from or added to the expected return address of the dispatcher call to determine the final destination address. This calculated address directs execution to the next block of instructions, which will similarly end with a dispatcher that uniquely decodes and jumps to subsequent instruction blocks, continuing this process iteratively to control the program flow.
The following screenshot illustrates what a dispatcher instance looks like when constructed in IDA Pro. Notice the scattered addresses present even within instruction dispatchers, which result from the obfuscator transforming fallthrough instructions—instructions that naturally follow the preceding instruction—into pairs of conditional branches that use opposite conditions. This ensures that one branch is always taken, effectively creating an unconditional jump. Additionally, a mov instruction that functions as a no-op is inserted to split these branches, further obscuring the control flow.
Figure 8: Example of an instruction dispatcher and all of its components
The core logic for any dispatcher can be categorized into the following four phases:
Preservation of Execution Context
Each dispatcher selects a single working register (e.g., RSI as depicted in the screenshot) during the obfuscation process. This register is used in conjunction with the stack to carry out the intended decoding operations and dispatch.
The RFLAGS register in turn is safeguarded by employing pushfq and popfq instructions before carrying out the decoding sequence.
Retrieval of Encoded Displacement
Each dispatcher retrieves a 32-bit encoded displacement located at the return address of its corresponding call instruction. This encoded displacement serves as the basis for determining the next destination address.
Decoding Sequence
Each dispatcher employs a unique decoding sequence composed of the following arithmetic and logical instructions: xor, sub, add, mul, imul, div, idiv, and, or, and not. This variability ensures that no two dispatchers operate identically, significantly increasing the complexity of the control flow.
Termination and Dispatch
The ret instruction is strategically used to simultaneously signal the end of the dispatcher function and redirect the program’s control flow to the previously calculated destination address.
It is reasonable to infer that the obfuscator utilizes a template similar to the one illustrated in Figure 9 when applying its transformations to the original binary:
Figure 9: Instruction dispatcher template
Opaque Predicates
ScatterBrain uses a series of seemingly trivial opaque predicates (OP) that appear straightforward to analysts but significantly challenge contemporary binary analysis frameworks, especially when used collectively. These opaque predicates effectively disrupt static CFG recovery techniques not specifically designed to counter their logic. Additionally, they complicate symbolic execution approaches as well by inducing path explosions and hindering path prioritization. In the following sections, we will showcase a few examples produced by ScatterBrain.
test OP
This opaque predicate is constructed around the behavior of the testinstruction when paired with an immediate zero value. Given that the testinstruction effectively performs a bitwise AND operation, the obfuscator exploits the fact that any value bitwise AND-ed with zero always invariably results in zero.
Here are some abstracted examples we can find in a protected binary—abstracted in the sense that all instructions are not guaranteed to follow one another directly; other forms of mutations can be between them as can instruction dispatchers.
test bl, 0
jnp loc_56C96 ; we never satisfy these conditions
------------------------------
test r8, 0
jo near ptr loc_3CBC8
------------------------------
test r13, 0
jnp near ptr loc_1A834
------------------------------
test eax, 0
jnz near ptr loc_46806
Figure 10: Test opaque predicate examples
To grasp the implementation logic of this opaque predicate, the semantics of the testinstruction and its effects on the processor’s flags register are required. The instruction can affect six different flags in the following manner:
Overflow Flag (OF): Always cleared
Carry Flag (CF): Always cleared
Sign Flag (SF): Set if the most significant bit (MSB) of the result is set; otherwise cleared
Zero Flag (ZF): Set if the result is 0; otherwise cleared
Parity Flag (PF): Set if the number of set bits in the least significant byte (LSB) of the result is even; otherwise cleared
Auxiliary Carry Flag (AF): Undefined
Applying this understanding to the sequences produced by ScatterBrain, it is evident that the generated conditions can never be logically satisfied:
Sequence
Condition Description
test <reg>, 0; jo
OFis always cleared
test <reg>, 0; jnae/jc/jb
CFis always cleared
test <reg>, 0; js
Resulting value will always be zero; therefore, SFcan never be set
test <reg>, 0; jnp/jpo
The number of bits in zero is always zero, which is an even number; therefore, PFcan never be set
test <reg>, 0; jne/jnz
Resulting value will always be zero; therefore, ZFwill always be set
Table 1: Test opaque predicate understanding
jcc OP
The opaque predicate is designed to statically obscure the original immediate branch targets for conditional branch (jcc) instructions. Consider the following examples:
test eax, eax
ja loc_3BF9C
ja loc_2D154
test r13, r13
jns loc_3EA84
jns loc_53AD9
test eax, eax
jnz loc_99C5
jnz loc_121EC
cmp eax, FFFFFFFF
jz loc_273EE
jz loc_4C227
Figure 11: jcc opaque predicate examples
The implementation is straightforward: each original jccinstruction is duplicated with a bogus branch target. Since both jccinstructions are functionally identical except for their respective branch destinations, we can determine with certainty that the first jccin each pair is the original instruction. This original jccdictates the correct branch target to follow when the respective condition is met, while the duplicated jccserves to confuse analysis tools by introducing misleading branch paths.
Stack-Based OP
The stack-based opaque predicate is designed to check whether the current stack pointer (rsp) is below a predetermined immediate threshold—a condition that can never be true. It is consistently implemented by pairing the cmp rsp instruction with a jb (jump if below) condition immediately afterward.
cmp rsp, 0x8d6e
jb near ptr unk_180009FDA
Figure 12: Stack-based opaque predicate example
This technique inserts conditions that are always false, causing CFG algorithms to follow both branches and thereby disrupt their ability to accurately reconstruct the control flow.
Import Protection
The obfuscator implements a sophisticated import protection layer. This mechanism conceals the binary ‘s dependencies by transforming each original callor jmpinstruction directed at an import through a unique stub dispatcher routine that knows how to dynamically resolve and invoke the import in question.
Figure 13: Illustration of all the components involved in the import protection
It consists of the following components:
Import-specific encrypteddata: Each protected import is represented by a unique dispatcher stub and a scattered data structure that stores RVAs to both the encrypted dynamic-link library (DLL) and application programming interface (API) names. We refer to this structure as obf_imp_t. Each dispatcher stub is hardcoded with a reference to its respective obf_imp_t.
Dispatcher stub: This is an obfuscated stub that dynamically resolves and invokes the intended import. While every stub shares an identical template, each contains a unique hardcoded RVA that identifies and locates its corresponding obf_imp_t.
Resolver routine: Called from the dispatcher stub, this obfuscated routine resolves the import and returns it to the dispatcher, which facilitates the final call to the intended import. It begins by locating the encrypted DLL and API names based on the information in obf_imp_t. After decrypting these names, the routine uses them to resolve the memory address of the API.
Import decryption routine: Called from the resolver routine, this obfuscated routine is responsible for decrypting the DLL and API name blobs through a custom stream cipher implementation. It uses a hardcoded 32-bit salt that is unique per protected sample.
Fixup Table: Present only in headerless mode, this is a relocation fixup table that the loader in headerless mode uses to correct all memory displacements to the following import protection components:
Encrypted DLL names
Encrypted API names
Import dispatcher references
Dispatcher Stub
The core of the import protection mechanism is the dispatcher stub. Each stub is tailored to an individual import and consistently employs a leainstruction to access its respective obf_imp_t, which it passes as the only input to the resolver routine.
push rcx ; save RCX
lea rcx, [rip+obf_imp_t] ; fetch import-specific obf_imp_t
push rdx ; save all other registers the stub uses
push r8
push r9
sub rsp, 28h
call ObfImportResolver ; resolve the import and return it in RAX
add rsp, 28h
pop r9 ; restore all saved registers
pop r8
pop rdx
pop rcx
jmp rax ; invoke resolved import
Figure 14: Deobfuscated import dispatcher stub
Each stub is obfuscated through the mutation mechanisms outlined earlier. This applies to the resolver and import decryption routines as well. The following is what the execution flow of a stub can look like. Note the scattered addresses that while presented sequentially are actually jumping all around the code segment due to the instruction dispatchers.
obf_imp_tis the central data structure that contains the relevant information to resolve each import. It has the following form:
struct obf_imp_t { // sizeof=0x18
uint32_t CryptDllNameRVA; // NOTE: will be 64-bits, due to padding
uint32_t CryptAPINameRVA; // NOTE: will be 64-bits, due to padding
uint64_t ResolvedImportAPI; // Where the resolved address is stored
};
Figure 16: obf_imp_t in its original C struct source form
It is processed by the resolver routine, which uses the embedded RVAs to locate the encrypted DLL and API names, decrypting each in turn. After decrypting each name blob, it usesLoadLibraryAto ensure the DLL dependency is loaded in memory and leveragesGetProcAddressto retrieve the address of the import.
The import decryption logic is implemented using a Linear Congruential Generator (LCG) algorithm to generate a pseudo-random key stream, which is then used in a XOR-based stream cipher for decryption. It operates on the following formula:
Xn + 1 = (a • Xn+ c) mod 232
where:
ais always hardcoded to 17and functions as the multiplier
cis a unique 32-bit constant determined by the encryption context and is unique per-protected sample
We refer to it as the imp_decrypt_const
mod 232 confines the sequence values to a 32-bit range
The decryption logic initializes with a value from the encrypted data and iteratively generates new values using the outlined LCG formula. Each iteration produces a byte derived from the calculated value, which is then XOR’ed with the corresponding encrypted byte. This process continues byte-by-byte until it reaches a termination condition.
A fully recovered Python implementation for the decryption logic is provided in Figure 18.
Figure 18: Complete Python implementation of the import string decryption routine
Import Fixup Table
The import relocation fixup table is a fixed-size array composed of two 32-bit RVA entries. The first RVA represents the memory displacement of where the data is referenced from. The second RVA points to the actual data in question. The entries in the fixup table can be categorized into three distinct types, each corresponding to a specific import component:
Encrypted DLL names
Encrypted API names
Import dispatcher references
Figure 19: Illustration of the import fixup table
The location of the fixup table is determined by the loader’s metadata, which specifies an offset from the start of the .data section to the start of the table. During initialization, the loader is responsible for applying the relocation fixups for each entry in the table.
Figure 20: Loader metadata that shows the Import fixup table entries and metadata used to find it
Recovery
Effective recovery from an obfuscated binary necessitates a thorough understanding of the protection mechanisms employed. While deobfuscation often benefits from working with an intermediate representation (IR) rather than the raw disassembly—an IR provides more granular control in undoing transformations—this obfuscator preserves the original compiled code, merely enveloping it with additional protection layers. Given this context, our deobfuscation strategy focuses on stripping away the obfuscator’s transformations from the disassembly to reveal the original instructions and data. This is achieved through a series of hierarchical phases, where each subsequent phase builds upon the previous one to ensure comprehensive deobfuscation.
We categorize this approach into three distinct categories that we eventually integrate:
CFG Recovery
Restoring the natural control flow by removing obfuscation artifacts at the instruction and basic block levels. This involves two phases:
Accounting for instruction dispatchers: Addressing the core of control flow protection that obscure the execution flow
Function identification andrecovery: Cataloging scattered instructions and reassembling them into their original function counterparts
Import Recovery
Original Import Table: The goal is to reconstruct the original import table, ensuring that all necessary library and function references are accurately restored.
Binary Rewriting
Generating Deobfuscated Executables: This process entails creating a new, deobfuscated executable that maintains the original functionality while removing ScatterBrain’s modifications.
Given the complexity of each category, we concentrate on the core aspects necessary to break the obfuscator by providing a guided walkthrough of our deobfuscator’s source code and highlighting the essential logic required to reverse these transformations. This step-by-step examination demonstrates how each obfuscation technique is methodically undone, ultimately restoring the binary’s original structure.
Our directory structure reflects this organized approach:
Figure 21: Directory structure of our deobfuscator library
This comprehensive recovery process not only restores the binaries to their original state but also equips analysts with the tools and knowledge necessary to combat similar obfuscation techniques in the future.
CFG Recovery
The primary obstacle disrupting the natural control flow graph is the use of instruction dispatchers. Eliminating these dispatchers is our first priority in obtaining the CFG. Afterward, we need to reorganize the scattered instructions back into their original function representations—a problem known as function identification, which is notoriously difficult to generalize. Therefore, we approach it using our specific knowledge about the obfuscator.
Linearizing the Scattered CFG
Our initial step in recovering the original CFG is to eliminate the scattering effect induced by instruction dispatchers. We will transform all dispatcher call instructions into direct branches to their resolved targets. This transformation linearizes the execution flow, making it straightforward to statically pursue the second phase of our CFG recovery. This will be implemented via brute-force scanning, static parsing, emulation, and instruction patching.
Function Identification and Recovery
We leverage a recursive descent algorithm that employs a depth-first search (DFS) strategy applied to known entry points of code, attempting to exhaust all code paths by “single-stepping” one instruction at a time. We add additional logic to the processing of each instruction in the form of “mutation rules” that stipulate how each individual instruction needs to be processed. These rules aid in stripping away the obfuscator’s code from the original.
Removing Instruction Dispatchers
Eliminating instruction dispatchers involves identifying each dispatcher location and its corresponding dispatch target. Recall that the target is a uniquely encoded 32-bit displacement located at the return address of the dispatcher call. To remove instruction dispatchers, it is essential to first understand how to accurately identify them. We begin by categorizing the defining properties of individual instruction dispatchers:
Target of a Near Call
Dispatchers are always the destination of a near call instruction, represented by the E8 opcode followed by a 32-bit displacement.
References Encoded 32-Bit Displacement at Return Address
Dispatchers reference the encoded 32-bit displacement located at the return address on the stack by performing a 32-bit read from the stack pointer. This displacement is essential for determining the next execution target.
Pairing of pushfqand popfqInstructions to Safeguard Decoding
Dispatchers use a pair of pushfq and popfq instructions to preserve the state of the RFLAGS register during the decoding process. This ensures that the dispatcher does not alter the original execution context, maintaining the integrity of register contents.
End with aretInstruction
Each dispatcher concludes with a ret instruction, which not only ends the dispatcher function but also redirects control to the next set of instructions, effectively continuing the execution flow.
Leveraging the aforementioned categorizations, we implement the following approach to identify and remove instruction dispatchers:
Brute-Force Scanner for Near Call Locations
Develop a scanner that searches for all near call instructions within the code section of the protected binary. This scanner generates a huge array of potential call locations that may serve as dispatchers.
Implementation of a Fingerprint Routine
The brute-force scan yields a large number of false positives, requiring an efficient method to filter them. While emulation can filter out false positives, it is computationally expensive to do it for the brute-force results.
Introduce a shallow fingerprinting routine that traverses the disassembly of each candidate to identify key dispatcher characteristics, such as the presence of pushfq and popfq sequences. This significantly improves performance by eliminating most false positives before concretely verifying them through emulation.
Emulation of Targets to Recover Destinations
Emulate execution starting from each verified call site to accurately recover the actual dispatch targets. Emulating from the call site ensures that the emulator processes the encoded offset data at the return address, abstracting away the specific decoding logic employed by each dispatcher.
A successful emulation also serves as the final verification step to confirm that we have identified a dispatcher.
Identification of Dispatch Targets via ret Instructions
Utilize the terminating ret instruction to accurately identify the dispatch target within the binary.
The ret instruction is a definitive marker indicating the end of a dispatcher function and the point at which control is redirected, making it a reliable indicator for target identification.
Brute-Force Scanner
The following Python code implements the brute-force scanner, which performs a comprehensive byte signature scan within the code segment of a protected binary. The scanner systematically identifies all potential callinstruction locations by scanning for the 0xE8 opcode associated with near call instructions. The identified addresses are then stored for subsequent analysis and verification.
Figure 22: Python implementation of the brute-force scanner
Fingerprinting Dispatchers
The fingerprinting routine leverages the unique characteristics of instruction dispatchers, as detailed in the Instruction Dispatchers section, to statically identify potential dispatcher locations within a protected binary. This identification process utilizes the results from the prior brute-force scan. For each address in this array, the routine disassembles the code and examines the resulting disassembly listing to determine if it matches known dispatcher signatures.
This method is not intended to guarantee 100% accuracy, but rather serve as a cost-effective approach to identifying call locations with a high likelihood of being instruction dispatchers. Subsequent emulation will be employed to confirm these identifications.
Successful Decoding of a callInstruction
The identified location must successfully decode to a call instruction. Dispatchers are always invoked via a call instruction. Additionally, dispatchers utilize the return address from the call site to locate their encoded 32-bit displacement.
Absence of Subsequent callInstructions
Dispatchers must not contain any call instructions within their disassembly listing. The presence of any call instructions within a presumed dispatcher range immediately disqualifies the call location as a dispatcher candidate.
Absence of Privileged Instructions and Indirect Control Transfers
Similarly to call instructions, the dispatcher cannot include privileged instructions or indirect unconditional jmps. Any presence of any such instructions invalidates the call location.
Detection of pushfqand popfqGuard Sequences
The dispatcher must contain pushfq and popfq instructions to safeguard the RFLAGS register during decoding. These sequences are unique to dispatchers and suffice for a generic identification without worrying about the differences that arise between how the decoding takes place.
Figure 23 is the fingerprint verification routine that incorporates all the aforementioned characteristics and validation checks given a potential call location:
Figure 23: The dispatch fingerprint routine
Emulating Dispatchers to Resolve Destination Targets
After filtering potential dispatchers using the fingerprinting routine, the next step is to emulate them in order to recover their destination targets.
Figure 24: Emulation sequence used to recover dispatcher destination targets
The Python code in Figure 24 performs this logic and operates as follows:
Initialization of the Emulator
Creates the core engine for simulating execution (EmulateIntel64), maps the protected binary image (imgbuffer) into the emulator’s memory space, maps the Thread Environment Block (TEB) as well to simulate a realistic Windows execution environment, and creates an initial snapshot to facilitate fast resets before each emulation run without needing to reinitialize the entire emulator each time.
MAX_DISPATCHER_RANGE specifies the maximum number of instructions to emulate for each dispatcher. The value 45 is chosen arbitrarily, sufficient given the limited instruction count in dispatchers even with the added mutations.
A try/except block is used to handle any exceptions during emulation. It is assumed that exceptions result from false positives among the potential dispatchers identified earlier and can be safely ignored.
Emulating Each Potential Dispatcher
For each potential dispatcher address (call_dispatch_ea), the emulator’s context is restored to the initial snapshot. The program counter (emu.pc) is set to the address of each dispatcher. emu.stepi() executes one instruction at the current program counter, after which the instruction is analyzed to determine whether we have finished.
If the instruction is a ret, the emulation has reached the dispatch point.
The dispatch target address is read from the stack using emu.parse_u64(emu.rsp).
The results are captured by d.dispatchers_to_target, which maps the dispatcher address to the dispatch target. The dispatcher address is additionally stored in the d.dispatcher_locs lookup cache.
The break statement exits the inner loop, proceeding to the next dispatcher.
Patching and Linearization
After collecting and verifying every captured instruction dispatcher, the final step is to replace each call location with a direct branch to its respective destination target. Since both near call and jmp instructions occupy 5 bytes in size, this replacement can be seamlessly performed by merely patching the jmp instruction over the call.
Figure 25: Patching sequence to transform instruction dispatcher calls to unconditional jmps to their destination targets
We utilize the dispatchers_to_target map, established in the previous section, which associates each dispatcher call location with its corresponding destination target. By iterating through this map, we identify each dispatcher call location and replace the original call instruction with a jmp. This substitution redirects the execution flow directly to the intended target addresses.
This removal is pivotal to our deobfuscation strategy as it removes the intended dynamic dispatch element that instruction dispatchers were designed to provide. Although the code is still scattered throughout the code segment, the execution flow is now statically deterministic, making it immediately apparent which instruction leads to the next one.
When we compare these results to the initial screenshot from the Instruction Dispatcher section, the blocks still appear scattered. However, their execution flow has been linearized. This progress allows us to move forward to the second phase of our CFG recovery.
Figure 26: Linearized instruction dispatcher control flow
Function Identification and Recovery
By eliminating the effects of instruction dispatchers, we have linearized the execution flow. The next step involves assimilating the dispersed code and leveraging the linearized control flow to reconstruct the original functions that comprised the unprotected binary. This recovery phase involves several stages, including raw instruction recovery, normalization, and the construction of the final CFG.
Function identification and recovery is encapsulated in the following two abstractions:
Recovered instruction (RecoveredInstr): The fundamental unit for representing individual instructions recovered from an obfuscated binary. Each instance encapsulates not only the raw instruction data but also metadata essential for relocation, normalization, and analysis within the CFG recovery process.
Recovered function (RecoveredFunc): The end result of successfully recovering an individual function from an obfuscated binary. It aggregates multiple RecoveredInstr instances, representing the sequence of instructions that constitute the unprotected function. The complete CFG recovery process results in an array of RecoveredFunc instances, each corresponding to a distinct function within the binary. We will utilize these results in the final Building Relocations in Deobfuscated Binaries section to produce fully deobfuscated binaries.
We do not utilize a basic block abstraction for our recovery approach given the following reasons. Properly abstracting basic blocks presupposes complete CFG recovery, which introduces unnecessary complexity and overhead for our purposes. Instead, it is simpler and more efficient to conceptualize a function as an aggregation of individual instructions rather than a collection of basic blocks in this particular deobfuscation context.
Figure 27: RecoveredInstr type definition
Figure 28: RecoveredFunc type definition
DFS Rule-Guided Stepping Introduction
We opted for a recursive-depth algorithm given the following reasons:
Natural fit for code traversal: DFS allows us to infer function boundaries based solely on the flow of execution. It mirrors the way functions call other functions, making it intuitive to implement and reason about when reconstructing function boundaries. It also simplifies following the flow of loops and conditional branches.
Guaranteed execution paths: We concentrate on code that is definitely executed. Given we have at least one known entry point into the obfuscated code, we know execution must pass through it in order to reach other parts of the code. While other parts of the code may be more indirectly invoked, this entry point serves as a foundational starting point.
By recursively exploring from this known entry, we will almost certainly encounter and identify virtually all code paths and functions during our traversal.
Adapts to instruction mutations: We tailor the logic of the traversal with callbacks or “rules” that stipulate how we process each individual instruction. This helps us account for known instruction mutations and aids in stripping away the obfuscator’s code.
The core data structures involved in this process are the following: CFGResult, CFGStepState, and RuleHandler:
CFGResult: Container for the results of the CFG recovery process. It aggregates all pertinent information required to represent the CFG of a function within the binary, which it primarily consumes from CFGStepState.
CFGStepState: Maintains the state throughout the CFG recovery process, particularly during the controlled-step traversal. It encapsulates all necessary information to manage the traversal state, track progress, and store intermediate results.
Recovered cache: Stores instructions that have been recovered for a protected function without any additional cleanup or verification. This initial collection is essential for preserving the raw state of the instructions as they exist within the obfuscated binary before any normalization or validation processes are applied after. It is always the first pass of recovery.
Normalized cache: The final pass in the CFG recovery process. It transforms the raw instructions stored in the recovered cache into a fully normalized CFG by removing all obfuscator-introduced instructions and ensuring the creation of valid, coherent functions.
Exploration stack: Manages the set of instruction addresses that are pending exploration during the DFS traversal for a protected function. It determines the order in which instructions are processed and utilizes a visited set to ensure that each instruction is processed only once.
Obfuscator backbone: A mapping to preserve essential control flow links introduced by the obfuscator
RuleHandler: Mutation rules are merely callbacks that adhere to a specific function signature and are invoked during each instruction step of the CFG recovery process. They take as input the current protected binary, CFGStepState, and the current step-in instruction. Each rule contains specific logic designed to detect particular types of instruction characteristics introduced by the obfuscator. Based on the detection of these characteristics, the rules determine how the traversal should proceed. For instance, a rule might decide to continue traversal, skip certain instructions, or halt the process based on the nature of the mutation.
Figure 29: CFGResult type definition
Figure 30: CFGStepState type definition
Figure 31: RuleHandler type definition
The following figure is an example of a rule that is used to detect the patched instruction dispatchers we introduced in the previous section and differentiating them from standard jmpinstructions:
Figure 32: RuleHandler example that identifies patched instruction dispatchers and differentiates them from standard jmp instructions
DFS Rule-Guided Stepping Implementation
The remaining component is a routine that orchestrates the CFG recovery process for a given function address within the protected binary. It leverages the CFGStepState to manage the DFS traversal and applies mutation rules to decode and recover instructions systematically. The result will be an aggregate of RecoveredInstrinstances that constitute the first pass of raw recovery:
Figure 33: Flow chart of our DFS rule-guided stepping algorithm
The following Python code directly implements the algorithm outlined in Figure 33. It initializes the CFG stepping state and commences a DFS traversal starting from the function’s entry address. During each step of the traversal, the current instruction address is retrieved from the to_explore exploration stack and checked against the visitedset to prevent redundant processing. The instruction at the current address is then decoded, and a series of mutation rules are applied to handle any obfuscator-induced instruction modifications. Based on the outcomes of these rules, the traversal may continue, skip certain instructions, or halt entirely.
Recovered instructions are appended to the recoveredcache, and their corresponding mappings are updated within the CFGStepState. The to_explore stack is subsequently updated with the address of the next sequential instruction to ensure systematic traversal. This iterative process continues until all relevant instructions have been explored, culminating in a CFGResult that encapsulates the fully recovered CFG.
With the raw instructions successfully recovered, the next step is to normalize the control flow. While the raw recovery process ensures that all original instructions are captured, these instructions alone do not form a cohesive and orderly function. To achieve a streamlined control flow, we must filter and refine the recovered instructions—a process we refer to as normalization. This stage involves several key tasks:
Updating branch targets: Once all of the obfuscator-introduced code (instruction dispatchers and mutations) are fully removed, all branch instructions must be redirected to their correct destinations. The scattering effect introduced by obfuscation often leaves branches pointing to unrelated code segments.
Merging overlapping basic blocks: Contrary to the idea of a basic block as a strictly single-entry, single-exit structure, compilers can produce code in which one basic block begins within another. This overlapping of basic blocks commonly appears in loop structures. As a result, these overlaps must be resolved to ensure a coherent CFG.
Proper function boundary instruction: Each function must begin and end at well-defined boundaries within the binary’s memory space. Correctly identifying and enforcing these boundaries is essential for accurate CFG representation and subsequent analysis.
Simplifying with Synthetic Boundary Jumps
Rather than relying on traditional basic block abstractions—which can impose unnecessary overhead—we employ synthetic boundary jumps to simplify CFG normalization. These artificial jmp instructions link otherwise disjointed instructions, allowing us to avoid splitting overlapping blocks and ensuring that each function concludes at a proper boundary instruction. This approach also streamlines our binary rewriting process when reconstructing the recovered functions into the final deobfuscated output binary.
Merging overlapping basic blocks and ensuring functions have proper boundary instructions amount to the same problem—determining which scattered instructions should be linked together. To illustrate this, we will examine how synthetic jumps effectively resolve this issue by ensuring that functions conclude with the correct boundary instructions. The exact same approach applies to merging basic blocks together.
Synthetic Boundary Jumps to Ensure Function Boundaries
Consider an example where we have successfully recovered a function using our DFS-based rule-guided approach. Inspecting the recovered instructions in the CFGState reveals a mov instruction as the final operation. If we were to reconstruct this function in memory as-is, the absence of a subsequent fallthrough instruction would compromise the function’s logic.
Figure 35: Example of a raw recovery that does not end with a natural function boundary instruction
To address this, we introduce a synthetic jump whenever the last recovered instruction is not a natural function boundary (e.g., ret, jmp, int3).
Figure 36: Simple Python routine that identifies function boundary instructions
We determine the fallthrough address, and if it points to an obfuscator-introduced instruction, we continue forward until reaching the first regular instruction. We call this traversal “walking the obfuscator’s backbone”:
Figure 37: Python routine that implements walking the obfuscator’s backbone logic
We then link these points with a synthetic jump. The synthetic jump inherits the original address as metadata, effectively indicating which instruction it is logically connected to.
Figure 38: Example of adding a synthetic boundary jmp to create a natural function boundary
Updating Branch Targets
After normalizing the control flow, adjusting branch targets becomes a straightforward process. Each branch instruction in the recovered code may still point to obfuscator-introduced instructions rather than the intended destinations. By iterating through thenormalized_flowcache (generated in the next section), we identify branching instructions and verify their targets using thewalk_backboneroutine.
This ensures that all branch targets are redirected away from the obfuscator’s artifacts and correctly aligned with the intended execution paths. Notice we can ignore callinstructions given that any non-dispatcher callinstruction is guaranteed to always be legitimate and never part of the obfuscator’s protection. These will, however, need to be updated during the final relocation phase outlined in the Building Relocations in Deobfuscated Binaries section.
Once recalculated, we reassemble and decode the instructions with updated displacements, preserving both correctness and consistency.
Figure 39: Python routine responsible for updating all branch targets
Putting It All Together
Putting it all together, we developed the following algorithm that builds upon the previously recovered instructions, ensuring that each instruction, branch, and block is properly connected, resulting in a completely recovered and deobfuscated CFG for an entire protected binary. We utilize the recovered cache to construct a new, normalized cache. The algorithm employs the following steps:
Iterate Over All Recovered Instructions
Traverse all recovered instructions produced from our DFS-based stepping approach.
Add Instruction to Normalized Cache
For each instruction, add it to the normalized cache, which captures the results of the normalization pass.
Identify Boundary Instructions
Determine whether the current instruction is a boundary instruction.
If it is a boundary instruction, skip further processing of this instruction and continue to the next one (return to Step 1).
Calculate Expected Fallthrough Instruction
Determine the expected fallthrough instruction by identifying the sequential instruction that follows the current one in memory.
Verify Fallthrough Instruction
Compare the calculated fallthrough instruction with the next instruction in the recovered cache.
If the fallthrough instruction is not the next sequential instruction in memory,check whether it’s a recovered instruction we already normalized:
If it is, add a synthetic jump to link the two together in the normalized cache.
If it is not, obtain the connecting fallthrough instruction from the recovery cache and append it to the normalized cache.
If the fallthrough instruction matches the next instruction in the recovered cache:
Do nothing, as the recovered instruction already correctly points to the fallthrough. Proceed to Step 6.
Handle Final Instruction
Check if the current instruction is the final instruction in the recovered cache.
If it is the final instruction:
Add a final synthetic boundary jump, because if we reach this stage, we failed the check in Step 3.
Continue iteration, which will cause the loop to exit.
If it is not the final instruction:
Continue iteration as normal (return to Step 1).
Figure 40: Flow chart of our normalization algorithm
The Python code in Figure 41 directly implements these normalization steps. It iterates over the recovered instructions and adds them to a normalized cache (normalized_flow), creates a linear mapping, and identifies where synthetic jumps are required. When a branch target points to obfuscator-injected code, it walks the backbone (walk_backbone) to find the next legitimate instruction. If the end of a function is reached without a natural boundary, a synthetic jump is created to maintain proper continuity. After the completion of the iteration, every branch target is updated (update_branch_targets), as illustrated in the previous section, to ensure that each instruction is correctly linked, resulting in a fully normalized CFG:
Figure 41: Python implementation of our normalization algorithm
Observing the Results
After applying our two primary passes, we have nearly eliminated all of the protection mechanisms. Although import protection remains to be addressed, our approach effectively transforms an incomprehensible mess into a perfectly recovered CFG.
For example, Figure 42 and Figure 43 illustrate the before and after of a critical function within the backdoor payload, which is a component of its plugin manager system. Through additional analysis of the output, we can identify functionalities that would have been impossible to delineate, much less in such detail, without our deobfuscation process.
Figure 42: Original obfuscated shadow::PluginProtocolCreateAndConfigure routine
Figure 43: Completely deobfuscated and functional shadow::PluginProtocolCreateAndConfigure routine
Import Recovery
Recovering and restoring the original import table revolves around identifying which import location is associated with which import dispatcher stub. From the stub dispatcher, we can parse the respective obf_imp_treference in order to determine the protected import that it represents.
We pursue the following logic:
Identify each valid call/jmp location associated to an import
The memory displacement for these will point to the respective dispatcher stub.
For HEADERLESS mode, we need to first resolve the fixup table to ensure the displacement points to a valid dispatcher stub.
For each valid location traverse the dispatcher stub to extract the obf_imp_t
The obf_imp_tcontains the RVAs to the encrypted DLL and API names.
Implement the string decryption logic
We need to reimplement the decryption logic in order to recover the DLL and API names.
This was already done in the initial Import Protection section.
We encapsulate the recovery of imports with the following RecoveredImportdata structure:
Figure 44: RecoveredImport type definition
RecoveredImportserves as the result produced for each import that we recover. It contains all the relevant data that we will use to rebuild the original import table when producing the deobfuscated image.
Locate Protected Import CALL and JMP Sites
Each protected import location will be reflected as either an indirect near call (FF/2) or an indirect near jmp (FF/4):
Figure 45: Disassembly of import calls and jmps representation
Indirect near calls and jmps fall under the FF group opcode where the Reg field within the ModR/M byte identifies the specific operation for the group:
/2: corresponds to CALL r/m64
/4: corresponds to JMP r/m64
Taking an indirect near call as an example and breaking it down looks like the following:
FF: group opcode.
15: ModR/M byte specifying CALL r/m64 with RIP-relative addressing.
15 is encoded in binary as 00010101
Mod (bits 6-7): 00
Indicates either a direct RIP-relative displacement or memory addressing with no displacement.
Reg (bits 3-5): 010
Identifies the call operation for the group
R/M (bits 0-2): 101
In 64-bit mode with Mod 00and R/M101, this indicates RIP-relative addressing.
<32-bit displacement>: added to RIPto compute the absolute address.
To find each protected import location and their associated dispatcher stubs we implement a trivial brute force scanner that locates all potential indirect near call/jmps via their first two opcodes.
Figure 46: Brute-force scanner to locate all possible import locations
The provided code scans the code section of a protected binary to identify and record all locations with opcode patterns associated with indirect call and jmp instructions. This is the first step we take, upon which we apply additional verifications to guarantee it is a valid import site.
Resolving the Import Fixup Table
We have to resolve the fixup table when we recover imports for the HEADERLESS protection in order to identify which import location is associated with which dispatcher. The memory displacement at the protected import site will be paired with its resolved location inside the table. We use this displacement as a lookup into the table to find its resolved location.
Let’s take a jmpinstruction to a particular import as an example.
Figure 47: Example of a jmp import instruction including its entry in the import fixup table and the associated dispatcher stub
The jmpinstruction’s displacement references the memory location 0x63A88, which points to garbage data. When we inspect the entry for this import in the fixup table using the memory displacement, we can identify the location of the dispatcher stub associated with this import at 0x295E1. The loader will update the referenced data at 0x63A88with 0x295E1, so that when the jmpinstruction is invoked, execution is appropriately redirected to the dispatcher stub.
Figure 48 is the deobfuscated code in the loader responsible for resolving the fixup table. We need to mimic this behavior in order to associate which import location targets which dispatcher.
$_Loop_Resolve_ImpFixupTbl:
mov ecx, [rdx+4] ; fixup , either DLL, API, or ImpStub
mov eax, [rdx] ; target ref loc that needs to be "fixed up"
inc ebp ; update the counter
add rcx, r13 ; calculate fixup fully (r13 is imgbase)
add rdx, 8 ; next pair entry
mov [r13+rax+0], rcx ; update the target ref loc w/ full fixup
movsxd rax, dword ptr [rsi+18h] ; fetch imptbl total size, in bytes
shr rax, 3 ; account for size as a pair-entry
cmp ebp, eax ; check if done processing all entries
jl $_Loop_Resolve_ImpTbl
Figure 48: Deobfuscated disassembly of the algorithm used to resolve the import fixup table
Resolving the import fixup table requires us to have first identified the data section within the protected binary and the metadata that identifies the import table (IMPTBL_OFFSET, IMPTBL_SIZE). The offset to the fixup table is from the start of the data section.
Figure 49: Python re-implementation of the algorithm used to resolve the import fixup table
Having the start of the fixup table, we simply iterate one entry at a time and identify which import displacement (location) is associated with which dispatcher stub (fixup).
Recovering the Import
Having obtained all potential import locations from the brute-force scan and accounted for relocations in HEADERLESS mode, we can proceed with the final verifications to recover each protected import. The recovery process is conducted as follows:
Decode the location into a valid call or jmp instruction
Any failure in decoding indicates that the location does not contain a valid instruction and can be safely ignored.
Use the memory displacement to locate the stub for the import
In HEADERLESS mode, each displacement serves as a lookup key into the fixup table for the respective dispatcher.
Extract the obf_imp_tstructure within the dispatcher
This is achieved by statically traversing a dispatcher’s disassembly listing.
The first lea instruction encountered will contain the reference to the obf_imp_t.
Process the obf_imp_tto decrypt both the DLL and API names
Utilize the two RVAs contained within the structure to locate the encrypted blobs for the DLL and API names.
Decrypt the blobs using the outlined import decryption routine.
Figure 50: Loop that recovers each protected import
The Python code iterates through every potential import location (potential_stubs) and attempts to decode each presumed call or jmp instruction to an import. A try/except block is employed to handle any failures, such as instruction decoding errors or other exceptions that may arise. The assumption is that any error invalidates our understanding of the recovery process and can be safely ignored. In the full code, these errors are logged and tracked for further analysis should they arise.
Next, the code invokes a GET_STUB_DISPLACEMENT helper function that obtains the RVA to the dispatcher associated with the import. Depending on the mode of protection, one of the following routines is used:
Figure 51: Routines that retrieve the stub RVA based on the protection mode
The recover_import_stubfunction is utilized to reconstruct the control flow graph (CFG) of the import stub, while _extract_lea_refexamines the instructions in the CFG to locate the leareference to the obf_imp_t. The GET_DLL_API_NAMESfunction operates similarly to GET_STUB_DISPLACEMENT, accounting for slight differences depending on the protection mode:
Figure 52: Routines that decrypt the DLL and API blobs based on the protection mode
After obtaining the decrypted DLL and API names, the code possesses all the necessary information to reveal the import that the protection conceals. The final individual output of each import entry is captured in a RecoveredImport object and two dictionaries:
d.imports
This dictionary maps the address of each protected import to its recovered state. It allows for the association of the complete recovery details with the specific location in the binary where the import occurs.
d.imp_dict_builder
This dictionary maps each DLL name to a set of its corresponding API names. It is used to reconstruct the import table, ensuring a unique set of DLLs and the APIs utilized by the binary.
This systematic collection and organization prepare the necessary data to facilitate the restoration of the original functionality in the deobfuscated output. In Figure 53 and Figure 54, we can observe these two containers to showcase their structure after a successful recovery:
Figure 53: Output of the d.imports dictionary after a successful recovery
Figure 54: Output of the d.imp_dict_builder dictionary after a successful recovery
Observing the Final Results
This final step—rebuilding the import table using this data—is performed by the build_import_table function in the pefile_utils.py source file. This part is omitted from the blog post due to its unavoidable length and the numerous tedious steps involved. However, the code is well-commented and structured to thoroughly address and showcase all aspects necessary for reconstructing the import table.
Nonetheless, the following figure illustrates how we generate a fully functional binary from a headerless-protected input. Recall that a headerless-protected input is a raw, headerless PE binary, almost analogous to a shellcode blob. From this blob we produce an entirely new, functioning binary with the entirety of its import protection completely restored. And we can do the same for all protection modes.
Figure 55: Display of completely restored import table for a binary protected in HEADERLESS mode
Building Relocations in Deobfuscated Binaries
Now that we can fully recover the CFG of protected binaries and provide complete restoration of the original import tables, the final phase of the deobfuscator involves merging these elements to produce a functional deobfuscated binary. The code responsible for this process is encapsulated within the recover_output64.py and the pefile_utils.pyPython files.
The rebuild process comprises two primary steps:
Building the Output Image Template
Building Relocations
1. Building the Output Image Template
Creating an output image template is essential for generating the deobfuscated binary. This involves two key tasks:
Template PE Image: A Portable Executable (PE) template that serves as the container for the output binary that incorporates the restoration of all obfuscated components. We also need to be cognizant of all the different characteristics between in-memory PE executables and on-file PE executables.
Handling Different Protection Modes: Different protection modes and input stipulate different requirements.
Headerless variants have their file headers stripped. We must account for these variations to accurately reconstruct a functioning binary.
Selective protection preserves the original imports to maintain functionality as well as includes a specific import protection for all the imports leveraged within the selected functions.
2. Building Relocations
Building relocations is a critical and intricate part of the deobfuscation process. This step ensures that all address references within the deobfuscated binary are correctly adjusted to maintain functionality. It generally revolves around the following two phases:
Calculating Relocatable Displacements: Identifying all memory references within the binary that require relocation. This involves calculating the new addresses where these references will point to. The technique we will use is generating a lookup table that maps original memory references to their new relocatable addresses.
Apply Fixups: Modifies the binary’s code to reflect the new relocatable addresses. This utilizes the aforementioned lookup table to apply necessary fixups to all instruction displacements that reference memory. This ensures that all memory references within the binary correctly point to their intended locations.
We intentionally omit the details of showcasing the rebuilding of the output binary image because, while essential to the deobfuscation process, it is straightforward enough and just overly tedious to be worthwhile examining in any depth. Instead, we focus exclusively on relocations, as they are more nuanced and reveal important characteristics that are not as apparent but must be understood when rewriting binaries.
Overview of the Relocation Process
Rebuilding relocations is a critical step in restoring a deobfuscated binary to an executable state. This process involves adjusting memory references within the code so that all references point to the correct locations after the code has been moved or modified. On the x86-64 architecture, this primarily concerns instructions that use RIP-relative addressing, a mode where memory references are relative to the instruction pointer.
Relocation is necessary when the layout of a binary changes, such as when code is inserted, removed, or shifted during deobfuscation. Given our deobfuscation approach extracts the original instructions from the obfuscator, we are required to relocate each recovered instruction appropriately into a new code segment. This ensures that the deobfuscated state preserves the validity of all memory references and that the accuracy of the original control and data flow is sustained.
Understanding Instruction Relocation
Instruction relocation revolves around the following:
Instruction’s memory address: the location in memory where an instruction resides.
Instruction’s memory memory references: references to memory locations used by the instruction’s operands.
Consider the following two instructions as illustrations:
Figure 56: Illustration of two instructions that require relocation
Unconditional jmpinstructionThis instruction is located at memory address 0x1000.It references its branch target at address 0x4E22. The displacement encoded within the instruction is 0x3E1D, which is used to calculate the branch target relative to the instruction’s position. Since it employs RIP-relative addressing, the destination is calculated by adding the displacement to the length of the instruction and its memory address.
leainstructionThis is the branch target for the jmpinstruction located at 0x4E22. It also contains a memory reference to the data segment, with an encoded displacement of 0x157.
When relocating these instructions, we must address both of the following aspects:
Changing the instruction’s address: When we move an instruction to a new memory location during the relocation process, we inherently change its memory address. For example, if we relocate this instruction from 0x1000 to 0x2000, the instruction’s address becomes 0x2000.
Adjusting memory displacements: The displacement within the instruction (0x3E1Dfor the jmp, 0x157for the lea) is calculated based on the instruction’s original location and the location of its reference. If the instruction moves, the displacement no longer points to the correct target address. Therefore, we must recalculate the displacement to reflect the instruction’s new position.
Figure 57: Updated illustration demonstration of what relocation would look like
When relocating instructions during the deobfuscation process, we must ensure accurate control flow and data access. This requires us to adjust both the instruction’s memory address and any displacements that reference other memory locations. Failing to update these values invalidates the recovered CFG.
What Is RIP-Relative Addressing?
RIP-relative addressing is a mode where the instruction references memory at an offset relative to the RIP (instruction pointer) register, which points to the next instruction to be executed. Instead of using absolute addresses, the instruction encapsulates the referenced address via a signed 32-bit displacement from the current instruction pointer.
Addressing relative to the instruction pointer exists on x86 as well, but only for control-transfer instructions that support a relative displacement (e.g., JCC conditional instructions, near CALLs, and near JMPs). The x64 ISA extended this to account for almost all memory references being RIP-relative. For example, most data references in x64 Windows binaries are RIP-relative.
An excellent tool to visualize the intricacies of a decoded Intel x64 instruction is ZydisInfo. Here we use it to illustrate how a LEA instruction (encoded as488D151B510600) references RIP-relative memory at 0x6511b.
Figure 58: ZydisInfo output for the lea instruction
For most instructions, the displacement is encoded in the final four bytes of the instruction. When an immediate value is stored at a memory location, the immediate follows the displacement. Immediate values are restricted to a maximum of 32 bits, meaning 64-bit immediates cannot be used following a displacement. However, 8-bit and 16-bit immediate values are supported within this encoding scheme.
Figure 59: ZydisInfo output for the mov instruction storing an immediate operand
Displacements for control-transfer instructions are encoded as immediate operands, with the RIP register implicitly acting as the base. This is evident when decoding a jnz instruction, where the displacement is directly embedded within the instruction and calculated relative to the current RIP.
Figure 60: ZydisInfo output for the jnz instruction with an immediate operand as the displacement
Steps in the Relocation Process
For rebuilding relocations we take the following approach:
Rebuilding the code section and creating a relocation mapWith the recovered CFG and imports, we commit the changes to a new code section that contains the fully deobfuscated code. We do this by:
Function-by-function processing: rebuild each function one at a time. This allows us to manage the relocation of each instruction within its respective function.
Tracking instruction locations: As we rebuild each function, we track the new memory locations of each instruction. This involves maintaining a global relocation dictionary that maps original instruction addresses to their new addresses in the deobfuscated binary. This dictionary is crucial for accurately updating references during the fixup phase.
Applying fixupsAfter rebuilding the code section and establishing the relocation map, we proceed to modify the instructions so that their memory references point to the correct locations in the deobfuscated binary. This restores the binary’s complete functionality and is achieved by adjusting memory references to code or data an instruction may have.
Rebuilding the Code Section and Creating a Relocation Map
To construct the new deobfuscated code segment, we iterate over each recovered function and copy all instructions sequentially, starting from a fixed offset—for example, 0x1000. During this process, we build a global relocation dictionary (global_relocs) that maps each instruction to its relocated address. This mapping is essential for adjusting memory references during the fixup phase.
The global_relocs dictionary uses a tuple as the key for lookups, and each key is associated with the relocated address of the instruction it represents. The tuple consists of the following three components:
Original starting address of the function: The address where the function begins in the protected binary. It identifies the function to which the instruction belongs.
Original instruction address within the function: The address of the instruction in the protected binary. For the first instruction in a function, this will be the function’s starting address.
Synthetic boundary JMP flag: A boolean value indicating whether the instruction is a synthetic boundary jump introduced during normalization. These synthetic instructions were not present in the original obfuscated binary, and we need to account for them specifically during relocation because they have no original address.
Figure 61: Illustration of how the new code segment and relocation map are generated
The following Python code implements the logic outlined in Figure 61. Error handling and logging code has been stripped for brevity.
Figure 62: Python logic that implements the building of the code segment and generation of the relocation map
Initialize current offset Set the starting point in the new image buffer where the code section will be placed. The variable curr_off is initialized to starting_off, which is typically 0x1000. This represents the conventional start address of the .text section in PE files. For SELECTIVE mode, this will be the offset to the start of the protected function.
Iterate over recovered functions Loop through each recovered function in the deobfuscated control flow graph (d.cfg). func_ea is the original function entry address, and rfn is a RecoveredFunc object encapsulating the recovered function’s instructions and metadata.
Handle the function start address first
Set function’s relocated start address: Assign the current offset to rfn.reloc_ea, marking where this function will begin in the new image buffer.
Update global relocation map: Add an entry to the global relocation map d.global_relocs to map the original function address to its new location.
Iterate over each recovered instruction Loop through the normalized flow of instructions within the function. We use the normalized_flow as it allows us to iterate over each instruction linearly as we apply it to the new image.
Set instruction’s relocated address: Assign the current offset to r.reloc_ea, indicating where this instruction will reside in the new image buffer.
Update global relocation map: Add an entry to d.global_relocs for the instruction, mapping its original address to the relocated address.
Update the output image: Write the instruction bytes to the new image buffer d.newimgbuffer at the current offset. If the instruction was modified during deobfuscation (r.updated_bytes), use those bytes; otherwise, use the original bytes (r.instr.bytes).
Advance the offset: Increment curr_off by the size of the instruction to point to the next free position in the buffer and move on to the next instruction until the remainder are exhausted.
Align current offset to 16-byte boundaryAfter processing all instructions in a function, align curr_off to the next 16-byte boundary. We use 8 bytes as an arbitrary pointer-sized value from the last instruction to pad so that the next function won’t conflict with the last instruction of the previous function. This further ensures proper memory alignment for the next function, which is essential for performance and correctness on x86-64 architectures. Then repeat the process from step 2 until all functions have been exhausted.
This step-by-step process accurately rebuilds the deobfuscated binary’s executable code section. By relocating each instruction, the code prepares the output template for the subsequent fixup phase, where references are adjusted to point to their correct locations.
Applying Fixups
After building the deobfuscated code section and relocating each recovered function in full, we apply fixups to correct addresses within the recovered code. This process adjusts the instruction bytes in the new output image so that all references point to the correct locations. It is the final step in reconstructing a functional deobfuscated binary.
We categorize fixups into three distinct categories, based primarily on whether they apply to control flow or data flow instructions. We further distinguish between two types of control flow instructions: standard branching instructions and those introduced by the obfuscator through the import protection. Each type has specific nuances that require tailored handling, allowing us to apply precise logic to each category.
Import Relocations: These involve calls and jumps to recovered imports.
Control Flow Relocations: All standard control flow branching branching instructions.
Data Flow Relocations: Instructions that reference static memory locations.
Using these three categorizations, the core logic boils down to the following two phases:
Resolving displacement fixups
Differentiate between displacements encoded as immediate operands (branching instructions) and those in memory operands (data accesses and import calls).
Calculate the correct fixup values for these displacements using the d.global_relocs map generated prior.
Update the output image buffer
Once the displacements have been resolved, write the updated instruction bytes into the new code segment to reflect the changes permanently.
To achieve this, we utilize several helper functions and lambda expressions. The following is a step-by-step explanation of the code responsible for calculating the fixups and updating the instruction bytes.
Figure 63: Helper routines that aid in applying fixups
Define lambda helper expressions
PACK_FIXUP: packs a 32-bit fixup value into a little-endian byte array.
CALC_FIXUP: calculates the fixup value by computing the difference between the destination address (dest) and the end of the current instruction (r.reloc_ea + size), ensuring it fits within 32 bits.
IS_IN_DATA: checks if a given address is within the data section of the binary. We exclude relocating these addresses, as we preserve the data section at its original location.
Resolve fixups for each instruction
Import and data flow relocations
Utilize the resolve_disp_fixup_and_apply helper function as both encode the displacement within a memory operand.
Control flow relocations
Use the resolve_imm_fixup_and_apply helper as the displacement is encoded in an immediate operand.
During our CFG recovery, we transformed each jmp and jcc instruction to its near jump equivalent (from 2 bytes to 6 bytes) to avoid the shortcomings of 1-byte short branches.
We force a 32-bit displacement for each branch to guarantee a sufficient range for every fixup.
Update the output image buffer
Decode the updated instruction bytes to have it reflect within the RecoveredInstrthat represents it.
Write the updated bytes to the new image buffer
updated_bytesreflects the final opcodes for a fully relocated instruction.
With the helpers in place, the following Python code implements the final processing for each relocation type.
Figure 64: The three core loops that address each relocation category
Import Relocations: The first for loop handles fixups for import relocations, utilizing data generated during the Import Recovery phase. It iterates over every recovered instructionrwithin therfn.relocs_importscache and does the following:
Prepare updated instruction bytes: initialize r.updated_byteswith a mutable copy of the original instruction bytes to prepare it for modification.
Retrieve import entry and displacement: obtain the import entry from the imports dictionaryd.importsand retrieve the new RVA from d.import_to_rva_map using the import’s API name.
Apply fixup: use theresolve_disp_fixup_and_apply helper to calculate and apply the fixup for the new RVA. This adjusts the instruction’s displacement to correctly reference the imported function.
Update image buffer: write r.updated_bytesback into the new image usingupdate_reloc_in_img. This finalizes the fixup for the instruction in the output image.
Control Flow Relocations: The second for loop handles fixups for control flow branching relocations (call, jmp, jcc). Iterating over each entryin rfn.relocs_ctrlflow, it does the following:
Retrieve destination: extract the original branch destination target from the immediate operand.
Get relocated address: reference the relocation dictionaryd.global_relocsto obtain the branch target’s relocated address. If it’s a call target, then we specifically look up the relocated address for the start of the called function.
Apply fixup: useresolve_imm_fixup_and_applyto adjust the branch target to its relocated address.
Update buffer: finalize the fixup by writingr.updated_bytesback into the new image using update_reloc_in_img.
Data Flow Relocations: The final loop handles the resolution of all static memory references stored withinrfn.relocs_dataflow. First, we establish a list of KNOWN instructions that require data reference relocations. Given the extensive variety of such instructions, this categorization simplifies our approach and ensures a comprehensive understanding of all possible instructions present in the protected binaries. Following this, the logic mirrors that of the import and control flow relocations, systematically processing each relevant instruction to accurately adjust their memory references.
After reconstructing the code section and establishing the relocation map, we proceeded to adjust each instruction categorized for relocation within the deobfuscated binary. This was the final step in restoring the output binary’s full functionality, as it ensures that each instruction accurately references the intended code or data segments.
Observing the Results
To demonstrate our deobfuscation library for ScatterBrain, we conduct a test study showcasing its functionality. For this test study, we select three samples: a POISONPLUG.SHADOW headerless backdoor and two embedded plugins.
We develop a Python script, example_deobfuscator.py, that consumes from our library and implements all of the recovery techniques outlined earlier. Figure 65 and Figure 66 showcase the code within our example deobfuscator:
Figure 65: The first half of the Python code in example_deobfuscator.py
Figure 66: The second half of the Python code in example_deobfuscator.py
Running example_deobfuscator.py we can see the following. Note, it takes a bit given we have to emulate more than 16,000 instruction dispatchers that were found within the headerless backdoor.
Figure 67: The three core loops that address each relocation category
Focusing on the headerless backdoor both for brevity and also because it is the most involved in deobfuscating, we first observe its initial state inside the IDA Pro disassembler before we inspect the output results from our deobfuscator. We can see that it is virtually impenetrable to analysis.
Figure 68: Observing the obfuscated headerless backdoor in IDA Pro
After running our example deobfuscator and producing a brand new deobfuscated binary, we can see the drastic difference in output. All the original control flow has been recovered, all of the protected imports have been restored, and all required relocations have been applied. We also account for the deliberately removed PE header of the headerless backdoor that ScatterBrain removes.
Figure 69: Observing the deobfuscated headerless backdoor in IDA Pro
Given we produce functional binaries as part of the output, the subsequent deobfuscated binary can be either run directly or debugged within your favorite debugger of choice.
Figure 70: Debugging the deobfuscated headerless backdoor in everyone’s favorite debugger
Conclusion
In this blog post, we delved into the sophisticated ScatterBrain obfuscator used by POISONPLUG.SHADOW, an advanced modular backdoor leveraged by specific China-nexus threat actors GTIG has been tracking since 2022. Our exploration of ScatterBrain highlighted the intricate challenges it poses for defenders. By systematically outlining and addressing each protection mechanism, we demonstrated the significant effort required to create an effective deobfuscation solution.
Ultimately, we hope that our work provides valuable insights and practical tools for analysts and cybersecurity professionals. Our dedication to advancing methodologies and fostering collaborative innovation ensures that we remain at the forefront of combating sophisticated threats like POISONPLUG.SHADOW. Through this exhaustive examination and the introduction of our deobfuscator, we contribute to the ongoing efforts to mitigate the risks posed by highly obfuscated malware, reinforcing the resilience of cybersecurity defenses against evolving adversarial tactics.
Special thanks to Conor Quigley and Luke Jenkins from the Google Threat Intelligence Group for their contributions to both Mandiant and Google’s efforts in understanding and combating the POISONPLUG threat. We also appreciate the ongoing support and dedication of the teams at Google, whose combined efforts have been crucial in enhancing our cybersecurity defenses against sophisticated adversaries.
Google Kubernetes Engine (GKE) provides users with a lot of options when it comes to configuring their cluster networks. But with today’s highly dynamic environments, GKE platform operators tell us that they want more flexibility when it comes to changing up their configurations. To help, today we are excited to announce a set of features and capabilities designed to make GKE cluster and control-plane networking more flexible and easier to configure.
Specifically, we’ve decoupled GKE control-plane access from node-pool IP configuration, providing you with granular control over each aspect. Furthermore, we’ve introduced enhancements to each sub-component, including:
Cluster control-plane access
Added a DNS-based approach to accessing the control plane. In addition, you can now enable or disable IP-based or DNS-based access to control-plane endpoints at any time.
Each node-pool now has its own configuration, and you can now detach or attach a public IP for each node-pool independently at any time during the node-pool’s lifecycle.
You can now change a cluster’s default configuration of attaching a public IP on the newly provisioned node pools at any time. This configuration change doesn’t require you to re-create your cluster.
Regardless of how you configure a cluster’s control-plane access, or attach and detach a public IP from a node pool, the traffic between nodes to the cluster’s control plane always remains private, no matter what.
With these new changes, going forward:
GKE platform admins and operators can now easily switch between less restrictive networking configurations (e.g., control plane and/or nodes accessible from the internet) and the most restrictive configurations, where only authorized users can access the control plane, and nodes are not exposed to the internet. The decision to make a cluster public or private is no longer immutable, giving customers more flexibility without having to make upfront decisions.
There are more ways to connect to the GKE control plane. In addition to IP-based access, we now introduce DNS-based access to the control plane. You can use IAM and authentication-based policies to add policy-based, dynamic security to access the GKE control plane.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud containers and Kubernetes’), (‘body’, <wagtail.rich_text.RichText object at 0x3e9c95d658e0>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectpath=/marketplace/product/google/container.googleapis.com’), (‘image’, None)])]>
Previous challenges
Due to the complexities and varieties of customers’ workloads and use cases, it is important to provide a simple and flexible way for customers to configure and operate the connectivity to GKE control plane and GKE nodes.
Control-plane connectivity and node-pool configuration are a key part of configuring GKE. We’ve continuously enhanced GKE’s networking capabilities to address customer concerns, providing more options for secure and flexible connectivity, including capabilities such as private clusters, VPC Peering-based connectivity, Private Service Connect-based connectivity, and private/public node pools.
While there have been a lot of improvements in configuration, usability and secure connectivity, there were still certain configuration challenges when it comes to complexity, usability and scale, such as:
InflexibleGKE control plane and node access configuration: GKE customers need to make an upfront one-way decision whether to create a private or public cluster during the cluster creation process. This configuration could not be changed unless the cluster is re-created.
The node pool network IP/ type configuration could not be changed once a cluster was created.
Confusing terms such as Public / Private clusters, creating confusion as to whether the configuration is for control-plane access or node-pool configuration.
Benefits of the new features
With these changes to GKE networking, we hope you will see benefits in the following areas.
Flexibility:
Clusters now have unified and flexible configuration. Clusters with or without external endpoints all share the same architecture and support the same functionality. You can secure access to clusters based on controls and best practices that meet your needs. All communication between the nodes in your cluster and the control plane use a private internal IP address.
You can change the control plane access and cluster node configuration settings at any time without having to re-create the cluster.
Security:
DNS-based endpoints with VPC Service Controls provide a multi-layer security model that protects your cluster against unauthorized networks as well as from unauthorized identities accessing the control plane. VPC Service Controls integrate with Cloud Audit Logs to monitor access to the control plane.
Private nodes and the workloads running on them are not directly accessible from the public internet, significantly reducing the potential for external attacks targeting your workloads.
You can block control plane access from Google Cloud external IP addresses or from external IP addresses to fully isolate the cluster control plane and reduce exposure to potential security threats.
Compliance: If you work in an industry with strict data-access and storage regulations, private nodes help ensure that sensitive data remains within your private network.
Control: Private nodes give you granular control over how traffic flows in and out of your cluster. You can configure firewall rules and network policies to allow only authorized communication. If you operate across a multi-cloud environment, private nodes can help you establish secure and controlled communication between different environments.
Getting started
Accessing the cluster control plane
There are now several ways to access a cluster’s control plane: via traditional public or private IP-based endpoints, and the new DNS-based endpoint. Whereas IP-based endpoints entail tedious IP address configuration (including static authorized network configuration, allowing private accessing from any regions, etc.), DNS-based endpoints offer a simplified, IAM policy-based, dynamic, flexible and more secure way to access a cluster’s control plane.
With these changes, you can now configure the cluster’s control plane to be reachable by all three endpoints (DNS-based, public or private IP-based) at same time, locking the cluster down to the granularity of a single endpoint in any permutation that you would like. You can apply your desired configuration at cluster creation time or adjust it later.
Here’s how to configure access for GKE node-pools.
GKE Standard Mode: In GKE Standard mode of operation, a private IP is always attached to every node no matter what. This private IP is used for private connectivity to the cluster’s control plane.
You can add or remove a public IP to all nodes in a node-pool at node-pool creation time. This configuration can be performed on each node-pool independently.
Each cluster has a default behavior flag that’s used at node-pool creation time if the flag is not explicitly set beforehand during node-pool creation time.
Note: Mutating a cluster’s default state does not change behavior of existing node pools. The new state is used only when a new node-pool is being created.
GKE Autopilot mode of operation: All workloads running on nodes with or without a public IP are based on the cluster’s default behavior. You can override the cluster’s default behavior on each workload independently by adding the following nodeSelector to your Pod specification:
However, overriding a cluster’s default behavior causes all workloads for which behavior hasn’t been explicitly set to be rescheduled), to run on nodes that match the cluster’s default behavior.
Conclusion
Given the complexity and variety of workloads that run on GKE, it’s important to have a simple and flexible way to configure and operate the connectivity to the GKE control plane and nodes. We hope these enhancements to GKE control-plane connectivity and node-pool configuration will bring new levels of flexibility and simplicity to GKE operations. For further details and documentation, please see:
More and more customers deploy their workloads on Google Cloud. But what if your workloads are sitting in another cloud? Planning, designing, and implementing a migration of your workloads, data, and processes is not an easy task. It gets even harder if you have to meet requirements that have an impact on the migration, such as avoiding downtime (also known as a zero-downtime migration). Moreover, some migrations require a certain amount of refactoring, for example, adapting your workloads to a new environment. This opens up a series of challenges, especially if you’re dealing with third-party or legacy software. You might also need to adapt your deployment and operational processes to work with your new environment.
And what if you don’t want to migrate all your workloads? Even if you’re not moving everything to Google Cloud, adopting a multicloud approach is still a migration. Many organizations choose to keep some workloads in their current cloud provider while moving others to Google Cloud.
Although managing workloads across multiple clouds has its own challenges, particularly when it comes to workload distribution and inter-cloud connectivity, a well-executed multicloud strategy lets you maintain flexibility, avoid vendor lock-in, and improve system resilience.
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3e9c95d1ca30>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
To help you in your migration journey, we published a series of reference guides about migrating from Amazon Web Services (AWS) to Google Cloud. This series aims to help you design, plan, and implement a migration process from AWS to Google Cloud. It can also help decision makers who are evaluating migration opportunities and want to explore what it looks like to migrate. For example, the series includes guides that cover migration journeys, such as:
These guides follow the phases of the Google Cloud migration framework (assess, plan, migrate, optimize) in the context of specific AWS to Google Cloud migration use cases.
This approach helps to avoid big-bang, risky migrations, when working on each migration plan task. For details about completing each task of this migration plan, see the AWS to Google Cloud migration guides.
Ready to learn more? Learn more about migrating to Google Cloud and discover how Google Cloud Consulting can help you learn, build, operate and succeed.
Organizations are increasingly using Confidential Computing to help protect their sensitive data in use as part of their data protection efforts. Today, we are excited to highlight new Confidential Computing capabilities that make it easier for organizations of all sizes to adopt this important privacy-preserving technology.
1. Confidential GKE Nodes on the general-purpose C3D machine series for GKE Standard mode, generally available
Previously, Confidential GKE Nodes were only available on two machine series powered by the 2nd and 3rd Gen AMD EPYC™ processors: the general-purpose N2D machine series and the compute-optimized C2D machine series. Today, Confidential GKE Nodes are also generally available on the newer, more performant C3D machine series with AMD SEV in GKE Standard mode.
The general-purpose C3D machine series is powered by 4th Gen AMD EPYC™ (Genoa) processors to deliver optimal, reliable, and consistent performance. Customers often use Confidential GKE Nodes to address potential concerns about cloud provider risk, especially since no code changes are required to enable it.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud security products’), (‘body’, <wagtail.rich_text.RichText object at 0x3e9c95d16be0>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
2. Confidential GKE Nodes on GKE Autopilot mode, generally available
Google Kubernetes Engine (GKE) offers two modes of operation: Standard and Autopilot. In Standard mode, you manage the underlying infrastructure, including configuring the individual nodes. In Autopilot mode, GKE manages the underlying infrastructure such as node configuration, autoscaling, auto-upgrades, baseline security configurations, and baseline networking configuration.
Previously, Confidential GKE Nodes were only offered on GKE Standard mode. Today, Confidential GKE Nodes are generally available on GKE Autopilot mode with the general purpose N2D machine series running with AMD Secure Encryption Virtualization (AMD SEV). This means that you can now use Confidential GKE Nodes to protect your data in use without having to manage the underlying infrastructure.
Confidential GKE Nodes can be enabled on new GKE Autopilot clusters with no code changes. Simply add the command --enable-confidential-nodes during new cluster creation. Additional pricing does apply and this new offering is available in all regions that offer the N2D machine series. Go here to get started today.
3. Confidential Space with Intel TDX-based Confidential VMs, in preview
Confidential Space allows multiple parties to securely collaborate on computations using their combined data without revealing their individual datasets to each other or to the operator enabling this collaboration. This is achieved by isolating data within a Trusted Execution Environment (TEE).
We are seeing adoption and need for these capabilities that are putting sensitive data to use in a private and compliant manner in financial services, Web3, and other industries.
Confidential Space is built on Confidential VMs. Previously, Confidential Space was only available on Confidential VMs with AMD Secure Encryption Virtualization (AMD SEV) enabled. Today, Confidential Space is also available on Confidential VMs with Intel Trust Domain Extensions (Intel TDX) enabled in preview.
Confidential Space with Intel TDX enabled offers data confidentiality, data integrity, and hardware-rooted attestation, further enhancing security. Confidential Space with Intel TDX runs on the general purpose C3 machine series, which are powered by 4th Gen Intel Xeon Scalable CPUs.
These performant C3 VMs also have Intel Advanced Matrix Extensions (Intel AMX), a new built-in accelerator that helps improve the performance of deep-learning training and inference on the CPU, on by default. Confidential Space supporting the additional confidential computing type provides users greater flexibility in selecting the right CPU platform based on performance, cost, and security requirements. Learn more about Confidential Space or check out this new Youtube video about Intel TDX.
4. Confidential VMs with NVIDIA H100 GPUs, in preview
We expanded our capabilities for secure computation last year when we unveiled Confidential VMs on the accelerator-optimized A3 machine series with NVIDIA H100 GPUs. This offering extends hardware-based data protection from the CPU to GPUs, helping to ensure the confidentiality and integrity of artificial intelligence (AI), machine learning (ML), and scientific simulation workloads leveraging GPUs can be protected while data is in use.
Today, these confidential GPUs are available in preview. Confidential VMs on the A3 machine series protects data and code in use, so that means sensitive training data or data labels, proprietary models or model weights, and top secret queries remain protected even during compute-intensive operations, like training, fine tuning, or serving.
This groundbreaking technology combines the power of Confidential Computing and accelerated computing to enable customers to harness the potential of AI while helping to maintain high levels of data security and IP protection, which can open new possibilities for innovation in regulated industries and collaborative AI development.
You can sign up here to try Confidential VMs with NVIDIA H100 GPUs. To learn more, check out our previous announcements on this offering here and here.
What’s coming in 2025
Google Cloud is committed to expanding Confidential Computing to more products and services because we want customers to have easy access to the latest in security innovation. Whether that’s adding Confidential Computing support to newer hardware or on accelerators or to services like GKE Autopilot, we aim to provide our customers with a comprehensive set of Confidential Computing solutions.
Confidential Computing is an essential technology for protecting sensitive data in the cloud, and we look forward to innovating with you in this space. You can explore the Confidential Computing products here.
Today, an increasing number of organizations are using GPUs to run inference1 on their AI/ML models. Since the number of GPUs needed to serve a single inference workload varies, organizations need more granularity in the number of GPUs in their virtual machines (VMs) to keep costs low while scaling with user demand.
You can use A3 High VMs powered by NVIDIA H100 80GB GPUs in multiple generally available machine types of 1NEW, 2NEW, 4NEW, and 8 GPUs.
Accessing smaller H100 machine types
All A3 machine types are available through the fully managed Vertex AI, as nodes through Google Kubernetes Engine (GKE), and as VMs through Google Compute Engine.
Vertex AI Model Garden and Online Prediction (Spot)
Spot
DWS Flex Start mode
a3-highgpu-2g NEW (2 GPUs, 160 GB)
Vertex AI Model Garden and Online Prediction (On-demanda, Spot)
a3-highgpu-4g NEW (4 GPUs, 320 GB)
a3-highgpu-8g (8 GPUs, 640 GB)
Vertex AI Online Prediction (On-Demand, Spot)
Vertex AI Training (On-demand, Spot, DWS Flex Start mode )
On-demand
Spot
DWS Flex Start mode
DWS Calendar mode
a3-megagpu-8g (8 GPUs, 640 GB)
aAvailable only through Model Garden owned capacity.
Google Kubernetes Engine
For almost a decade, GKE has been the platform-of-choice for running web applications and microservices, and now it provides a cost efficient, highly scalable, and open platform for training and serving AI workloads. GKE Autopilot reduces operational cost and offers workload-level SLAs, and is a fantastic choice for inference workloads — bring your workload and let Google do the rest. You can use the 1, 2, and 4 A3 High GPU machine types through both GKE Standard and GKE Autopilot modes of operation.
Below are two examples of creating node pools in your GKE cluster with a3-highgpu-1g machine type using Spot VMs and Dynamic Workload Scheduler Flex Start mode.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud infrastructure’), (‘body’, <wagtail.rich_text.RichText object at 0x3e447b6804c0>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/compute’), (‘image’, None)])]>
Using Spot VMs with GKE
Here’s how to request and deploy a3-highgpu-1g Spot VM on GKE using the gcloud API.
Vertex AI is Google Cloud’s fully managed, unified AI development platform for building and using predictive and generative AI. With the new 1, 2, and 4 A3 High GPU machine types, Model Garden customers can deploy hundreds of open models cost-effectively and with strong performance.
What our customers are saying
“We use Google Kubernetes Engine to run the backend for our AI-assisted software development product. Smaller A3 machine types have enabled us to reduce the latency of our real-time code assist models by 36% compared to A2 machine types, significantly improving user experience.” – Eran Dvey Aharon, VP R&D, Tabnine
Get started today
At Google Cloud, our goal is to provide you with the flexibility you need to run inference for your AI and ML models cost-effectively as well as with great performance. The availability of A3 High VMs using NVIDIA H100 80GB GPUs in smaller machine types provides you with the granularity you need to scale with user demand while keeping costs in check.
1. AI or ML inference is the process by which a trained AI model uses its training data to calculate output data or make predictions about new data points or scenarios.
What goes into your Kubernetes software? Understanding the origin of the software components you deploy is crucial for mitigating risks and ensuring the trustworthiness of your applications. To do this, you need to know your software supply chain.
Google Cloud is committed to providing tools and features that enhance software supply chain transparency, and today we’re excited to announce that you can now verify the integrity of Google Kubernetes Engine components with SLSA, the Supply-chain Levels for Software Artifacts framework.
SLSA is a set of standards that can help attest the integrity of software components. We’ve begun to publish SLSA Verification Summary Attestations (VSAs) for GKE’s Container-Optimized OS (COS) virtual machine (VM) images to GitHub. We’ve also enhanced Google Compute Engine (GCE) audit logs to include VM image identifiers, and begun to route GKE Kubernetes Control Plane GCE audit logs to customer projects. This allows you to use SLSA VSAs to authenticate the VM images used in your GKE clusters.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud security products’), (‘body’, <wagtail.rich_text.RichText object at 0x3e447b64f970>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
GCE audit logs improvements
Google Compute Engine audit logs now include the GCE image ID in records related to instance creation events (such as insert, bulk insert, and update operations) when an instance is created from an image. This allows you to trace the precise image used to launch each instance even if an image is deleted and recreated with the same name, as each image instance has a unique immutable ID.
The ID is used to uniquely identify the image when verifying its provenance and integrity using the SLSA VSAs described below. This can provide an invaluable audit trail for security and compliance purposes.
We introduced a new attachDisks field, under usedResources in the metadata field, that for attached disks records the source image name, source image id, and whether it was used as the boot disk. You can find this information in the Logs explorer using a query like:
GCE instance insert audit log record with VM image id field
GKE Control Plane audit and integrity logs now forwarded to your project
New GKE clusters running version 1.29 or later now forward their Control Plane GCE audit logs records for insert, bulk insert, and update operations, and their Shielded VM integrity logs, to the customer project hosting the GKE cluster.
You can identify Control Plane VM instance log records by the presence of a new metadata field. To view the logs use a log explorer query like:
code_block
<ListValue: [StructValue([(‘code’, ‘resource.type=”gce_instance” AND (jsonPayload.metadata.isKubernetesControlPlaneVM=”true” OR protoPayload.metadata.isKubernetesControlPlaneVM=”true”)’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e447b64f790>)])]>
Additionally, we’ve added to the forwarded logs a new parentResource map under metadata with two fields, parentResourceType, with a value of “gke_cluster”, and parentResourceId, with the cluster hash as a value, so you can tell which cluster the VMs in the forwarded log records belong to in case you have more than one cluster per project.
This enhancement allows you to gain visibility into the VM images used to create the Control Plane VMs, and the integrity status of the instances, further strengthening your ability to audit and secure your GKE clusters.
KCP VM instance audit log record forwarded to customer project
GKE bolsters VM image verification with SLSA VSAs
Google Kubernetes Engine (GKE) is taking a significant step forward in supply chain security by publishing SLSA Verification Summary Attestations (VSAs) for GKE Container Optimized OS (COS) based VM images. These attestations are available in the Google Cloud GKE VSA GitHub repository. This initiative can provide you with cryptographic proof of the integrity and provenance of the GKE VM images you’re using, help ensure that they haven’t been tampered with, and that they originate from a trusted source.
To locate a VSA for the COS VM image used in your GKE VM instances. Look in the folders at the root of the GitHub repository:
The folder gke-master-images:78064567238 contains VSAs for the Kubernetes control plane VM images.
The folder gke-node-images:238739202978 contains the VSAs for the node VM images.
Using the image ID found in the audit logs you can locate the matching VSA. For example, gke-node-images:238739202978/gke-12811-gke1044000-cos-109-17800-218-52-c-pre:3031893369549136349.intoto.jsonl is the VSA for the the node VM image with an id of 3031893369549136349.
Independent verification with slsa-verifier
You can independently verify the authenticity of GKE VM images using the open-source slsa-verifier tool. This tool allows you to validate the integrity of your GKE VM images by combining the GCE image name and ID, the VSA, and Google’s VSA public signing key.
The public key is
code_block
<ListValue: [StructValue([(‘code’, ‘—–BEGIN PUBLIC KEY—–rnMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeGa6ZCZn0q6WpaUwJrSk+PPYEscarn3Xkk3UrxvbQtoZzTmq0zIYq+4QQl0YBedSyy+XcwAMaUWTouTrB05WhYtg==rn—–END PUBLIC KEY—–‘), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e447b64f460>)])]>
To verify a VM image use slsa-verifier as follows:
VM_IMAGE_PROJECT_NAME is the name of the project hosting the VM image (e.g., gke-node-images)
VM_IMAGE_NAME is the image name (e.g., gke-12811-gke1044000-cos-109-17800-218-52-c-pre)
VM_IMAGE_ID is the image ID (e.g. 30318933695491363493)
KEY_PATH is the path to the saved public key
Next steps
These enhancements reflect Google Cloud’s commitment to providing you with the tools and capabilities needed to help build and manage secure, transparent software supply chains. To learn more about how to verify the integrity of the GKE control plane check out the user guide. You can find more information on securing your GKE cluster in the documentation.
Comprehensive agent evaluation is essential for building the next generation of reliable AI. It’s not enough to simply check the outputs; we need to understand the “why” behind an agent’s actions – its reasoning, decision-making process, and the path it takes to reach a solution.
That’s why today, we’re thrilled to announce Vertex AI Gen AI evaluation service is now in public preview. This new feature empowers developers to rigorously assess and understand their AI agents. It includes a powerful set of evaluation metrics specifically designed for agents built with different frameworks, and provides native agent inference capabilities to streamline the evaluation process.
In this post, we’ll explore how evaluation metrics work and share an example of how you can apply this to your agents.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud AI and ML’), (‘body’, <wagtail.rich_text.RichText object at 0x3e447b328e20>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/vertex-ai/’), (‘image’, None)])]>
Evaluate agents using Vertex AI Gen AI evaluation service
Our evaluation metrics can be grouped in two categories: final response and trajectory evaluation.
Final response asks a simple question: does your agent achieve its goals? You can define custom final response criteria to measure success according to your specific needs. For example, you can assess whether a retail chatbot provides accurate product information or if a research agent summarizes findings effectively, using appropriate tone and style.
To look below the surface, we offer trajectory evaluation to analyze the agent’s decision-making process. Trajectory evaluation is crucial for understanding your agent’s reasoning, identifying potential errors or inefficiencies, and ultimately improving performance. We offer six trajectory evaluation metrics to help you answer these questions:
1. Exact match: Requires the AI agent to produce a sequence of actions (a “trajectory”) that perfectly mirrors the ideal solution.
2. In-order match: The agent’s trajectory needs to include all the necessary actions in the correct order, but it might also include extra, unnecessary steps. Imagine following a recipe correctly but adding a few extra spices along the way.
3. Any-order match: Even more flexible, this metric only cares that the agent’s trajectory includes all the necessary actions, regardless of their order. It’s like reaching your destination, regardless of the route you take.
4. Precision: This metric focuses on the accuracy of the agent’s actions. It calculates the proportion of actions in the predicted trajectory that are also present in the reference trajectory. A high precision means the agent is making mostly relevant actions.
5. Recall: This metric measures the agent’s ability to capture all the essential actions. It calculates the proportion of actions in the reference trajectory that are also present in the predicted trajectory. A high recall means the agent is unlikely to miss crucial steps.
6. Single-tool use: This metric checks for the presence of a specific action within the agent’s trajectory. It’s useful for assessing whether an agent has learned to utilize a particular tool or capability.
Compatibility meets flexibility
Vertex AI Gen AI evaluation service supports a variety of agent architectures.
With today’s launch, you can evaluate agents built with Reasoning Engine (LangChain on Vertex AI), the managed runtime for your agentic applications on Vertex AI. We also support agents built by open-source frameworks, including LangChain, LangGraph, and CrewAI – and we are planning to support upcoming Google Cloud services to build agents.
For maximum flexibility, you can evaluate agents using a custom function that processes prompts and returns responses. To make your evaluation experience easier, we offer native agent inference and automatically log all results in Vertex AI experiments.
Agent evaluation in action
Let’s say you have the following LangGraph customer support agent, and you aim to assess both the responses it generates and the sequence of actions (or “trajectory”) it undertakes to produce those responses.
To assess an agent using Vertex AI Gen AI evaluation service, you start preparing an evaluation dataset. This dataset should ideally contain the following elements:
User prompt: This represents the input that the user provides to the agent.
Reference trajectory: This is the expected sequence of actions that the agent should take to provide the correct response.
Generated trajectory: This is the actual sequence of actions that the agent took to generate a response to the user prompt.
Response: This is the generated response, given the agent’s sequence of actions.
A sample evaluation dataset is shown below.
After you gather your evaluation dataset, define the metrics that you want to use to evaluate the agent. For a complete list of metrics and their interpretations, refer to Evaluate Gen AI agents. Some metrics you can define are listed here:
Notice that the response_follows_trajectory_metric is a custom metric that you can define to evaluate your agent.
Standard text generation metrics, such as coherence, may not be sufficient when evaluating AI agents that interact with environments, as these metrics primarily focus on text structure. Agent responses should be assessed based on their effectiveness within the environment. Vertex AI Gen AI Evaluation service allows you to define custom metrics, like response_follows_trajectory_metric, that assess whether the agent’s response logically follows from its tool choices. For more information on these metrics, please refer to the official notebook.
With your evaluation dataset and metrics defined, you can now run your first agent evaluation job on Vertex AI. Please see the code sample below.
code_block
<ListValue: [StructValue([(‘code’, ‘# Import libraries rnimport vertexairnfrom vertexai.preview.evaluation import EvalTaskrnrn# Initiate Vertex AI sessionrnvertexai.init(project=”my-project-id”, location=”my-location”, experiment=”evaluate-langgraph-agent)rnrn# Define an EvalTaskrnresponse_eval_tool_task = EvalTask(rn dataset=byod_eval_sample_dataset,rn metrics=response_tool_metrics,rn)rnrn# Run evaluationrnresponse_eval_tool_result = response_eval_tool_task.evaluate( experiment_run_name=”response-over-tools”)’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e447b64ad30>)])]>
To run the evaluation, initiate an `EvalTask` using the predefined dataset and metrics. Then, run an evaluation job using the evaluate method. Vertex AI Gen AI evaluation tracks the resulting evaluation as an experiment run within Vertex AI Experiments, the managed experiment tracking service on Vertex AI. The evaluation results can be viewed both within the notebook and the Vertex AI Experiments UI. If you’re using Colab Enterprise, you can also view the results in the Experiment side panel as shown below.
Vertex AI Gen AI evaluation service offers summary and metrics tables, providing detailed insights into agent performance. This includes individual user input, trajectory results, and aggregate results for all user input and trajectory pairs across all requested metrics.
Access to these granular evaluation results enables you to create meaningful visualizations of agent performance, including bar and radar charts like the one below:
Get started today
Explore the Vertex AI Gen AI evaluation service in public preview and unlock the full potential of your agentic applications.
Tchibo, a well-known coffee retailer and lifestyle brand based in Germany, needed a faster, smarter way to manage and interpret vast amounts of customer feedback across its diverse product offerings and sales channels. To meet this need, they adopted the AlloyDB for PostgreSQL database, harnessing its advanced analytics and AI capabilities to streamline data retrieval and provide real-time insights.
In this guest post from Henning Kosmallaand Dominik Nowatschin, we learn how Tchibo’s migration accelerated feedback analysis by a factor of 10, empowering Tchibo’s teams to respond quickly to customer needs and reinforcing the company’s commitment to customer-centric innovation.
At Tchibo, we’re about more than just coffee — we’re constantly brewing new ways to connect with our customers.
We’ve grown from a coffee-focused business to a multi-channel retail model, spanning our own stores, e-commerce, and shop-in-shop sections in grocery stores. This setup allows us to serve a diverse customer base, each with unique needs and preferences, while offering an evolving selection of non-food items — from apparel to kitchenware — delivering “a new world every week.”
But it’s not always a smooth pour. Rising global challenges, from inflation to new AI-driven customer expectations, require us to make data-driven decisions quickly to stay competitive. Our previous cloud database solution could handle basic data retrieval, but it couldn’t keep up with the scale and complexity of the data we rely on across our three sales channels. As our data needs grew, we faced a number of issues: our query speeds slowed which delayed access to customer data, we suffered from labor-intensive feedback compilation, and we had difficulty extracting actionable insights from diverse data sources.
Queries often exceeded 10 seconds, even for straightforward insights. And compiling customer feedback reports required up to three days of manual work to sort, categorize, and analyze. We also lacked the flexibility to support advanced, AI-driven applications. This limited our ability to implement innovative tools like retrieval-augmented generation (RAG) workflows, which combine structured and unstructured data for deeper context in AI queries. That’s why we turned to AlloyDB — to power the insights that keep customers at the center of every decision we make.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud databases’), (‘body’, <wagtail.rich_text.RichText object at 0x3e447b250f10>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/products?#databases’), (‘image’, None)])]>
Finding the perfect blend of speed and scale
AlloyDB provided a powerful solution to the limitations we faced with our old database. Its advanced analytics capabilities, built-in vector search, and familiar PostgreSQL foundation offered the speed, adaptability, and usability we needed to serve up insights as fresh and fast as our coffee. One of our most impactful applications, Customer Voice, gives employees instant access to relevant customer feedback. This tool compiles data from product reviews and other sources into actionable summaries, answering questions like, “How do customers feel about our new coffee pad machine?” with concise, actionable summaries.
AlloyDB serves as the foundation of our Customer Voice application, managing a complete data pipeline to support real-time feedback analysis. Its architecture efficiently handles data storage, search, and query processing, so Tchibo teams can gain fresh perspectives from customer insights. Here’s how AlloyDB supports our specific needs:
Data storage: AlloyDB organizes customer feedback and product meta-information in a flexible structure, supporting both standard and advanced queries. This setup allows us to run traditional queries (e.g., “return all reviews with a positive sentiment”) as well as nearest-neighbor (NN) searches using embedding columns to add depth and relevance to the data.
Query interpretation: When employees pose questions to the Customer Voice assistant, a large language model (LLM)—currently Claude 3.5 Sonnet on Vertex AI—interprets the query, identifying core topics like product or category to deliver targeted, relevant answers.
Retrieval and filtering: AlloyDB combines structured queries, NN searches, and reranking/filtering steps to retrieve relevant reviews. The LLM further enriches the data with clustering and summary statistics, providing a full view of customer opinions.
Presentation: Customer Voice delivers these insights through a streamlined interface that highlights individual reviews, key statistics, and summaries, making it easy for employees to act on the information.
Serving data to perfection to fuel decision-making
AlloyDB has transformed Tchibo’s approach to data by enabling faster, deeper, and more scalable access to the customer feedback and analytics we rely on for decision-making.
Supporting high-performance analytics and RAG workflows, AlloyDB now delivers nearly instant insights. Complex queries that once took up to 10 seconds now return results in about a second, enabling faster, data-driven decisions across teams. Generating detailed customer feedback reports previously took days of manual effort. With AlloyDB, this process now takes seconds. This leap has strengthened our commitment to staying connected to customer needs and preferences in real time.
Furthermore, the fully managed operation of AlloyDB has reduced operational overhead, simplifying our ability to scale as data demands grow. Although continuity wasn’t our primary consideration in choosing AlloyDB, its 99.99% SLA availability provides valuable reliability for supporting long-term goals.
Beyond Customer Voice, AlloyDB also supports broader AI initiatives, such as an internal chatbot for intranet queries, giving us the flexibility to scale various retrieval-augmented generation (RAG) use cases efficiently across the organization. Looking forward, we’re exploring expanded AlloyDB capabilities to integrate more structured and unstructured data into our analytics. Partnering with Google Cloud, we’re positioned to explore new data solutions to serve up richer insights, driving growth and innovation at Tchibo.
Ready to get started with AlloyDB in your own environment? Check out the following resources:
It’s a new year, which means new beginnings, fresh starts, ambitious resolutions, and the sinking feeling that you still have outdated tech slowing down and creating unnecessary costs for your business. Thankfully, there’s a no-cost, easy to deploy New Year’s resolution to add to your tech stack.
With just a USB stick, and a side of enthusiasm, you can install ChromeOS Flex and breathe new life into your existing hardware, transforming aging laptops, kiosks, and more into fast, secure, and modern devices. It’s the perfect solution for businesses hoping to refresh devices, improve security, and embrace sustainability – all while saving money. And going into 2025, we’ve certified over 600 devices to work effortlessly with ChromeOS Flex, ensuring that almost every business can benefit from it.
So many organizations have been able to benefit from ChromeOS Flex, from Mercado Libre upgrading devices to improve contact center productivity by 25%, to Strawberry Hotels who deployed ChromeOS Flex to 2,000 devices in under 48 hours to evade ransomware and bolster security. And as ChromeOS Flex helps businesses modernize, we’re always looking for ways to improve. In fact, we’ve made a few updates to support the ever evolving needs of businesses as we head into the new year.
Boosting security, streamlining deployment
We’re bringing the convenience of zero-touch enrollment to ChromeOS Flex. This means that devices installing ChromeOS Flex for the first time can be automatically enrolled into your business domain. With automatic enrollment, IT admins can quickly configure devices with the necessary policies and applications, saving valuable time and resources while helping end-users get up and running more quickly.
Additionally, admins can opt in to allow ChromeOS Flex devices to receive manufacturer provided firmware updates. By keeping firmware up to date, businesses can ensure their devices are protected against the latest threats and benefit from performance optimizations and bug fixes.
Keep your devices running
While we continue to invest in ChromeOS Flex, we also acknowledge that it’s easy for hardware deployments to fall behind. Later this year, it’s expected that hundreds of millions of Windows 10 PCs will lose support. If this is affecting your business, that doesn’t mean your only option is to invest in new devices.
Instead of scrapping perfectly good hardware, ChromeOS Flex can be a tool to modernize without the need to purchase an entirely new device. This gives your deployment a new lease on life, with much of the speed, security, and reliability that comes with ChromeOS. And this not only saves on substantial device costs, but also contributes to environmental sustainability by reducing e-waste.
With ChromeOS device management, it’s easy to manage policies for your entire fleet, all while robust security features protect you from the latest threats. For example, data loss prevention keeps sensitive information secure and remotely wiping a lost or stolen device is as easy as a few clicks. Best of all, you can manage both ChromeOS Flex and ChromeOS devices side by side with the Google Admin console.
Flex your tech in 2025
There has never been a better time to embrace solutions that support your business with enhanced security, lower costs, and sustainability in mind.
ChromeOS Flex delivers on all fronts, and we’re continuing to invest in it as a secure, easy-to-manage, and sustainable solution for businesses. And stay tuned, as we have even more improvements to share with you later this year!
Want to learn more? Visit our website to see how ChromeOS Flex can breathe new life into your devices.
Last year, Google Cloud and Oracle forged a strategic partnership to accelerate cloud transformation for businesses, allowing them to integrate Oracle’s robust database capabilities within Google Cloud’s environment.
This partnership applies to Oracle databases, as well as the applications that run alongside them. That means migrating existing Oracle database applications to Google Cloud, where customers enjoy the benefits of our performance, stability and security. Oracle also offers a variety of enterprise and middleware applications that aren’t databases per se that you may want to migrate. And of course, you can build and deploy new cloud-native applications using Oracle databases and services on Google Cloud’s services and infrastructure.
This blog post explores migrating Oracle applications and databases to Google Cloud. It details various migration paths, from fully managed services to customized options, with an emphasis on how containerization and GraalVM can help optimize performance and scalability.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud databases’), (‘body’, <wagtail.rich_text.RichText object at 0x3ea5331f3490>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/products?#databases’), (‘image’, None)])]>
Oracle databases in Google Cloud
Google Cloud offers several migration paths for your Oracle workloads, from fully managed to highly customized, offering flexibility depending on your needs.
High performance:
Exadata Cloud Service in Google Cloud: Run Oracle’s high-performance hardware, Exadata, directly within Google’s data centers. This is ideal for demanding workloads that need the maximum available speed and power.
Fully managed:
Autonomous Database on Google Cloud: Google Cloud handles everything, making it super easy to run Oracle. It automates all the essential management tasks — provisioning, patching, scaling, backup, and recovery — so you can focus on what matters most to your business.
As highly managed offerings, both of the above migration paths require very few (if any) changes to your Oracle Database instances, and the physical migration itself can be performed by our support team.
Flexible options:
When you choose one of these flexible options, however, there are several things to think about to get the best possible performance from your new cloud-based environment.
Compute Engine: Run Oracle on virtual machines, giving you control and flexibility. This is good for “lift and shift” migrations of existing systems.
Google Kubernetes Engine (GKE): Using OraOperator, Oracle’s Database Operator for Kubernetes, you can deploy Oracle pluggable databases; you can read more about the use cases for this below.
Containerize Oracle-based applications
This partnership is designed to provide businesses with the flexibility they need to deploy and manage Oracle applications on Google Cloud, accelerating migration, modernization and innovation. A key part of that is support for containerization, which gives development teams high levels of agility and scalability. Whether your Oracle-based application is built with Java, Python, .NET, or runs on WebLogic, containers provide a consistent and isolated environment, abstracting away the underlying complexities of programming languages and frameworks.
Google Cloud offers a wide range of container-based runtimes for you to choose from.
Google Kubernetes Engine (GKE) is a powerful and versatile container orchestration platform that simplifies the deployment and management of diverse workloads on Google Cloud. Designed for businesses running complex setups and microservices architectures, GKE leverages Google’s global network and high-performance VMs to provide an optimized infrastructure for running containers smoothly and reliably. It automatically adjusts resources to meet real-time demand, optimizing both cost and performance through autoscaling and node auto-provisioning. Moreover, GKE’s broad support for container runtimes and advanced networking capabilities helps it efficiently handle everything from basic web applications to demanding machine learning workloads, written in any language.
Then there’s Cloud Run, a fully managed serverless platform for deploying containerized applications. Designed for applications that need to scale automatically based on demand, you simply provide your code or container image and Cloud Run takes care of the rest. It automatically scales resources up or down in response to traffic, meaning you only pay when your application is actively processing requests. By eliminating server management tasks like provisioning and scaling, Cloud Run frees you to concentrate on building and deploying applications, making it a good fit for microservices, APIs, web apps, and event-driven functions. Further, Cloud Run’s automatic base image updates helps your containers benefit from the latest security patches and operating system upgrades, minimizing vulnerabilities and significantly reducing maintenance overhead.
Best of all, containerizing your Oracle based applications with open standards means you don’t have to choose your path upfront. Don’t want to manage Kubernetes? Start with Cloud Run! Need more flexibility later on? Easily move to GKE. Because both services leverage standard OCI containers and are supported by CNCF, you have full interoperability and freedom of deployment choice for your containerized apps, now and in the future.
Running a containerized Oracle Database
Running your Oracle workload on Exadata within Google Cloud is the gold standard; however if that is not possible then you may want to explore running Oracle on GKE:
Running Oracle Database on GKE is suitable for non-RAC and non-Exadata workloads, or when Exadata is not supported in a given region.
This is a good solution for businesses that use a DBaaS strategy and have many Oracle Databases that can be lifecycled (deployed, updated, removed) using OraOperator.
Designed for integration within modern DevOps workflows and continuous deployment strategies, this approach is useful if you need to spin up or tear down databases quickly.
In this context, the afore-mentioned OraOperator provides multi-tenancy support. This is a valuable feature for running Oracle databases on GKE, allowing multiple isolated databases to run on the same GKE cluster, helping to improve resource utilization and reduce costs.
Turbocharging your Java applications
Many Oracle applications are written in Java. Oracle GraalVM creates native Java applications. This tool uses Ahead-of-Time (AOT) compilation, converting Java bytecode directly into native executables. By removing the reliance on a Java Virtual Machine (JVM) during execution, these applications start much faster and consume less resources (CPU and memory). GraalVM essentially allows Java to behave like languages such as C or C++, producing standalone executables that are perfect for resource-constrained environments like microservices and serverless functions in cloud-native deployments.
If you want to run your Oracle app on GKE, we highly recommend using a Java native image, which allows for highly optimized resource utilization compared with running a Java application from within a VM. That’s because native images have a smaller footprint, allowing you to run more pods per worker node, maximizing your GKE cluster resources and potentially reducing costs. Reduced pod startup latency also contributes to faster rollout and recovery times for deployments, improving the agility of your application.
And finally, using a Java native image enhances the security posture of your containers within the GKE cluster through the use of features such as:
advanced vulnerability insights, which provides scanning and vulnerability detection in Java, Go, Javascript, and Python language packages of your running containers
vulnerability scanning built into Artifact Registry, using a sandboxed environment to provide a secure and isolated environment for your Java native images
minimal base images that only include necessary dependencies in your Java native image, reducing the potential attack surface and minimizing the risk of vulnerabilities
Meanwhile, running your Java native application on Cloud Run provides near-instantaneous startup times, minimizing latency for initial requests and improving overall responsiveness. The Cloud Run startup CPU boost feature provides additional CPU duringinstance startup time and for 10 seconds after the instance has started.
In addition, Cloud Run is well-suited for event-driven applications, where functions are triggered by events. This is a common use case for native Java images, which, with their lack of a Java Runtime Environment, can respond to events quickly.
Finally, many organizations want to transition to Java 21. This adoption is largely motivated by the advantages of its LTS (long term support) designation which guarantees ongoing updates, security patches, increased runtime performance, new features and technical support for an extended period. Migrating older Oracle applications to GKE or Cloud Run is a good opportunity to modernize them to Java 21 at the same time.
Support from our partners
Moving your Oracle databases and applications to Google Cloud is a big task but you don’t have to go it alone. Google Cloud has partnered with experienced Oracle experts to help make your migration smooth and successful. These partners bring deep knowledge of Oracle technologies and proven methodologies to guide you through the process. Reach out to our ecosystem of partners who are available to help accelerate the success of your Oracle workloads on Google Cloud.
As the adoption of container workloads increases, so does the need to establish and maintain a consistent, strong Kubernetes security posture. Failing to do so can have significant consequences for the risk posture of an organization. Nearly 50% of organizations experienced revenue or customer loss due to container and Kubernetes security incidents, according to the 2024 State of Kubernetes Security Report.
Org policies are your friend to help you achieve pervasive security across your cloud infrastructure. In particular, you can use custom Organization Policies to enforce many of the CIS Benchmarks proactively, ensuring that you’ve established proper guardrails for Google Kubernetes Engine (GKE) standard and autopilot in your infrastructure.
Importantly for GKE security, custom Organization Policies can be used to enforce compliance with the CIS Benchmark for GKE.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud security products’), (‘body’, <wagtail.rich_text.RichText object at 0x3ea532ccca30>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
What is CIS Benchmark for GKE?
The CIS Benchmark is a comprehensive set of security recommendations and best practices designed to enhance the security posture of GKE clusters. It provides a list of recommendations to assess and mitigate potential vulnerabilities, covering areas such as authentication and authorization, network security, and IAM.
Using the CIS benchmark can help with two crucial tasks: reducing the risk of cyberattacks and ensuring compliance with industry standards. Custom Organization Policy supports the Cluster and Nodepool resources and many CIS recommendations for GKE can be enforced.
Achieving compliance with custom organization policies
Custom Organization Policies can help you enforce your own granular security and compliance controls. They allow you to define constraints and policies using Common Expression Language (CEL) to restrict specific configurations and actions within your cloud environment.
Using custom Organization Policies can ensure that new and existing GKE clusters adhere to security standards. These policies act as preventative controls, applied at the Google Cloud level, and can be integrated with any provisioning tool. Doing so can help provide a consistent and automated approach to security enforcement.
Custom Organization Policies also support safe rollout features including dry run and simulation, which allow organizations to test and ensure policy changes don’t disrupt operations before enforcing them in a production environment.
Some of the critical CIS recommendations for GKE you can enforce using custom Organization Policies include:
Making sure only private clusters using private endpoints and nodes can be provisioned.
Nodes are configured with Secure Boot enabled.
Container-Optimized OS is used for nodes.
Here are some examples of how to implement custom Organization Policies for GKE:
Enforcing usage of private clusters
code_block
<ListValue: [StructValue([(‘code’, ‘name: organizations/<ORG_ID>/customConstraints/custom.gkeRequirePrivateNodesrnresource_types:rn- container.googleapis.com/Clusterrncondition: resource.privateClusterConfig.enablePrivateNodes == falsernaction_type: DENYrnmethod_types:rn- CREATErn- UPDATErndisplay_name: Require GKE private nodesrndescription: Enforce that GKE clusters are created as private clusters with private nodes’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3ea535d58250>)])]>
Ensuring nodes are configured to use Container-Optimized OS
code_block
<ListValue: [StructValue([(‘code’, ‘name: organizations/<ORG_ID>/customConstraints/custom.gkeRequireCOSImagernresource_types:rn- container.googleapis.com/NodePoolrncondition: resource.config.imageType != “COS_CONTAINERD”rnaction_type: DENYrnmethod_types:rn- CREATErn- UPDATErndisplay_name: Require Container-Optimized OS on node poolsrndescription: Enforce the nodes pool are using Container-Optimized OS for running containers’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3ea535d58d30>)])]>
Custom Organization Policies can help organizations to incorporate security at the infrastructure’s base layer. This proactive approach prevents misconfigurations and vulnerabilities much earlier, reducing security risks and remediation costs.
Simplify onboarding with custom organization policy library
To simplify the adoption of custom Organization Policies, we have developed a library of policies accessible on the Google Cloud Professional Services GitHub public repository. The library already provides around 80 ready-to-use policies that translate security and compliance recommendations into actionable controls for a Google Cloud environment.
The library includes recommendations from the CIS Benchmark for GKE described above, with more than 30 controls already available. With this library, organizations can quickly and efficiently adopt security best practices using custom Organization Policies. Here are some key features and benefits of this library:
The library provides a starting point to gain inspiration from and can help make it easy to add policies to meet security and compliance standards. Policies can further be customized to meet your unique needs.
These policies can be integrated with your provisioning tools, enabling automated enforcement of security best practices. Integration with both gcloud and Terraform by using Cloud Foundation Fabric modules.
The library will continue to grow to include new policies and enhancements based on the new services that will be compatible with custom organization policies. They already supports more than 30 Google Cloud services
How to get started
By using custom organization policies, you can establish a robust security posture and mitigate potential vulnerabilities more easily than before. The custom Organization Policy library available on GitHub provides a great start for any organization to implement compliance and security controls for both GKE Standard and Autopilot, but also for many other services.
We encourage you to explore the GitHub policy library repository and use the power of custom Organization Policies to enforce security controls in your organization.
Platform engineering, one of Gartner’s top 10 strategic technology trends for 2024, is rapidly becoming indispensable for enterprises seeking to accelerate software delivery and improve developer productivity. How does it do that? Platform engineering is about providing the right infrastructure, tools, and processes that enable efficient, scalable software development, deployment, and management, all while minimizing the cognitive burden on developers.
To uncover the secrets to platform engineering success, Google Cloud partnered with Enterprise Strategy Group (ESG) on a comprehensive research study of 500 global IT professionals and application developers working at organizations with at least 500 employees, all with formal platform engineering teams. Our goal was to understand whether they had adopted platform engineering, and if so, the impact that has had on their company’s software delivery capabilities.
The resulting report, “Building Competitive Edge With Platform Engineering: A Strategic Guide,” reveals common patterns, expectations, and actionable best practices for overcoming challenges and fully leveraging platform engineering. This blog post highlights some of the most powerful insights from this study.
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3ea535c97cd0>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
Platform engineering is no longer optional
The research confirms that platform engineering is no longer a nascent concept. 55% of the global organizations we invited to participate have already adopted platform engineering. Of those, 90% plan to expand its reach to more developers. Furthermore, 85% of companies using platform engineering report that their developers rely on the platform to succeed. These figures highlight that platform engineering is no longer just a trend; it’s becoming a vital strategy for organizations seeking to unlock the full potential of their cloud and IT investments and gain a competitive edge.
Figure 1: 55% of 900+ global organizations surveyed have adopted platform engineering
Three keys to platform engineering success
The report identifies three critical components that are central to the success of mature platform engineering leaders.
Fostering close collaboration between platform engineers and other teams to ensure alignment
Adopting a “platform as a product” approach, which involves treating the developer platform with a clear roadmap, communicated value, and tight feedback loops
Defining success by measuring performance through clear metrics such as deployment frequency, failure recovery time, and lead time for changes
It’s noteworthy that while many organizations have begun their platform engineering journey, only 27% of adopters have fully integrated these three key components in their practices, signaling a significant opportunity for further improvements.
AI: platform engineering’s new partner
One of the most compelling insights of this report is the synergistic relationship between platform engineering and AI. A remarkable 86% of respondents believe that platform engineering is essential to realizing the full business value of AI. At the same time, a vast majority of companies view AI as a catalyst for advancing platform engineering, with 94% of organizations identifying AI to be ‘Critical’ or ‘Important’ to the future of platform engineering.
Beyond speed: key benefits of platform engineering
The study also identified three cohorts of platform engineering adopters — nascent, established, and leading — based on whether and how much adopters had embraced the above-mentioned three key components of platform engineering success. The study shows that leading adopters gain more in terms of speed, efficiency, and productivity, and offers guidance for nascent and established adopters to improve their overall platform engineering maturity to gain more benefits.
The report also identified some additional benefits of platform engineering, including:
Improved employee satisfaction, talent acquisition & retention: mature platforms foster a positive developer experience that directly impacts company culture. Developers and IT pros working for organizations with mature developer platforms are much more likely to recommend their workplace to their peers.
Accelerated time to market: mature platform engineering adopters have significantly shortened time to market. 71% of leading adopters of platform engineering indicated they have significantly accelerated their time to market, compared with 28% of less mature adopters.
Don’t go it alone
A vast majority (96%) of surveyed organizations are leveraging open-source tools to build their developer platforms. Moreover, most (84%) are partnering with external vendors to manage and support their open-source environments. Co-managed platforms with a third party or a cloud partner benefit from a higher degree of innovation. Organizations with co-managed platforms allocate an average of 47% of their developers’ productive time to innovation and experimentation, compared to just 38% for those that prefer to manage their platforms with internal staff.
Ready to succeed? Explore the full report
While this blog provides a glimpse into the key findings from this study, the full report goes much further, revealing key platform engineering strategies and practices that will help you stay ahead of the curve. Download the report to explore additional topics, including:
The strategic considerations of centralized and distributed platform engineering teams
The key drivers behind platform engineering investments
Top priorities driving platform adoption for developers, ensuring alignment with their needs
Key pain points to anticipate and navigate on the road to platform engineering success
How platform engineering boosts productivity, performance, and innovation across the entire organization
The strategic importance of open source in platform engineering for competitive advantage
The transformative role of platform engineering for AI/ML workloads as adoption of AI increases
How to develop the right platform engineering strategy to drive scalability and innovation
Technology has transformed our lives and social interactions at an unprecedented speed and scale, creating new opportunities. To adapt to this reality, L’Oréal has established itself as a leader in Beauty Tech, promoting personalized, inclusive, and responsible beauty accessible to all, under the banner “Beauty for Each, powered by Beauty Tech.”
This convergence of Beauty Tech is evident in augmented beauty products, smart devices, enhanced marketing, online and offline services, and digital platforms, all powered by information and communication technologies, data, and artificial intelligence. L’Oréal is committed to developing innovative solutions that elevate the beauty experience and contribute to a future where beauty is accessible, sustainable, and caters to the diverse needs and aspirations of individuals worldwide.
L’Oréal, the world’s largest cosmetics company, has for years leveraged AI to enhance digital solutions for its employees and provide personalized experiences for customers. In this blog, we will describe how L’Oréal’s Tech Accelerator built a scalable and end-to-end MLOps platform using Google Cloud. This platform accelerates the deployment of AI models, enabling the team to rapidly innovate.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud AI and ML’), (‘body’, <wagtail.rich_text.RichText object at 0x3ea5330a9790>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/vertex-ai/’), (‘image’, None)])]>
Our MLOps vision and requirements
To accelerate AI initiatives and optimize product development, L’Oréal Tech Accelerator sought to build a reusable, secure, and user-friendly Machine Learning Operations (MLOps) platform on Google Cloud. This platform aims to:
Streamline workflows and enhance collaboration, reducing friction between teams and accelerating time to market.
Ensure security and best practices that promote consistent, well-documented processes to minimize errors.
Enable rapid adoption through an intuitive platform that requires minimal training.
This approach fosters a more cohesive and efficient development environment, ultimately leading to higher quality products and greater agility in responding to evolving business needs.
Overview of L’Oréal’s MLOps platform
To understand the Tech Accelerator’s MLOps platform, let’s break down its key components. Here’s a simplified view of the process:
Labeled data preparation: Labeled data is gathered from various sources, including BigQuery, Google Cloud Storage, on-premise systems, and data lakes. It’s then processed and stored in a centralized location (such as BigQuery or Google Cloud Storage) to prepare it for training ML models
Training pipeline development: The team uses the Kubeflow SDK to define the flow and logic for the training pipeline. This pipeline automates the process of training the ML model.
Run training pipeline: The training pipeline is executed, generating a trained model artifact. This artifact is stored as a pickle file embedded in a Python library for easy access and deployment.
Prediction pipeline development: Using the Kubeflow SDK again, the team creates a prediction pipeline that utilizes the trained model to generate inferences on new data.
Run predict pipeline: The prediction pipeline is executed, generating inferences that are stored in BigQuery, Google Cloud Storage, or a data lake.
Validate trained model: The inference results from the prediction pipeline are used to evaluate the performance of the trained model. This involves calculating key accuracy metrics like F1-score and precision.
Push prediction pipeline to production: Up to this point, all pipeline components have been developed, tested, and validated manually by data scientists or ML engineers, and a new model version (or versions) has been created. The next step is to push the new version of the prediction pipeline, incorporating the new model version, to production. This deployment leverages development best practices, such as CI/CD pipelines.
The MLOps platform leverages DevOps principles to ensure a robust and efficient development lifecycle. This involves separating the ML development process into four distinct environments (i.e. Google Cloud projects):
DataOps: This environment provides a centralized repository for storing and managing all data assets, including labelled training data, model artifacts, and pipeline components. This ensures data consistency and accessibility throughout the ML workflow. Additionally, in this environment, the training pipeline is run to create the new version of the models.
Development: A dedicated space for testing new versions of prediction pipelines orchestrating multiple models. This environment allows for evaluating computation speeds, data coherence, end-to-end integration, and other performance aspects.
Staging: This environment mirrors the production setup, enabling rigorous testing and validation of the business expectations and requirements. By using staging data that closely resembles real-world data, potential business issues of prediction pipelines can be identified and addressed early on.
Production: The live environment where validated prediction pipelines and new versions of the models are deployed to generate real-time/batch predictions for L’Oréal’s Tech Accelerator applications and services, delivering value to end-users.
This structured approach, with its clear separation of environments, promotes efficient collaboration, minimizes risks, and ensures a smooth transition from development to production, ultimately enabling L’Oréal’s Tech Accelerator to deliver high-quality AI-powered beauty experiences. Note that, to further optimize efficiency and reduce costs, the training pipeline is executed only once within the DataOps environment. The resulting trained model is then deployed across the other environments. This eliminates the need to retrain the model in each environment, resulting in a significant cost reduction (up to 3x).
The figure above illustrates the relationship among the multiple environments and the required infrastructure. Notable points:
Model training pipelines output Python packages that embed the trained models.
The CI/CD pipeline outputs Kubeflow Pipelines (KFP) pipeline definitions and Docker images related to their components.
There are two distinct operational blocks: “Training” for creating new models and “Inference” for generating predictions.
Diving Deeper: Key Components of the MLOps Platform
At the core of the platform’s operation is KFP. To understand its role, let’s define what Vertex pipelines are:
“A pipeline is a definition of a workflow that composes one or more components together to form a computational directed acyclic graph (DAG). At runtime, each component execution corresponds to a single container execution, which may create ML artifacts. Pipelines may also feature control flow.” — Kubeflow Documentation
In this section, we’ll focus on how L’Oréal Tech Accelerator builds and manages the two main operational building blocks: “Training” and “Inference.”
Training pipeline
The training pipeline architecture is designed for efficiency and reproducibility. Here’s how it works:
Pipeline Definition and Components: The pipeline’s definition is fetched from a KFP artifact registry, while the container images that execute individual pipeline steps are retrieved from a Docker artifact registry. These artifacts are created and managed by a CI/CD pipeline, ensuring version control and consistency (as described in the “MLOps platform overview” section).
Model Training and Packaging: Once a new training pipeline run completes, the newly trained model is packaged into a Python library for easy deployment and integration.
Model Registry: This packaged model is then pushed to a Python artifact registry, creating a centralized repository of trained models. This allows for easy versioning, sharing, and deployment of models across different environments.
Inference pipeline
The inference pipeline follows a similar architecture to the training pipeline, ensuring consistency and efficiency in model deployment. Here’s how it works:
Pipeline Definition and Components: The inference pipeline’s definition, defined using KFP, is retrieved from an artifact registry. Similarly, the Docker images containing the necessary components for the pipeline are fetched from another artifact registry.
CI/CD Integration: These pipeline definitions and Docker images are created and deployed by the CI/CD pipeline, ensuring that the inference pipeline is always up-to-date and uses the latest validated components.
The modularity and dependency challenge
Traditional ML pipelines often rely on a single, shared codebase for their definition. This can lead to challenges when multiple teams need to collaborate and contribute to the pipeline’s development. Having all these teams work on the same codebase can create friction and slow down the development process due to:
Merge conflicts: When multiple teams edit the same files simultaneously.
Integration challenges: Ensuring the different components developed by separate teams work together seamlessly.
Version control complexities: Managing different versions and updates of the pipeline.
Deployment bottlenecks: Coordinating deployments when different teams need to make changes.
For example (see figure above), if two teams are working on separate models (Model 1 and Model 2) within the same codebase, and one model’s pipeline fails, it can prevent the other model’s inference pipeline from running. This creates a single point of failure that can disrupt the entire system.
To address this, a more modular and independent approach to pipeline development is needed, where individual teams can work on their components without affecting others.
The figure above illustrates the ML pipeline definition and infrastructure for the example and issue explained previously
How we solved it
L’Oréal Tech Accelerator solution uses KFP artifact registry to enable a modular approach to pipeline development. This allows the creation of independent sub-pipelines, each with its own codebase and CI/CD pipeline. This separation offers significant benefits:
Independent Development: Teams can work autonomously on their sub-pipelines without interfering with each other’s progress or deployments. This reduces friction and accelerates development cycles.
Isolated Testing and Versioning: Each sub-pipeline can be tested and versioned independently, ensuring that changes in one component don’t inadvertently affect others.
Increased Agility: This modularity enables teams to quickly adapt and update their sub-pipelines without impacting the overall system.
Furthermore, L’Oréal’s Tech Accelerator introduces an additional codebase that acts as an orchestrator. This orchestrator assembles the individual sub-pipelines into a cohesive workflow, using the output artifacts of each sub-pipeline as building blocks. This approach combines the benefits of independent development with the power of a unified pipeline.
The figure above illustrates the ML pipeline definition and infrastructure of the solution
Example: Code snippet of an aggregated pipeline
The following code snippet demonstrates the simplicity of using an aggregation module to combine multiple prediction pipelines and models from different teams. This orchestration layer allows for seamless integration of individual components into a unified workflow.
L’Oréal’s modular MLOps platform, built on Google Cloud, has significantly boosted efficiency and agility in the AI development process. By empowering teams to work independently on their respective ML models, L’Oréal’s Tech Accelerator has accelerated development, improved collaboration, and enhanced the quality and reliability of its systems.
While the current platform offers significant advantages, the team continues to optimize it.
One focus area is addressing the challenges of large model artifacts, which can increase Docker image sizes and slow down pipelines. L’Oréal’s Tech Accelerator is exploring solutions like on-demand model downloading and API-driven inference to mitigate this and remain at the forefront of Beauty Tech innovation.
The authors would like to thank and acknowledge the following contributors to this blog: Kerebel Paul-Sirawit, DevOps & Cloud Lead, L’Oréal, Dr. Sokratis Kartakis, Generative AI Blackbelt, Google, Christophe Dubos, Principal Architect, Google. Opening image credits: Ben Hassett / Myrtille Revemont / Helena Rubinstein pour L’Oréal
Cloud Deploy is a fully managed continuous delivery platform that automates the delivery of your application. On top of existing automation features, customers tell us they want other ways to automate their deployments to keep their production environments reliable and up to date.
We’re happy to announce three new features to help with that, all in preview.
1. Repair rollouts
The new repair rollout automation rule lets you retry failed deployments or automatically roll back to a previously successful release when an error occurs. These errors could come in any phase of a deployment: a pre-deployment SQL migration, a misconfiguration detected when talking to a GKE cluster, or as part of a deployment verification step. In any of these cases, the repair rollout automation lets you retry the failed step a configurable number of times, perfect for those occasionally flaky end-to-end tests. If the retry succeeds, the rollout continues. If the retries fail (or none are configured) the repair rollout automation can also roll back to the previously successful release.
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3eeb4c05a550>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
2. Deploy policies
Automating deployments is powerful, but it can also be important to put some constraints on the automation. The new deploy policies feature is intended to limit what these automations (or users) can do. Initially, we’re launching a time-windows policy which can, for example, inhibit deployments during evenings, weekends, or during important events. While an on-caller with the Policy Overrider role could “break glass” to get around these policies, automated deployments won’t be able to trigger a rollout in the middle of your big demo.
3. Timed promotions
After a release is successfully rolled out, you may want to automatically deploy it to the next environment. Our previous auto-promote feature let you promote a release after a specified duration, for example moving it into prod 12 hours after it went to staging. But often you want promotions to happen on a schedule, not based on a delay. Within Google, for example, we typically recommend that teams promote from a dev environment into staging every Thursday, and then start a promotion into prod on Monday mornings. With the new timed promotion automation, Cloud Deploy can handle these scheduled promotions for you.
The future
Comprehensive, easy-to-use, and cost-effective DevOps tools are key to efficient software delivery, and it’s our hope that Cloud Deploy will help you implement complete CI/CD pipelines. Stay tuned as we introduce exciting new capabilities and features to Cloud Deploy in the months to come.
Employees love Chrome extensions because they boost productivity, streamline workflows and let them customize their browser. Chrome already helps IT and security teams manage and control extensions, and we’re excited to announce powerful new features designed to give businesses greater control and visibility over their Chrome extension ecosystem.
Introducing the Chrome Web Store for enterprises
Say hello to a curated Chrome Web Store experience for your end users! This new Chrome Web Store benefits organizations through:
Simplified Access: Employees can easily discover and install pre-approved extensions directly from the Chrome Web Store landing page.
Enhanced Security: Promote trusted extensions and minimize the risk of users installing potentially harmful or unvetted add-ons.
Increased Productivity: Equip your teams with the specific extension they need to be more efficient and excel in their roles. Plus make it easier than ever for them to find approved extensions with blocked tags and filtering for private extensions directly in through the search bar.
Customizable Interface: Admins gain greater control over the Web Store experience with the ability to tailor per OU/organization. Now admins can add company logos, create custom imagery and announcements, curate extension collections, implement category-based controls and add new collections for allowlisted items.
More Transparency: Coming early this year, admins will be able to create custom block messages on extension detail pages for more visibility into usage policies potentially resulting in less end user requests to IT.
Admins can easily configure the new Web Store experience through Chrome Enterprise Core, which is available to all organizations for no added cost.
Better visibility into risk score
Understanding the potential risks associated with extensions is crucial for maintaining a secure browsing environment. Third party provided risk scores by spin.AI are now easier to find directly in the Extension Usage Report. It allows admins to quickly sort through and identify any extensions with high scores and block them from your environment. Admins can also now export these scores using the CSV export or the Management API. With this new risk scores reporting feature, administrators gain valuable insights into the security posture of extensions within their organization.
Enhanced Control with New Extension Settings
Maintaining a secure and optimized browsing experience requires granular control. That’s why Later this year, we’re introducing a new Chrome Enterprise Core setting that empowers administrators to remotely remove extensions. Admins have been able to install, allow and block extensions but now they can effortlessly remove extensions from end-users’ browsers, ensuring immediate mitigation of potential security threats. This also blocks any future installs for the extension.
The Future of Enterprise Extensions
These new enhancements mark a significant step forward in enterprise extension management. By providing greater control, visibility, and security, Google is committed to empowering businesses to harness the full potential of Chrome extensions while safeguarding their critical assets.
Stay tuned for further updates and explore these exciting new capabilities to elevate your organization’s digital workspace! Learn more about managing Chrome Enterprise using Chrome Enterprise Core here.
Organizations continue to adopt the public cloud to deliver better business and IT outcomes. However, migration — and modernization — is a complex, multifaceted challenge. It involves understanding the current state of infrastructure and applications to determine what the future state should look like and how to get there. And all this occurs across multiple stakeholders using myriad tools, processes, and implementation partners. But wait, don’t let that scare you off – we have good news, too!
Google Cloud Migration Center, a unified platform to help accelerate migration from current on-premises or cloud environments to Google Cloud, is designed to streamline your cloud journey with intelligent, data-driven insights and actionable recommendations. This will help you make critical decisions on the optimal migration and modernization pathways for your organization.
Migration Center lets you discover and assess both servers and databases, which are critical components of your IT infrastructure. In this blog we will focus on databases discovery and assessment journey within Migration Center.
The databases discovery and assessment journey consists of three main phases:
Discovery: Collect database configuration (metadata only) using open-source collection scripts. Supported databases are Microsoft SQL Server, MySQL and PostgreSQL.
Technical fit assessment: Evaluate technical fit of source databases to Cloud SQL and/or AlloyDB along with a detailed technical report.
Cost estimate: Assess overall costs of running source databases in Cloud SQL or AlloyDB in different migration scenarios (e.g., in various regions, using multiple vs. single zones, etc.) using various rightsizing algorithms. You can combine cost estimates of your server and database migration in a single report to create a holistic estimate of your infrastructure migration to Google Cloud.
Let’s look at each of these in greater depth.
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3ee01836f190>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
Discovery and collection of source database details
Database configuration and metrics collection Collecting Microsoft SQL Server, MySQL and PostgreSQL source database configurations is executed using open-source collection scripts.
You can run the script in your source system either in the database server directly or from a remote server that has access to the database. Here is an example for running the script from a bastion/jump server:
Upon completion the script automatically creates an archive of the extracted configuration and metrics (one ZIP file per database instance) that you can upload into Migration Center.
Note: If you are running the script from a bastion/Jump server and not directly the database server, then pass the Parameter “–vmUserName” to collect the host’s specification details.
Uploading discovery artifacts to Migration Center
Once you’ve generated the output files, you can upload them to Migration Center to perform the product assessment by following these steps:
Open Google Cloud console > MIgration Center home page. Go to Discovery > Data Import > Add Files > Upload Files
In the next step, choose the file format as “Database Import File” and then upload all your DMA output zip files and click on Upload Files.
You can now review the uploaded database deployments in Migration Center Assets section.
Technical fit assessment
You can find all the discovered servers and databases in the Assets section of Migration Center. The list of database deployments surfaces high-level details of discovered databases along with a high-level fit score for Cloud SQL and AlloyDB.
To review a detailed configuration, click on the source databases you want to assess, and the available data: source logical database configurations, hosting server configurations, performance data, feasible cloud products, the fit score, and target instance sizing.
You can also review the technical fit report by clicking on the fit score (e.g., fit with effort), which lists unsupported features/parameters/flags that were identified in the source databases and that will need to be mitigated when migrating to Cloud SQL or AlloyDB.
Cost estimates
Assessing the cost of running servers and databases in the cloud in Migration Center is a three-step process:
Group together the servers and/or databases (in one or multiple groups) to be assessed.
Create one or more migration preferences that will serve as migration scenarios to assess.
Create a TCO report by assigning groups to up to four preferences.
1. Create group
To create the pricing report, you need to create a group using the assets you want to evaluate for cost estimates. Go to Discovery -> Groups -> Create Group.
Select the relevant servers and/or databases that you would like to assess as a group. We recommend creating groups that categorize your source environment (e.g., applications, environments, etc.).
2. Create migration preferences
Migration preferences are basically a set of parameters that you can define to represent a migration scenario. You can create multiple migration preferences and use them to generate “what if” scenarios in TCO reports to compare costs for the servers and databases in the cloud. For example, you can create migration preferences for different editions of Cloud SQL with options for high availability and backups enabled.
To create a migration preference, go to Migration Center > Discovery > Migration Preferences > Create MIgration Preference.
3. Create a TCO report for your server and database workloads
Go to Reports -> Create Reports -> TCO and detailed pricing report to start creating the report.
Select your Group/s of assets to assess.
Assign up to four migration preferences to each group as an input.
In the TCO report you can review source environment details, cost estimates for servers and databases, and export the report in Google Slides or CSV formats. Below is an example of the databases cost estimates:
With Migration Center, our goal is to put the information you need when doing a migration and modernization project at your fingertips, so you can reduce the time, energy, and cost of these projects. As you navigate your journey to the cloud, we’re here to help with Migration Center. Start using Migration Center today, or to learn more, check out the web pages for Migration Center and Rapid Discovery and Assessment Program (RAMP).
Does your organization use multiple data processing engines like BigQuery, Apache Spark, Apache Flink and Apache Hive? Wouldn’t it be great if you could provide a single source of truth for all of your analytics workloads? Now you can, with the public preview of BigQuery metastore, a fully managed, unified metadata service that provides processing engine interoperability while enabling consistent data governance.
BigQuery metastore is a highly scalable runtime metadata service that works with multiple engines, for example, BigQuery, Apache Spark, Apache Hive and Apache Flink, and supports the open Apache Iceberg table format. This allows analytics engines to query one copy of the data with a single schema, whether the data is stored in BigQuery storage tables, BigQuery tables for Apache Iceberg, or BigLake external tables. BigQuery metastore serves as a critical component for customers looking to migrate and modernize from legacy data lakes to a modern lakehouse architecture. Integrated deeply with BigQuery’s enterprise capabilities, this solution provides built-in security and governance for user interactions with data.
The challenges of metadata management
Traditionally, metastores and other metadata management systems are tightly coupled with data processing engines. If you are using multiple processing engines, that means maintaining multiple copies of the data and metadata persisted in different metastores. For example, when you create a table definition in Hive Metastore for querying from an open-source engine like Spark, you have to recreate the table definition to query the same data in BigQuery. You also have to build pipelines to keep table definitions synchronized across different metastores. This fragmentation can result in stale metadata, lack of visibility into data lineage, security and access challenges, and a subpar user experience.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud data analytics’), (‘body’, <wagtail.rich_text.RichText object at 0x3edff9e969a0>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/bigquery/’), (‘image’, None)])]>
A metastore for the lakehouse era
BigQuery metastore is designed for the lakehouse architecture, whichcombines the benefits of data lakes and data warehouseswithout having to manage both a data lake and a data warehouse — any data, any user, any workload, on a unified platform. It supports open data formats such as Apache Icebergthat are accessible by a variety of processing engines, including BigQuery, Spark, Flink and Hive. The unification of metadata across engines makes it easier to discover and use data, supporting self-service BI and ML tools to drive innovation, while maintaining data governance.
Furthermore, BigQuery metastore is serverless with no setup or configuration required and automatically scales with your workloads. This no-opsenvironment reduces TCO and democratizes your data for data analysts, data engineers and data scientists.
Key benefits of BigQuery metastore include:
Cross-engine interoperability: BigQuery metastore provides a single shared metastore for the lakehouse architecture, with a unified view of all metadata for all data sources in the lakehouse, making it easy for your users to find and understand the data they need. This enables query processing and DML for data stored in open and proprietary formats across object stores, BigQuery storage, and across analytics runtimes.
Support for open formats and catalogs: BigQuery metastore provides support for BigQuery storage tables, BigQuery tables for Apache Iceberg and external tables.
Built-in governance: BigQuery metastore is integrated with key governance capabilities provided in BigQuery, such as automated cataloging and universal search, business metadata, data profiling, data quality, fine-grained access controls, data masking, sharing, data lineage and audit logging.
Fully managed at BigQuery scale: Being a serverless, fully managed service, BigQuery metastore is very easy to use and has integrations with key engines (BigQuery, Spark, Hive and Flink). The infrastructure foundation used for BigQuery metastore ensures that it scales to the growing query processing volume of your application and can handle traffic at BigQuery scale.
BigQuery metastore in action
Now, let’s take a look at how to use BigQuery metastore. The PySpark script below sets up a Spark environment to interact with a BigQuery storage table, a BigQuery table for Apache Iceberg, and a BigQuery external table. Detailed documentation is provided here.
code_block
<ListValue: [StructValue([(‘code’, ‘from pyspark.sql import SparkSessionrnrnrn# Create a spark sessionrnspark = SparkSession.builder \rn.appName(“BigQuery Metastore Iceberg”) \rn.config(“spark.sql.catalog.CATALOG_NAME”, “org.apache.iceberg.spark.SparkCatalog”) \rn.config(“spark.sql.catalog.CATALOG_NAME.catalog-impl”, “org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog”) \rn.config(“spark.sql.catalog.CATALOG_NAME.gcp_project”, “PROJECT_ID”) \rn.config(“spark.sql.catalog.CATALOG_NAME.gcp_location”, “LOCATION”) \rn.config(“spark.sql.catalog.CATALOG_NAME.warehouse”, “WAREHOUSE_DIRECTORY”) \rn.getOrCreate()rnspark.conf.set(“viewsEnabled”,”true”)rnrnrn# Use the CATALOG_NAMErnspark.sql(“USE `CATALOG_NAME`;”)rnspark.sql(“USE NAMESPACE DATASET_NAME;”)rnrnrn# Configure spark for temp resultsrnspark.sql(“CREATE NAMESPACE IF NOT EXISTS MATERIALIZATION_NAMESPACE”);rnspark.conf.set(“materializationDataset”,”MATERIALIZATION_NAMESPACE”)rnrnrn# List the tables in the datasetrndf = spark.sql(“SHOW TABLES;”)rndf.show();rnrnrn# Query a BigQuery storage tablernsql = “””SELECT * FROM DATASET_NAME.TABLE_NAME”””rndf = spark.read.format(“bigquery”).load(sql)rndf.show()rnrnrn# Query a BigQuery table for Apache Icebergrnsql = “””SELECT * FROM DATASET_NAME.ICEBERG_TABLE_NAME”””rndf = spark.read.format(“bigquery”).load(sql)rndf.show()rnrnrn# Query a BigQuery read-only Apache Iceberg external tablernsql = “””SELECT * FROM DATASET_NAME.READONLY_ICEBERG_TABLE_NAME”””rndf = spark.read.format(“bigquery”).load(sql)rndf.show()’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3edff9e96b20>)])]>
To customize this script for your environment, simply replace the following variables:
WAREHOUSE_DIRECTORY: the URI of the Cloud Storage folder that contains your data warehouse
CATALOG_NAME: the name of the catalog that you’re using
MATERIALIZATION_NAMESPACE: the namespace for storing temporary results
Learn more
With the BigQuery metastore, you now have a modern, serverless solution to meet your metadata management needs, enabling cross-engine interoperability with built-in governance. To try out BigQuery metastore today, see the documentation. If you would like to migrate from Dataproc Metastore to BigQuery metastore, see the documentation on migration tooling.