gcp

As AI use increases, security remains a top concern, and we often hear that organizations are worried about risks that can come with rapid adoption. Google Cloud is committed to helping our customers confidently build and deploy AI in a secure, compliant, and private manner.

Today, we’re introducing a new solution that can help you mitigate risk throughout the AI lifecycle. We are excited to announce AI Protection, a set of capabilities designed to safeguard AI workloads and data across clouds and models — irrespective of the platforms you choose to use. 

AI Protection helps teams comprehensively manage AI risk by:

1. Discovering AI inventory in your environment and assessing it for potential vulnerabilities
2. Securing AI assets with controls, policies, and guardrails
3. Managing threats against AI systems with detection, investigation, and response capabilities

AI Protection is integrated with Security Command Center (SCC), our multicloud risk-management platform, so that security teams can get a centralized view of their AI posture and manage AI risks holistically in context with their other cloud risks. 

Discovering AI inventory

Effective AI risk management begins with a comprehensive understanding of where and how AI is used within your environment. Our capabilities help you automatically discover and catalog AI assets, including the use of models, applications, and data — and their relationships. 

Understanding what data supports AI applications and how it’s currently protected is paramount. Sensitive Data Protection (SDP) now extends automated data discovery to Vertex AI datasets to help you understand data sensitivity and data types that make up training and tuning data. It can also generate data profiles that provide deeper insight into the type and sensitivity of your training data. 

Once you know where sensitive data exists, AI Protection can use Security Command Center’s virtual red teaming to identify AI-related toxic combinations and potential paths that threat actors could take to compromise this critical data, and recommend steps to remediate vulnerabilities and make posture adjustments.

Securing AI assets

Model Armor, a core capability of AI Protection, is now generally available. It guards against prompt injection, jailbreak, data loss, malicious URLs, and offensive content. Model Armor can support a broad range of models across multiple clouds, so customers get consistent protection for the models and platforms they want to use — even if that changes in the future.

Today, developers can easily integrate Model Armor’s prompt and response screening into applications using a REST API or through an integration with Apigee. The ability to deploy Model Armor in-line without making any app changes is coming soon through integrations with Vertex AI and our Cloud Networking products.

“We are using Model Armor not only because it provides robust protection against prompt injections, jailbreaks, and sensitive data leaks, but also because we’re getting a unified security posture from Security Command Center. We can quickly identify, prioritize, and respond to potential vulnerabilities — without impacting the experience of our development teams or the apps themselves. We view Model Armor as critical to safeguarding our AI applications and being able to centralize the monitoring of AI security threats alongside our other security findings within SCC is a game-changer,” said Jay DePaul, chief cybersecurity and technology risk officer, Dun & Bradstreet.

Organizations can use AI Protection to strengthen the security of Vertex AI applications by applying postures in Security Command Center. These posture controls, designed with first-party knowledge of the Vertex AI architecture, define secure resource configurations and help organizations prevent drift or unauthorized changes.

Managing AI threats 

AI Protection operationalizes security intelligence and research from Google and Mandiant to help defend your AI systems. Detectors in Security Command Center can be used to uncover initial access attempts, privilege escalation, and persistence attempts for AI workloads. New detectors to AI Protection based on the latest frontline intelligence to help identify and manage runtime threats such as foundational model hijacking are coming soon.

“As AI-driven solutions become increasingly commonplace, securing AI systems is paramount and surpasses basic data protection. AI security — by its nature — necessitates a holistic strategy that includes model integrity, data provenance, compliance, and robust governance,” said Dr. Grace Trinidad, research director, IDC. 

“Piecemeal solutions can leave and have left critical vulnerabilities exposed, rendering organizations susceptible to threats like adversarial attacks or data poisoning, and added to the overwhelm experienced by security teams. A comprehensive, lifecycle-focused approach allows organizations to effectively mitigate the multi-faceted risks surfaced by generative AI, as well as manage increasingly expanding security workloads. By taking a holistic approach to AI protection, Google Cloud simplifies and thus improves the experience of securing AI for customers,” she said.

Complement AI Protection with frontline expertise

The Mandiant AI Security Consulting Portfolio offers services to help organizations assess and implement robust security measures for AI systems across clouds and platforms. Consultants can evaluate the end-to-end security of AI implementations and recommend opportunities to harden AI systems. We also provide red teaming for AI, informed by the latest attacks on AI services seen in frontline engagements.

Building on a secure foundation

Customers can also benefit from using Google Cloud’s infrastructure for building and running AI workloads. Our secure-by-design, secure-by-default cloud platform is built with multiple layers of safeguards, encryption, and rigorous software supply chain controls.  

For customers whose AI workloads are subject to regulation, we offer Assured Workloads to easily create controlled environments with strict policy guardrails that enforce controls such as data residency and customer-managed encryption. Audit Manager can produce evidence of regulatory and emerging AI standards compliance. Confidential Computing can help ensure data remains protected throughout the entire processing pipeline, reducing the risk of unauthorized access, even by privileged users or malicious actors within the system. 

Additionally, for organizations looking to discover unsanctioned use of AI, or shadow AI, in their workforce, Chrome Enterprise Premium can provide visibility into end-user activity as well as prevent accidental and intentional exfiltration of sensitive data in gen AI applications.

Next steps

Google Cloud is committed to helping your organization protect its AI innovations. Read more in this showcase paper from Enterprise Strategy Group and attend our upcoming online Security Talks event on March 12. 

To evaluate AI Protection in Security Command Center and explore subscription options, please contact a Google Cloud sales representative or authorized Google Cloud partner.

More exciting capabilities are coming soon and we will be sharing in-depth details on AI Protection and how Google Cloud can help you securely develop and deploy AI solutions at Google Cloud Next in Las Vegas, April 9 to 11.

Written by: Chuong Dong

Overview

In our day-to-day work, the FLARE team often encounters malware written in Go that is protected using garble. While recent advancements in Go analysis from tools like IDA Pro have simplified the analysis process, garble presents a set of unique challenges, including stripped binaries, function name mangling, and encrypted strings.

Garble’s string encryption, while relatively straightforward, significantly hinders static analysis. In this blog post, we’ll detail garble’s string transformations and the process of automatically deobfuscating them.

We’re also introducing GoStringUngarbler, a command-line tool written in Python that automatically decrypts strings found in garble-obfuscated Go binaries. This tool can streamline the reverse engineering process by producing a deobfuscated binary with all strings recovered and shown in plain text, thereby simplifying static analysis, malware detection, and classification.

Garble Obfuscating Compiler

Before detailing the GoStringUngarbler tool, we want to briefly explain how the garble compiler modifies the build process of Go binaries. By wrapping around the official Go compiler, garble performs transformations on the source code during compilation through Abstract Syntax Tree (AST) manipulation using Go’s go/ast library. Here, the obfuscating compiler modifies program elements to obfuscate the produced binary while preserving the semantic integrity of the program. Once transformed by garble, the program’s AST is fed back into the Go compilation pipeline, producing an executable that is harder to reverse engineer and analyze statically. 

While garble can apply a variety of transformations to the source code, this blog post will focus on its “literal” transformations. When garble is executed with the -literals flag, it transforms all literal strings in the source code and imported Go libraries into an obfuscated form. Each string is encoded and wrapped behind a decrypting function, thwarting static string analysis. 

For each string, the obfuscating compiler can randomly apply one of the following literal transformations. We’ll explore each in greater detail in subsequent sections.

• Stack transformation: This method implements runtime encoding to strings stored directly on the stack.

• Seed transformation: This method employs a dynamic seed-based encryption mechanism where the seed value evolves with each encrypted byte, creating a chain of interdependent encryption operations.

• Split transformation: This method fragments the encrypted strings into multiple chunks, each to be decrypted independently in a block of a main switch statement.

Stack Transformation

The stack transformation in garble implements runtime encrypting techniques that operate directly on the stack, using three distinct transformation types: simple, swap, and shuffle. These names are taken directly from the garble’s source code. All three perform cryptographic operations with the string residing on the stack, but each differs in complexity and approach to data manipulation.

• Simple transformation: This transformation applies byte-by-byte encoding using a randomly generated mathematical operator and a randomly generated key of equal length to the input string.
• Swap transformation: This transformation applies a combination of byte-pair swapping and position-dependent encoding, where pairs of bytes are shuffled and encrypted using dynamically generated local keys.
• Shuffle transformation: This transformation applies multiple layers of encryption by encoding the data with random keys, interleaving the encrypted data with its keys, and applying a permutation with XOR-based index mapping to scatter the encrypted data and keys throughout the final output.

Simple Transformation

This transformation implements a straightforward byte-level encoding scheme at the AST level. The following is the implementation from the garble repository. In Figure 1 and subsequent code samples taken from the garble repository, comments were added by the author for readability.

// Generate a random key with the same length as the input string
key := make([]byte, len(data))

// Fill the key with random bytes
obfRand.Read(key)

// Select a random operator (XOR, ADD, SUB) to be used for encryption
op := randOperator(obfRand)

// Encrypt each byte of the data with the key using the random operator
for i, b := range key {
    data[i] = evalOperator(op, data[i], b)
}

Figure 1: Simple transformation implementation

The obfuscator begins by generating a random key of equal length to the input string. It then randomly selects a reversible arithmetic operator (XOR, addition, or subtraction) that will be used throughout the encoding process.

The obfuscation is performed by iterating through the data and key bytes simultaneously, applying the chosen operator between each corresponding pair to produce the encoded output.

Figure 2 shows the decompiled code produced by IDA of a decrypting subroutine of this transformation type.

Swap Transformation

The swap transformation uses a byte-shuffling and encryption algorithm to encrypt a string literal. Figure 3 shows its implementation from the garble repository.

// Determines how many swap operations to perform based on data length
func generateSwapCount(obfRand *mathrand.Rand, dataLen int) int {
    // Start with number of swaps equal to data length
    swapCount := dataLen

    // Calculate maximum additional swaps (half of data length)
    maxExtraPositions := dataLen / 2

    // Add a random amount if we can add extra positions 
    if maxExtraPositions > 1 {
        swapCount += obfRand.Intn(maxExtraPositions)
    }

    // Ensure swap count is even by incrementing if odd
    if swapCount%2 != 0 {
        swapCount++
    }
    return swapCount
}

func (swap) obfuscate(obfRand *mathrand.Rand, data []byte) 
*ast.BlockStmt {
    // Generate number of swap operations to perform
    swapCount := generateSwapCount(obfRand, len(data))

    // Generate a random shift key
    shiftKey := byte(obfRand.Uint32())

    // Select a random reversible operator for encryption
    op := randOperator(obfRand)

    // Generate list of random positions for swapping bytes
    positions := genRandIntSlice(obfRand, len(data), swapCount)

    // Process pairs of positions in reverse order
    for i := len(positions) - 2; i >= 0; i -= 2 {
        // Generate a position-dependent local key for each pair
        localKey := byte(i) + byte(positions[i]^positions[i+1]) + shiftKey
        
        // Perform swap and encryption:
        // - Swap positions[i] and positions[i+1]
        // - Encrypt the byte at each position with the local key
        data[positions[i]], data[positions[i+1]] = evalOperator(op, 
data[positions[i+1]], localKey), evalOperator(op, data[positions[i]], 
localKey)
    }
...

Figure 3: Swap transformation implementation

The transformation begins by generating an even number of random swap positions, which is determined based on the data length plus a random number of additional positions (limited to half the data length). The compiler then randomly generates a list of random swap positions with this length.

The core obfuscation process operates by iterating through pairs of positions in reverse order, performing both a swap operation and encryption on each pair. For each iteration, it generates a position-dependent local encryption key by combining the iteration index, the XOR result of the current position pair, and a random shift key. This local key is then used to encrypt the swapped bytes with a randomly selected reversible operator.

Figure 4 shows the decompiled code produced by IDA of a decrypting subroutine of the swap transformation.

Shuffle Transformation

The shuffle transformation is the most complicated of the three stack transformation types. Here, garble applies its obfuscation by encrypting the original string with random keys, interleaving the encrypted data with its keys, and scattering the encrypted data and keys throughout the final output. Figure 5 shows the implementation from the garble repository.

// Generate a random key with the same length as the original string
key := make([]byte, len(data))
obfRand.Read(key)

// Constants for the index key size bounds
const (
    minIdxKeySize = 2
    maxIdxKeySize = 16
)

// Initialize index key size to minimum value
idxKeySize := minIdxKeySize

// Potentially increase index key size based on input data length
if tmp := obfRand.Intn(len(data)); tmp > idxKeySize {
    idxKeySize = tmp
}

// Cap index key size at maximum value
if idxKeySize > maxIdxKeySize {
    idxKeySize = maxIdxKeySize
}

// Generate a secondary key (index key) for index scrambling
idxKey := make([]byte, idxKeySize)
obfRand.Read(idxKey)

// Create a buffer that will hold both the encrypted data and the key
fullData := make([]byte, len(data)+len(key))

// Generate random operators for each position in the full data buffer
operators := make([]token.Token, len(fullData))
for i := range operators {
    operators[i] = randOperator(obfRand)
}

// Encrypt data and store it with its corresponding key
// First half contains encrypted data, second half contains the key
for i, b := range key {
    fullData[i], fullData[i+len(data)] = evalOperator(operators[i], 
data[i], b), b
}

// Generate a random permutation of indices
shuffledIdxs := obfRand.Perm(len(fullData))

// Apply the permutation to scatter encrypted data and keys
shuffledFullData := make([]byte, len(fullData))
for i, b := range fullData {
    shuffledFullData[shuffledIdxs[i]] = b
}

// Prepare AST expressions for decryption
args := []ast.Expr{ast.NewIdent("data")}
for i := range data {
    // Select a random byte from the index key
    keyIdx := obfRand.Intn(idxKeySize)
    k := int(idxKey[keyIdx])
    
    // Build AST expression for decryption:
    // 1. Uses XOR with index key to find the real positions of data 
and key
    // 2. Applies reverse operator to decrypt the data using the 
corresponding key
    args = append(args, operatorToReversedBinaryExpr(
        operators[i],

        // Access encrypted data using XOR-ed index
        ah.IndexExpr("fullData", &ast.BinaryExpr{X: ah.IntLit(shuffledIdxs[i] 
^ k), Op: token.XOR, Y: ah.CallExprByName("int", ah.IndexExpr("idxKey", 
ah.IntLit(keyIdx)))}),
        
        // Access corresponding key using XOR-ed index
        ah.IndexExpr("fullData", &ast.BinaryExpr{X: 
ah.IntLit(shuffledIdxs[len(data)+i] ^ k), Op: token.XOR, Y: 
ah.CallExprByName("int", ah.IndexExpr("idxKey", ah.IntLit(keyIdx)))}),
    ))
}

Figure 5: Shuffle transformation implementation

Garble begins by generating two types of keys: a primary key of equal length to the input string for data encryption and a smaller index key (between two and 16 bytes) for index scrambling. The transformation process then occurs in the following four steps:

1. Initial encryption: Each byte of the input data is encrypted using a randomly generated reversible operator with its corresponding key byte.

2. Data interleaving: The encrypted data and key bytes are combined into a single buffer, with encrypted data in the first half and corresponding keys in the second half.

3. Index permutation: The key-data buffer undergoes a random permutation, scattering both the encrypted data and keys throughout the buffer.

4. Index encryption: Access to the permuted data is further obfuscated by XOR-ing the permuted indices with randomly selected bytes from the index key.

Figure 6 shows the decompiled code produced by IDA of a decrypting subroutine of the shuffle transformation.

Seed Transformation

The seed transformation implements a chained encoding scheme where each byte’s encryption depends on the previous encryptions through a continuously updated seed value. Figure 7 shows the implementation from the garble repository.

// Generate random initial seed value
seed := byte(obfRand.Uint32())

// Store original seed for later use in decryption
originalSeed := seed

// Select a random reversible operator for encryption
op := randOperator(obfRand)

var callExpr *ast.CallExpr

// Encrypt each byte while building chain of function calls
for i, b := range data {
   // Encrypt current byte using current seed value
   encB := evalOperator(op, b, seed)

   // Update seed by adding encrypted byte
   seed += encB

   if i == 0 {
       // Start function call chain with first encrypted byte
       callExpr = ah.CallExpr(ast.NewIdent("fnc"), ah.IntLit(int(encB)))
   } else {
       // Add subsequent encrypted bytes to function call chain
       callExpr = ah.CallExpr(callExpr, ah.IntLit(int(encB)))
   }
}
...

Figure 7: Seed transformation implementation

Garble begins by randomly generating a seed value to be used for encryption. As the compiler iterates through the input string, each byte is encrypted by applying the random operator with the current seed, and the seed is updated by adding the encrypted byte. In this seed transformation, each byte’s encryption depends on the result of the previous one, creating a chain of dependencies through the continuously updated seed.

In the decryption setup, as shown in the IDA decompiled code in Figure 8, the obfuscator generates a chain of calls to a decrypting function. For each encrypted byte starting with the first one, the decrypting function applies the operator to decrypt it with the current seed and updates the seed by adding the encrypted byte to it. Because of this setup, subroutines of this transformation type are easily recognizable in the decompiler and disassembly views due to the multiple function calls it makes in the decryption process.

Split Transformation

The split transformation is one of the more sophisticated string transformation techniques by garble, implementing a multilayered approach that combines data fragmentation, encryption, and control flow manipulation. Figure 10 shows the implementation from the garble repository.

func (split) obfuscate(obfRand *mathrand.Rand, data []byte) 
*ast.BlockStmt {
    var chunks [][]byte

    // For small input, split into single bytes
    // This ensures even small payloads get sufficient obfuscation
    if len(data)/maxChunkSize < minCaseCount {
        chunks = splitIntoOneByteChunks(data)
    } else {
        chunks = splitIntoRandomChunks(obfRand, data)
    }

    // Generate random indexes for all chunks plus two special cases:
    // - One for the final decryption operation
    // - One for the exit condition
    indexes := obfRand.Perm(len(chunks) + 2)

    // Initialize the decryption key with a random value
    decryptKeyInitial := byte(obfRand.Uint32())
    decryptKey := decryptKeyInitial
    
    // Calculate the final decryption key by XORing it with 
position-dependent values
    for i, index := range indexes[:len(indexes)-1] {
        decryptKey ^= byte(index * i)
    }

    // Select a random reversible operator for encryption
    op := randOperator(obfRand)

    // Encrypt all data chunks using the selected operator and key
    encryptChunks(chunks, op, decryptKey)

    // Get special indexes for decrypt and exit states
    decryptIndex := indexes[len(indexes)-2]
    exitIndex := indexes[len(indexes)-1]
    
    // Create the decrypt case that reassembles the data
    switchCases := []ast.Stmt{&ast.CaseClause{
        List: []ast.Expr{ah.IntLit(decryptIndex)},
        Body: shuffleStmts(obfRand,
            // Exit case: Set next state to exit
            &ast.AssignStmt{
                Lhs: []ast.Expr{ast.NewIdent("i")},
                Tok: token.ASSIGN,
                Rhs: []ast.Expr{ah.IntLit(exitIndex)},
            },
            // Iterate through the assembled data and decrypt each byte
            &ast.RangeStmt{
                Key: ast.NewIdent("y"),
                Tok: token.DEFINE,
                X:   ast.NewIdent("data"),
                Body: ah.BlockStmt(&ast.AssignStmt{
                    Lhs: []ast.Expr{ah.IndexExpr("data", ast.NewIdent("y"))},
                    Tok: token.ASSIGN,
                    Rhs: []ast.Expr{
                        // Apply the reverse of the encryption operation
                        operatorToReversedBinaryExpr(
                            op,
                            ah.IndexExpr("data", ast.NewIdent("y")),
                            // XOR with position-dependent key
                            ah.CallExpr(ast.NewIdent("byte"), &ast.BinaryExpr{
                                X:  ast.NewIdent("decryptKey"),
                                Op: token.XOR,
                                Y:  ast.NewIdent("y"),
                            }),
                        ),
                    },
                }),
            },
        ),
    }}

    // Create switch cases for each chunk of data
    for i := range chunks {
        index := indexes[i]
        nextIndex := indexes[i+1]
        chunk := chunks[i]

        appendCallExpr := &ast.CallExpr{
            Fun:  ast.NewIdent("append"),
            Args: []ast.Expr{ast.NewIdent("data")},
        }
	 ...
        // Create switch case for this chunk
        switchCases = append(switchCases, &ast.CaseClause{
            List: []ast.Expr{ah.IntLit(index)},
            Body: shuffleStmts(obfRand,
                // Set next state
                &ast.AssignStmt{
                    Lhs: []ast.Expr{ast.NewIdent("i")},
                    Tok: token.ASSIGN,
                    Rhs: []ast.Expr{ah.IntLit(nextIndex)},
                },
                // Append this chunk to the collected data
                &ast.AssignStmt{
                    Lhs: []ast.Expr{ast.NewIdent("data")},
                    Tok: token.ASSIGN,
                    Rhs: []ast.Expr{appendCallExpr},
                },
            ),
        })
    }

    // Final block creates the state machine loop structure
    return ah.BlockStmt(
        ...

        // Update decrypt key based on current state and counter
        Body: ah.BlockStmt(
            &ast.AssignStmt{
                Lhs: []ast.Expr{ast.NewIdent("decryptKey")},
                Tok: token.XOR_ASSIGN,
                Rhs: []ast.Expr{
                    &ast.BinaryExpr{
                        X:  ast.NewIdent("i"),
                        Op: token.MUL,
                        Y:  ast.NewIdent("counter"),
                    },
                },
            },
            // Main switch statement as the core of the state machine
            &ast.SwitchStmt{
                Tag:  ast.NewIdent("i"),
                Body: ah.BlockStmt(shuffleStmts(obfRand, switchCases...)...),
            }),

Figure 10: Split transformation implementation

The transformation begins by splitting the input string into chunks of varying sizes. Shorter strings are broken into individual bytes, while longer strings are divided into random-sized chunks of up to four bytes.

The transformation then constructs a decrypting mechanism using a switch-based control flow pattern. Rather than processing chunks sequentially, the compiler generates a randomized execution order through a series of switch cases. Each case handles a specific chunk of data, encrypting it with a position-dependent key derived from both the chunk’s position and a global encryption key.

In the decryption setup, as shown in the IDA decompiled code in Figure 11, the obfuscator first collects the encrypted data by going through each chunk in their corresponding order. In the final switch case, the compiler performs a final pass to XOR-decrypt the encrypted buffer. This pass uses a continuously updated key that depends on both the byte position and the execution path taken through the switch statement to decrypt each byte.

GoStringUngarbler: Automatic String Deobfuscator

To systematically approach string decryption automation, we first consider how this can be done manually. From our experience, the most efficient manual approach leverages dynamic analysis through a debugger. Upon finding a decrypting subroutine, we can manipulate the program counter to target the subroutine’s entry point, execute until the ret instruction, and extract the decrypted string from the return buffer.

To perform this process automatically, the primary challenge lies in identifying all decrypting subroutines introduced by garble’s transformations. Our analysis revealed a consistent pattern—decrypted strings are always processed through Go’s runtime_slicebytetostring function before being returned by the decrypting subroutine. This observation provides a reliable anchor point, allowing us to construct regular expression (regex) patterns to automatically detect these subroutines.

String Encryption Subroutine Patterns

Through analyzing the disassembled code, we have identified consistent instruction patterns for each string transformation variant. For each transformation on 64-bit binaries, rbx is used to store the decrypted string pointer, and rcx is assigned with the length of the decrypted string. The main difference between the transformations is the way these two registers are populated before the call to runtime_slicebytetostring.

# Go compiler v1.21 -> v1.23 (x64)

# Stack Transformation Epilogue Pattern 

48 8D 5C ?? ??     lea     rbx, [rsp+<num>]            # decrypted string pointer
B9 ?? ?? ?? ??     mov     ecx, <num>                  # decrypted string length
E8 ?? ?? ?? ??     call    runtime_slicebytetostring
48 83 ?? ??        add     rsp, <num>                  # epilogue clean up
5D                 pop     rbp
C3                 retn

---------------------------------------
# Split Transformation Epilogue Pattern

31 C0              xor     eax, eax
48 89 ??           mov     rbx, <reg>                  # decrypted string pointer
48 89 ??           mov     rcx, <reg>                  # decrypted string length
E8 ?? ?? ?? ??     call    runtime_slicebytetostring
48 83 ?? ??        add     rsp, <num>
5D                 pop     rbp
C3                 retn

---------------------------------------
# Seed Transformation Epilogue Pattern

48 8B ??           mov     rbx, [<reg>]                # decrypted string pointer
48 8B ?? ??        mov     rcx, [<reg>+8]              # decrypted string length
31 C0              xor     eax, eax
E8 ?? ?? ?? ??     call    runtime_slicebytetostring
48 83 ?? ??        add     rsp, <num>
5D                 pop     rbp
C3                 retn

Figure 12: Epilogue patterns of garble’s decrypting subroutines

Through the assembly patterns in Figure 12, we develop regex patterns corresponding to each of garble’s transformation types, which allows us to automatically identify string decrypting subroutines with high precision.

To extract the decrypted string, we must find the subroutine’s prologue and perform instruction-level emulation from this entry point until runtime_slicebytestring is called. For binaries of Go versions v1.21 to v1.23, we observe two main patterns of instructions in the subroutine prologue that perform the Go stack check.

# Go prologue pattern 1

49 3B ?? ??             cmp     rsp, [<reg>+<num>]
0F 86 ?? ?? ?? ??       jbe     <offset>

------------------------
# Go prologue pattern 2

49 3B ?? ??             cmp     rsp, [<reg>+<num>]
76 ??                   jbe     short <offset>

Figure 13: Prologue instruction patterns of Go subroutines

These instruction patterns in the Go prologue serve as reliable entry point markers for emulation. The implementation in GoStringUngarbler leverages these structural patterns to establish reliable execution contexts for the unicorn emulation engine, ensuring accurate string recovery across various garble string transformations.

Figure 14 shows the output of our automated extraction framework, where GoStringUngarbler is able to identify and emulate all decrypting subroutines.

From these instruction patterns, we have derived a YARA rule for detecting samples that are obfuscated with garble’s literal transformation. The rule can be found in Mandiant’s GitHub repository.

Deobfuscation: Subroutine Patching

While extracting obfuscated strings can aid malware detection through signature-based analysis, this alone is not useful for reverse engineers conducting static analysis. To aid reverse engineering efforts, we’ve implemented a binary deobfuscation approach leveraging the emulation results. 

Although developing an IDA plugin would have streamlined our development process, we recognize that not all malware analysts have access to, or prefer to use, IDA Pro. To make our tool more accessible, we developed GoStringUngarbler as a standalone Python utility to process binaries protected by garble. The tool can deobfuscate and produce functionally identical executables with recovered strings stored in plain text, improving both reverse engineering analysis and malware detection workflows.

For each identified decrypting subroutine, we implement a strategic patching methodology, replacing the original code with an optimized stub while padding the remaining subroutine space with INT3 instructions (Figure 15).

xor     eax, eax                           ; clear return register
lea     rbx, <string addr>                 ; Load effective address of decrypted string 
mov     ecx, <string len>                  ; populate string length
call    runtime_slicebytetostring          ; convert slice to Go string
ret                                        ; return the decrypted string

Figure 15: Function stub to patch over garble’s decrypting subroutines

Initially, we considered storing recovered strings within an existing binary section for efficient referencing from the patched subroutines. However, after examining obfuscated binaries, we found that there is not enough space within existing sections to consistently accommodate the deobfuscated strings. On the other hand, adding a new section, while feasible, would introduce unnecessary complexity to our tool.

Instead, we opt for a more elegant space utilization strategy by leveraging the inherent characteristics of garble’s string transformations. In our tool, we implement in-place string storage by writing the decrypted string directly after the patched stub, capitalizing on the guaranteed available space from decrypting routines:

1. Stack transformation: The decrypting subroutine stores and processes encrypted strings on the stack, providing adequate space through their data manipulation instructions. The instructions originally used for pushing encrypted data onto the stack create a natural storage space for the decrypted string.

2. Seed transformation: For each character, the decrypting subroutine requires a call instruction to decrypt it and update the seed. This is more than enough space to store the decrypted bytes.

3. Split transformation: The decrypting subroutine contains multiple switch cases to handle fragmented data recovery and decryption. These extensive instruction sequences guarantee sufficient space for the decrypted string data.

Figure 16 and Figure 17 show the disassembled and decompiled output of our patching framework, where GoStringUngarbler has deobfuscated a decrypting subroutine to display the recovered original string.

Downloading GoStringUngarbler

GoStringUngarbler is now available as an open-source tool in Mandiant’s GitHub repository. 

The installation requires Python3 and Python dependencies from the requirements.txt file.

Future Work

Deobfuscating binaries generated by garble presents a specific challenge—its dependence on the Go compiler for obfuscation means that the calling convention can evolve between Go versions. This change can potentially invalidate the regular expression patterns used in our deobfuscation process. To mitigate this, we’ve designed GoStringUngarbler with a modular plugin architecture. This allows for new plugins to be easily added with updated regular expressions to handle variations introduced by new Go releases. This design ensures the tool’s long-term adaptability to future changes in garble’s output.

Currently, GoStringUngarbler primarily supports garble–obfuscated PE and ELF binaries compiled with Go versions 1.21 through 1.23. We are continuously working to expand this range as the Go compiler and garble are updated.

Acknowledgments

Special thanks to Nino Isakovic and Matt Williams for their review and continuous feedback throughout the development of GoStringUngarbler. Their insights and suggestions have been invaluable in shaping and refining the tool’s final implementation.

We are also grateful to the FLARE team members for their review of this blog post publication to ensure its technical accuracy and clarity.

Additional thanks to OALabs for their valuable insights from their initial research on garble’s string encryption.

Finally, we want to acknowledge the developers of garble for their outstanding work on this obfuscating compiler. Their contributions to the software protection field have greatly advanced both offensive and defensive security research on Go binary analysis.

Unico is a leading biometric verification and authentication company addressing the global challenges of identity management and fraud prevention. 

With nearly two decades of experience in the Brazilian market, Unico has become a reliable supplier to over 800 companies, including four of the top five largest banks and leading retailers. Since 2021, Unico has facilitated more than 1.2 billion authentications through digital identity. It is estimated that Unico’s solutions have thwarted $14 billion in fraud, by 2023. Valued at over $2.6 billion, the company stands as the second most valuable SaaS company in Latin America, backed by General Atlantics, SoftBank, and Goldman Sachs,  and was recognized as the third most innovative company in Latin America by Fast Company in 2024. 

Currently, working on its global expansion, Unico has an ambitious vision to become the main identity network in the world, thus moving beyond traditional ID verification and embracing a broader spectrum of identity-related technologies. In this article, we’ll explore how Google Cloud and Spanner — Google’s always-on, virtually unlimited-scale database — is helping Unico achieve this goal.

Why vector search shines in Unico’s solutions

Unico is committed to delivering innovative, cutting-edge digital identity solutions. A cornerstone of this effort is the use of vector search technology, which enables powerful capabilities like 1:N search —  the ability to search for a single face within a large set of many others. This technology drives Unico’s identity solutions by retrieving and ranking multiple relevant matches for a given query with high precision and speed. 

However, developing 1:N searches poses a significant challenge: efficiently verifying facial matches within databases containing millions or billions of registered face vectors. Comparing an individual’s facial characteristics against each entry one by one is impractical. To address this, vector databases are often employed to perform Approximate Nearest Neighbor searches (ANN) and return the top-N most similar faces. 

Unico found that Spanner supports vector search capabilities to solve these issues, providing: 

• Semantic retrieval: Leveraging vector embeddings, Unico’s solutions can retrieve results based on deeper semantic relationships rather than exact matches. This improves the quality of identity verification, such as identifying relevant facial matches even when minor variations exist between the source and target images.

• Diversity and relevance: Using algorithms like ANN and exact K-Nearest Neighbors (KNN) vector search balances the need for diverse and relevant results, ensuring high reliability in fraud detection and identity verification.

• Multimodal applications: Vector search supports embeddings from multiple data types, such as text, images, and audio, enabling its use in complex, multimodal identity scenarios.

• Hybrid search: Modern vector search frameworks combine similarity search with metadata filters, allowing tailored results based on context, such as region or user preferences.

By integrating vector search, Unico provides customers with faster and smarter fraud detection tools. Leveraging high-precision algorithms, these tools can identify fraudulent faces with exceptional accuracy, effectively safeguarding businesses and individuals against identity theft and other threats. This innovation not only solidifies Unico’s position as a technology leader but also underscores its mission to build a safer, and more trusted world by creating a unified ecosystem to validate people’s real identities.

Some results

Operating at low latency while maintaining accuracy is crucial for Unico’s business, especially in applications that demand real-time performance, such as banking. Spanner was Unico’s first choice because its integrated support for vector search eliminates the need for separate, specialized vector database solutions. 

Spanner provides transactional guarantees for operational data, delivers fresh and consistent vector search results, and offers horizontal scalability. Its features also include GRPC (Google Remote Procedure Call) support, geo partitioning, multi-region storage configuration, RAG and LLM integrations, high SLA levels (99.99%), and maintenance-free architecture. Spanner also currently supports KNN and ANN vector searches.

Unico currently operates 1:N services in Brazil and Mexico, storing more than 1 billion facial embeddings in Spanner to date. This setup enables Unico to achieve low latency at high percentiles, high throughput of 840 RPM, and a precision/recall of 96%. And it’s just the start — Unico processes around 35 million new faces every month and that number continues to grow.

Unico remains focused on growing its customer base, enhancing its existing products, and exploring new opportunities in international markets — with the aim of expanding the reach of its secure digital identity services beyond Brazil’s borders. With Spanner and the ability to tap into the full power of the Google Cloud’s ecosystem, Unico is confident that it can bring its ambitious vision to life and deliver innovative solutions that forge trust between people and companies.

Learn more:

• Discover how Spanner can transform your business.

• Get started on Spanner today with a 90 day free trial instance. Or try out Spanner Vector Search.

Learn how Spanner Vector Search, hybrid search, and Spanner Graph can supercharge AI apps at virtually unlimited scale.

We’re pleased to announce that Google has been recognized as a Leader in The Forrester Wave™: Data Security Platforms, Q1 2025 report. We believe this is a testament to our unwavering commitment to providing cutting-edge data security in the cloud. 

In today’s AI era, comprehensive data security is paramount. Organizations are grappling with increasingly sophisticated threats, growing data volumes, and the complexities of managing data across diverse environments. That’s why a holistic, integrated approach to data security is no longer a nice-to-have — it’s a necessity.

Building on strengths

Google received the highest scores possible in 10 criteria, including: Vision, Innovation, Data threat and risk visibility, Data access controls, Data masking or redaction, Encryption, Supplier risk, and Use cases for privacy, Information governance, and AI security and governance.

“Organizations focused on going all-in on cloud and Zero Trust — especially those innovating with data and AI — that desire an integrated experience should consider Google,” the report states. As AI adoption accelerates, the need for a data security platform that can protect sensitive data while boosting innovation is paramount. 

Learn more

We invite you to read the full report to understand why Google Cloud is a leader in this space and how we can help your organization. This independent analysis provides valuable insights as you evaluate your data security strategy. We’re excited to continue this journey with you.

^{Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. For more information, read about Forrester’s objectivity here .}

A few weeks ago, Google DeepMind released Gemini 2.0 for everyone, including Gemini 2.0 Flash, Gemini 2.0 Flash-Lite, and Gemini 2.0 Pro (Experimental). All models support up to at least 1 million input tokens, which makes it easier to do a lot of things – from image generation to creative writing. It’s also changed how we convert documents into structured data. Manual document processing is a slow and expensive problem, but Gemini 2.0 changed everything when it comes to chunking pdfs for RAG systems, and can even transform pdfs into insights.  

Today, we’ll take a deep dive into a multi-step approach using generative AI where you can use Gemini 2.0 to improve your document extraction by combining language models (LLMs) with structured, externalized rules. 

A multi-step approach to document extraction, made easy

A multi-step architecture, as opposed to relying on a single, monolithic prompt, offers significant advantages for robust extraction. This approach begins with modular extraction, where initial tasks are broken down into smaller, more focused prompts targeting specific content locations within a document. This modularity not only enhances accuracy but also reduces the cognitive load on the LLM.

Another benefit of a multi-step approach is externalized rule management. By managing post-processing rules externally, for instance, using Google Sheets or a BigQuery table, we gain the benefits of easy CRUD (Create, Read, Update, Delete) operations, improving both maintainability and version control of the rules. This decoupling also separates the logic of extraction from the logic of processing, allowing for independent modification and optimization of each.

Ultimately, this hybrid approach combines the strengths of LLM-powered extraction with a structured rules engine. LLMs handle the complexities of understanding and extracting information from unstructured data, while the rules engine provides a transparent and manageable system for enforcing business logic and decision-making. The following steps outline a practical implementation.

Step 1: Extraction

Let’s test a sample prompt with a configurable set of rules. This hands-on example will demonstrate how easily you can define and apply business logic to extracted data, all powered by the Gemini and Vertex AI.

First, we extract data from a document. Let’s use Google’s 2023 Environment Report as the source document. We use Gemini with the initial prompt to extract data. This is not a known schema, but a prompt we’ve created for the purposes of this story. To create specific response schemas, use controlled generation with Gemini.

The JSON output below, which we’ll assign to the variable `extracted_data`, represents the results of the initial data extraction by Gemini. This structured data is now ready for the next critical phase: applying our predefined business rules.

Step 2: Feed the extracted data into a rules engine 

Next, we’ll feed this `extracted_data` into a rules engine, which, in our implementation, is another call to Gemini, acting as a powerful and flexible rules processor. Along with the extracted data, we’ll provide a set of validation rules defined in the `analysis_rules` variable. This engine, powered by Gemini, will systematically check the extracted data for accuracy, consistency, and adherence to our predefined criteria. Below is the prompt we provide to Gemini to accomplish this, along with the rules themselves.

`analysis_rules`  is a JSON object that contains the business rules we want to apply to the extracted receipt data. Each rule defines a specific condition to check, an action to take if the condition is met, and an optional alert message if a violation occurs. The power of this approach lies in the flexibility of these rules; you can easily add, modify, or remove them without altering the core extraction process. The beauty of using Gemini is that the rules can be written in human-readable language and can be maintained by non-coders.

Step 3: Integrate your insights 

Finally – and crucially –  integrate the alerts and insights generated by the rules engine into existing data pipelines and workflows. This is where the real value of this multi-step process is unlocked. For our example, we can build robust APIs and systems using Google Cloud tools to automate downstream actions triggered by the rule-based analysis. Some examples of downstream tasks are:

• Automated task creation: Trigger Cloud Functions to create tasks in project management systems, assigning data verification to the appropriate teams.

• Data quality pipelines: Integrate with Dataflow to flag potential data inconsistencies in BigQuery tables, triggering validation workflows.

• Vertex AI integration: Leverage Vertex AI Model Registry for tracking data lineage and model performance related to extracted metrics and corrections made.

• Dashboard integration Use Looker, Google Sheets, or Data Studio to display alerts

• Human in the loop trigger: Build a trigger system for the Human in the loop, using Cloud Tasks, to show which extractions to focus on and double check.

Make document extraction easier today

This hands-on approach provides a solid foundation for building robust, rule-driven document extraction pipelines. To get started, explore these resources:

• Gemini for document understanding: For a comprehensive, one-stop solution to your document processing needs, check out Gemini for document understanding. It simplifies many common extraction challenges.

• Few-shot prompting: Begin your Gemini journey with few-shot prompting. This powerful technique can significantly improve the quality of your extractions with minimal effort, providing examples within the prompt itself.

• Fine-tuning Gemini models: When you need highly specialized, domain-specific extraction results, consider fine-tuning Gemini models. This allows you to tailor the model’s performance to your exact requirements.

Cloud SQL, Google Cloud’s fully managed database service for PostgreSQL, MySQL, and SQL Server workloads, offers strong availability SLAs, depending on which edition you choose: a 99.95% SLA, excluding maintenance for Enterprise edition; and a 99.99% SLA, including maintenance for Enterprise Plus. In addition, Cloud SQL offers numerous high availability and scalability features that are crucial for maintaining business continuity and minimizing downtime, especially for mission-critical databases. 

These features can help address some common database deployment challenges:

1. Combined read/write instances: Using a single instance for both reads and writes creates a single point of failure. If the primary instance goes down, both read and write operations are impacted. In the event that your storage is full and auto-scaling is disabled, even a failover would not help. 

2. Downtime during maintenance: Planned maintenance can disrupt business operations.

3. Time-consuming scaling: Manually scaling instance size for planned workload spikes is a lengthy process that requires significant planning.

4. Complex cross-region disaster recovery: Setting up and managing cross-region DR requires manual configuration and connection string updates after a failover.

In this blog, we show you how to maximize your business continuity efforts with Cloud SQL’s high availability and scalability features, as well as how to use Cloud SQL Enterprise Plus features to build resilient database architectures that can handle workload spikes, unexpected outages, and read scaling needs.

Architecting a highly available and robust database 

Using the Cloud SQL high availability feature, which automatically fails over to a standby instance, is a good starting point but not sufficient: scenarios such as storage full issues, regional outages, or failover problems can still cause disruptions. Separating read workloads from write workloads is essential for a more robust architecture.

A best-practice approach involves implementing Cloud SQL read replicas alongside high availability. Read traffic should be directed to dedicated read-replica instances, while write operations are handled by the primary instance. You can enable high availability either on the primary, the read replica(s), or both, depending on your specific requirements. This separation helps ensure that the primary can serve production traffic predictably, and that read operations can continue uninterrupted via the read replicas even when there is downtime. 

Below is a sample regional architecture with high availability and read-replica enabled.

You can deploy this architecture regionally across multiple zones or extend it cross-regionally for disaster recovery and geographically-distributed read access. A regional deployment with a highly available primary and a highly available read replica that spans three availability zones provides resilience against zonal failures: Even if two zones fail, the database remains accessible for both read and write operations after failover. Cross-region read replicas enhance this further, providing regional DR capabilities.

Cloud SQL Enterprise Plus features

Cloud SQL Enterprise Plus offers significant advantages for performance and availability:

1. Enhanced hardware: Run databases on high-performance hardware with up to 128 vCPUs and 824GB of RAM.

2. Data cache: Enable data caching for faster read performance.

3. Near-zero downtime operations: Experience near-zero downtime maintenance and sub-second (<1s) downtime for instance scaling.

4. Advanced disaster recovery: Streamline disaster recovery with failover to cross-region DR-Replica and automatic reinstatement of the old primary. The application can still connect using the same write endpoint, which is automatically assigned to the new primary after failover. 

Enterprise Plus edition addresses the previously mentioned challenges:

1. Improved performance: Benefit from higher core-to-memory ratios for better database performance.

2. Faster reads: Data caching improves read performance for read-heavy workloads. Read-cache can be enabled in the primary, the read-replica, or both as needed. 

3. Easy scaling: Scale instances quickly with minimal downtime (sub-second) to handle traffic spikes or planned events. Scale the instance down when traffic is low with sub-second downtime. 

4. Minimized maintenance downtime: Reduce downtime during maintenance to less than a second and provide better business continuity. 

5. Handle regional failures: Easily fail over to a cross-region DR replica, and Cloud SQL automatically rebuilds your architecture as the original region recovers. This lessens the hassle of DR drills and helps ensure application availability. 

6. Automatic IP address re-pointing: Leverage the write endpoint to automatically connect to the current primary after a switchover or failover and you don’t need to make any IP address changes on the application end. 

To test out these benefits quickly, there’s an easy, near-zero downtime upgrade option from Cloud SQL Enterprise edition to Enterprise Plus edition.

Planned maintenance best practices

While Cloud SQL Enterprise Plus offers near-zero downtime planned maintenance, following these best practices can further enhance business continuity: 

1. Staging environment testing: To identify potential issues, use the maintenance timing feature to deploy maintenance to test/staging environments at least a week before production.

2. Read-replica maintenance: Apply self-service maintenance to one of the read replicas before the primary instance to avoid simultaneous downtime for read and write operations. Make sure that the primary and other replicas are updated shortly afterwards, as we recommend maintaining the same maintenance version in the primary as well as all the other replicas. 

3. Maintenance window: Always configure a maintenance window during off-peak hours to control when maintenance is performed. 

4. Maintenance notifications: Opt in to maintenance notifications to make sure you receive an email at least one week before scheduled maintenance. 

5. Reschedule maintenance: Use the reschedule maintenance feature if a maintenance activity conflicts with a critical business period.

6. Deny maintenance period: Use the deny maintenance period feature to postpone maintenance for up to 90 days during sensitive periods. 

By combining these strategies, you can build highly available and scalable database solutions in Cloud SQL, helping to ensure your business continuity and minimize downtime. Refer to the maintenance FAQ for more detailed information.

What is the true cost of enterprise AI?

As a technology leader and a steward of company resources, understanding these costs isn’t just prudent – it’s essential for sustainable AI adoption. To help, we’ll unveil a comprehensive approach to understanding and managing your AI costs on Google Cloud, ensuring your organization captures maximum value from its AI investments.

Whether you’re just beginning your AI journey or scaling existing solutions, this approach will equip you with the insights needed to make informed decisions about your AI strategy.

Why understanding AI costs matters now

Google Cloud offers a vast and ever-expanding array of AI services, each with its own pricing structure. Without a clear understanding of these costs, you risk budget overruns, stalled projects, and ultimately, a failure to realize the full potential of your AI investments. This isn’t just about saving money; it’s about responsible AI development – building solutions that are both innovative and financially sustainable.

Breaking down the Total Cost of Ownership (TCO) for AI on Google Cloud

Let’s dissect the major cost components of running AI workloads on Google Cloud:

Let’s estimate costs with an example

Let’s illustrate this with a hypothetical, yet realistic, generative AI use case: Imagine you’re a retail customer with an automated customer support chatbot.

Scenario: A medium-sized e-commerce company wants to deploy a chatbot on their website to handle common customer inquiries (order status, returns, product information and more). They plan to use a pre-trained language model (like one available through Vertex AI Model Garden) and fine-tune it on their own customer support data.

Assumptions:

• Model: Fine-tuning a low latency language model (in this case we will use Gemini 1.5 Flash).

• Training data: 1 million customer support conversations (text data).

• Traffic: 100K chatbot interactions per day.

• Hosting: Vertex AI Prediction for serving the model.

• Fine-tuning frequency: Monthly.

Cost estimation

As the retail customer in this example, here’s how you might approach this. 

1. First, discover your model serving cost:

• Vertex AI Prediction (Gemini 1.5 Flash for Chat) pricing is modality-based pricing so in this case since our input and output is text, the usage unit will be characters. Let’s assume an average of 1000 input characters and 500 output characters per interaction.
• Cost per 1M characters input: $0.0375.
• Cost per 1M characters output: $0.15
• Input cost per day: 100,000 interactions * 1000 characters * $0.0375 / 1000000 = $3.75
• Output cost per day: 100,000 interactions * 500 characters * $0.15 / 1000000 characters = $7.5
• Total model serving cost per day: $11.25
• Total model serving cost per month (~30 days): ~$337

2. Second, identify your training and tuning costs:

In this scenario, we aim to enhance the model’s accuracy and relevance to our specific use case through fine-tuning. This involves inputting a million past chat interactions, enabling the model to deliver more precise and customized interactions.

• Cost per training tokens: $8 / M tokens
• Cost per training characters: $2 / M characters (where each token approximately equates to 4 characters)
• Tuning cost (first month): 1,000,000 conversation (training data) * 1500 characters (input + output) * 2 /1,000,000 = $3,000
• Tuning cost (subsequent month): 100,000 conversation (new training data) * 1500 characters (input + output) * 2 /1,000,000 = $300

3. Third, understand the cloud hosting costs:

Since we’re using Vertex AI Prediction, the underlying infrastructure is managed by Google Cloud. The cost is included in the per-request pricing. However, if we are self-managing the model on GKE or Compute Engine, we’d need to factor in VM costs, GPU/TPU costs (if applicable), and networking costs. For this example, we assume this is $0, as it is part of Vertex AI cost.

4. Fourth, define the training data storage and adapter layers costs:

The infrastructure costs for deploying machine learning models often raise concerns, but the data storage components can be economical at moderate scales. When implementing a conversational AI system, storing both the training data and the specialized model adapters represents a minor fraction of the overall costs. Let’s break down these storage requirements and their associated expenses.

• 1M conversations, assuming an average size of 5KB per conversation, would be roughly 5GB of data.
• Cloud Storage cost for 5GB is negligible: $0.1 per month.
• Adapter layers (fine-tuned model weights) might add another 1GB of storage. This would still be very inexpensive: $0.02 per month.
• Total storage cost per month: < $1/month

5. Fifth, consider the application layer and setup costs: 

This depends heavily on the specific application. In this case we are using Cloud Run Functions and Logging. Cloud Run to handle pre- and post-processing of chatbot requests (e.g., formatting, database lookups). In this case let’s assume we use request-based billing so we are only charged when it processes the request. In this example we are processing 3M requests per month (100K * 30) and assuming 1 sec for average execution time: $14.30

• Cloud Logging and Monitoring for tracking chatbot performance and debugging issues. Let’s estimate 100GB of logging volume (which is on higher end) and retaining the logs for 3 months: $28

Total application layer cost per month:~ $40

6. Finally, incorporate the Operational support cost:

This is the hardest to estimate, as it depends on the internal team’s size and responsibilities. Let’s assume a conservative estimate of 5 hours per week of an engineer’s time dedicated to monitoring and maintaining the chatbot, at an hourly rate of $100.

• Total operational support cost per month: 5 hours/week * 4 weeks/month * $100/hour = $2000
• Total estimated monthly cost (First month):
• $ 340 (Serving) + $3000 (Training) + $1 (Storage) + $40 (Application) + $2000 (Operational) = $5,381
• Total estimated monthly cost (Subsequent months):
• $340 (Serving) + $300 (Training) + $1 (Storage) + $40 (Application) + $2000 (Operational) = $2,681

You can find the full estimate of cost here. Note that this does not include tuning and operational cost as it is not available in pricing export yet. 

Once you have a good understanding of your AI costs, it is important to develop an optimization strategy that encompasses infrastructure choices, resource utilization, and monitoring practices to maintain performance while controlling expenses. By understanding the various cost components and leveraging Google Cloud’s tools and resources, you can confidently embark on your AI journey. Cost management isn’t a barrier; it’s an enabler. It allows you to experiment, innovate, and build transformative AI solutions in a financially responsible way. 

Get started

• Start understanding your AI costs today: Explore the Google Cloud Pricing Calculator and the Vertex AI Pricing Page.

• Learn more at Google Cloud Next: Register for the Google Next session on AI Investment to Impact: Unlocking Sustainable ROI with Google Cloud.

• Engage Google Cloud for expert guidance: Get expert help to design cost effect AI architectures, contact Google Cloud Consulting or PSO.

Written by: Joshua Goddard

Executive Summary

• Rosetta 2 is Apple’s translation technology for running x86-64 binaries on Apple Silicon (ARM64) macOS systems.

• Rosetta 2 translation creates a cache of Ahead-Of-Time (AOT) files that can serve as valuable forensic artifacts.

• Mandiant has observed sophisticated threat actors leveraging x86-64 compiled macOS malware, likely due to broader compatibility and relaxed execution policies compared to ARM64 binaries.

• Analysis of AOT files, combined with FSEvents and Unified Logs (with a custom profile), can assist in investigating macOS intrusions.

Introduction

Rosetta 2 (internally known on macOS as OAH) was introduced in macOS 11 (Big Sur) in 2020 to enable binaries compiled for x86-64 architectures to run on Apple Silicon (ARM64) architectures. Rosetta 2 translates signed and unsigned x86-64 binaries just-in-time or ahead-of-time at the point of execution. Mandiant has identified several new highly sophisticated macOS malware variants over the past year, notably compiled for x86-64 architecture. Mandiant assessed that this choice of architecture was most likely due to increased chances of compatibility on victim systems and more relaxed execution policies. Notably, macOS enforces stricter code signing requirements for ARM64 binaries compared to x86-64 binaries running under Rosetta 2, making unsigned ARM64 binaries more difficult to execute. Despite this, in the newly identified APT malware families observed by Mandiant over the past year, all were self-signed, likely to avoid other compensating security controls in place on macOS.

The Rosetta 2 Cache

When a x86-64 binary is executed on a system with Rosetta 2 installed, the Rosetta 2 Daemon process (oahd) checks if an ahead-of-time (AOT) file already exists for the binary within the Rosetta 2 cache directory on the Data volume at /var/db/oah/<UUID>/. The UUID value in this file path appears to be randomly generated on install or update. If an AOT file does not exist, one will be created by writing translation code to a .in_progress file and then renaming it to a .aot file of the same name as the original binary. The Rosetta 2 Daemon process then runs the translated binary.

The /var/db/oah directory and its children are protected and owned by the OAH Daemon user account _oahd. Interaction with these files by other user accounts is only possible if System Integrity Protection (SIP) is disabled, which requires booting into recovery mode.

The directories under /var/db/oah/<UUID>/ are binary UUID values that correspond to translated binaries. Specifically, these binary UUID values are SHA-256 hashes generated from a combination of the binary file path, the Mach-O header, timestamps (created, modified, and changed), size, and ownership information. If the same binary is executed with any of these attributes changed, a new Rosetta AOT cache directory and file is created. While the content of the binaries is not part of this hashing function, changing the content of a file on an APFS file system will update the changed timestamp, which effectively means content changes can cause the creation of a new binary UUID and AOT file. Ultimately, the mechanism is designed to be extremely sensitive to any changes to x86-64 binaries at the byte and file system levels to reduce the risk of AOT poisoning.

The Rosetta 2 cache binary UUID directories and the AOT files they contain appear to persist until macOS system updates. System updates have been found to cause the deletion of the cache directory (the Random UUID directory). After the upgrade, a directory with a different UUID value is created, and new Binary UUID directories and AOT files are created upon first launch of x86-64 binaries thereafter.

Translation and Universal Binaries

When universal binaries (containing both x86-64 and ARM64 code) are executed by a x86-64 process running through Rosetta 2 translation, the x86-64 version of these binaries is executed, resulting in the creation of AOT files.

In a Democratic People’s Republic of Korea (DPRK) crypto heist investigation, Mandiant observed a x86-64 variant of the POOLRAT macOS backdoor being deployed and the attacker proceeding to execute universal system binaries including ping, chmod, sudo, id, and cat through the backdoor. This resulted in AOT files being created and provided evidence of attacker interaction on the system through the malware (Figure 5).

In some cases, the initial infection vector in macOS intrusions has involved legitimate x86-64 code that executes malware distributed as universal binaries. Because the initial x86-64 code runs under Rosetta 2, the x86-64 versions of malicious universal binaries are executed, leaving behind Rosetta 2 artifacts, including AOT files. In one case, a malicious Python 2 script led to the downloading and execution of a malicious universal binary. The Python 2 interpreter ran under Rosetta 2 since no ARM64 version was available, so the system executed the x86-64 version of the malicious universal binary, resulting in the creation of AOT files. Despite the attacker deleting the malicious binary later, we were able to analyze the AOT file to understand its functionality.

Unified Logs

The Rosetta 2 Daemon emits logs to the macOS Unified Log; however, the binary name values are marked as private. These values can be configured to be shown in the logs with a custom profile installed. Informational logs are recorded for AOT file lookups, when cached AOT files are available and utilized, and when translation occurs and completes. For binaries that are not configured to log to the Unified Log and are not launched interactively, in some cases this was found to be the only evidence of execution within the Unified Logs. Execution may be correlated with other supporting artifacts; however, this is not always possible.

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Aot lookup request for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Translating image <private> -> <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Translation finished for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Aot lookup request for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Using cached aot <private> -> <private>

Figure 3: macOS Unified Logs showing Rosetta lookups, using cached files, and translating with private data disabled (default)

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Aot lookup request for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Translating image /Users/Mandiant/my_binary -> 
/var/db/oah/237823680d6bdb1e9663d60cca5851b63e79f6c
8e884ebacc5f285253c3826b8/1c65adbef01f45a7a07379621
b5800fc337fc9db90d8eb08baf84e5c533191d9/my_binary.in_progress

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Translation finished for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary(34180): 
Aot lookup request for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary(34180): 
Using cached aot /Users/Mandiant/my_binary -> 
/var/db/oah/237823680d6bdb1e9663d60cca5851b63e
79f6c8e884ebacc5f285253c3826b8/1c65adbef01f45a7
a07379621b5800fc337fc9db90d8eb08baf84e5c533191d9/my_binary.aot

Figure 4: macOS Unified Logs showing Rosetta lookups, using cached files, and translating with private data enabled (with custom profile installed)

FSEvents

FSEvents can be used to identify historical execution of x86-64 binaries even if Unified Logs or files in the Rosetta 2 Cache are not available or have been cleared. These records will show the creation of directories within the Rosetta 2 cache directory, the creation of .in_progress files, and then the renaming of the file to the AOT file, which will be named after the original binary.

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba3
5cff449a366ea59b26af7d54c59779cdfac161 
FolderEvent; FolderCreated; 35102230

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba35
cff449a366ea59b26af7d54c59779cdfac161/
com.apple.systemsettings.cache.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102231

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba35
cff449a366ea59b26af7d54c59779cdfac161/com.apple.systemsettings.cache.aot 
FileEvent; Renamed; 35102231

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599f
73f25fab17744eb0c51b8d0071c53bb878471 
FolderEvent; FolderCreated; 35102223

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599
f73f25fab17744eb0c51b8d0071c53bb878471/sudo.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102224

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599
f73f25fab17744eb0c51b8d0071c53bb878471/sudo.aot
FileEvent; Renamed; 35102224

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629 
FolderEvent; FolderCreated; 35102259

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629/id.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102260

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629/id.aot 
FileEvent; Renamed; 35102260
private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9 
FolderEvent; FolderCreated; 35102355

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9/chmod.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102356

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9/chmod.aot 
FileEvent; Renamed; 35102356

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde 
FolderEvent; FolderCreated; 35102671

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde/cat.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102672

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde/cat.aot 
FileEvent; Renamed; 35102672

Figure 5: Decoded FSEvents records showing the translation of a x86-64 POOLRAT variant on macOS, and subsequent universal system binaries executed by the malware as x86-64

AOT File Analysis

The AOT files within the Rosetta 2 cache can provide valuable insight into historical evidence of execution of x86-64 binaries. In multiple cases over the past year, Mandiant identified macOS systems being the initial entry vector by APT groups targeting cryptocurrency organizations. In the majority of these cases, Mandiant identified evidence of the attackers deleting the malware on these systems within a few minutes of a cryptocurrency heist being perpetrated. However, the AOT files were left in place, likely due to the protection by SIP and the relative obscurity of this forensic artifact.

From a forensic perspective, the creation and modification timestamps on these AOT files provide evidence of the first time a specified binary was executed on the system with a unique combination of the attributes used to generate the SHA-256 hash. These timestamps can be corroborated with other artifacts related to binary execution where available (for example, Unified Logs or ExecPolicy, XProtect, and TCC Databases), and file system activity through FSEvents records, to build a more complete picture of infection and possible attacker activity if child processes were executed.

Where multiple AOT files exist for the same origin binary under different Binary UUID directories in the Rosetta 2 cache, and the content (file hashes) of those AOT files is the same, this is typically indicative of a change in file data sections, or more commonly, file system metadata only.

Mandiant has previously shown that AOT files can be analyzed and used for malware identification through correlation of symbols. AOT files are Mach-O binaries that contain x86-64 instructions that have been translated from the original ARM64 code. They contain jump-backs into the original binary and contain no API calls to reference. Certain functionality can be determined through reverse engineering of AOT files; however, no static data, including network-based indicators or configuration data, are typically recoverable. In one macOS downloader observed in a notable DPRK cryptocurrency heist, Mandiant observed developer file path strings as part of the basic Mach-O information contained within the AOT file. The original binary was not recovered due to the attacker deleting it after the heist, so this provided useful data points to support threat actor attribution and malware family assessment.

/Users/crown/Library/Developer/Xcode/DerivedData/
DownAndMemload-becawjfobisdcocirecqedzcixcf/Build/Intermediates.noindex/
DownAndMemload.build/Release/DownAndMemload.build/
Objects-normal/x86_64/main.o
/Users/crown/Library/Developer/Xcode/DerivedData/
DownAndMemload-becawjfobisdcocirecqedzcixcf/Build/Intermediates.noindex/
DownAndMemload.build/Release/DownAndMemload.build/Objects-normal/
x86_64/queue.o
/Volumes/Data/Development/DownAndMemload/DownAndMemload/
_g_szServerUrl
_szgetpwuid
_szsleep

Figure 6: Interesting strings from an AOT file related to a malicious DPRK downloader that was unrecoverable

In any case, determining malware functionality is more effective using the original complete binary instead of the AOT file, because the AOT file lacks much of the contextual information present in the original binary. This includes static data and complete Mach-O headers.

Poisoning AOT Files

Much has been written within the industry about the potential for the poisoning of the Rosetta 2 cache through modification or introduction of AOT files. Where SIP is disabled, this is a valid attack vector. Mandiant has not yet seen this technique in the wild; however, during hunting or investigation activities, it is advisable to be on the lookout for evidence of AOT poisoning. The best way to do this is by comparing the contents of the ARM64 AOT files with what would be expected based on the original x86-64 executable. This can be achieved by taking the original x86-64 executable and using it to generate a known-good AOT file, then comparing this to the AOT file in the cache. Discrepancies, particularly the presence of injected shellcode, could indicate AOT poisoning.

Conclusion

There are several forensic artifacts on macOS that may record historical evidence of binary execution. However, in cases of advanced intrusions with forensically aware attackers, original binaries being deleted, and no further security monitoring solutions, combining FSEvents, Unified Logs, and, crucially, residual AOT files on disk has provided the residual evidence of intrusion on a macOS system.

Whilst signed macOS ARM64 binaries may be the future, for now AOT files and the artifacts surrounding them should be reviewed in analysis of any suspected macOS intrusion and leveraged for hunting opportunities wherever possible.

The behavior identified in the cases presented here was identified on various versions of macOS between 13.5 and 14.7.2. Future or previous versions of macOS and Rosetta 2 may behave differently.

Acknowledgements

Special thanks to Matt Holley, Mohamed El-Banna, Robert Wallace, and Adrian Hernandez.

It’s a persistent question: How do you know which generative AI model is the best choice for your needs? It all comes down to smart evaluation.

In this post, we’ll share how to perform pairwise model evaluations – a way of comparing two models directly against each other – using Vertex AI evaluation service and LLM Comparator. We’ll introduce each tool’s useful features, why the tools help us evaluate performance of LLMs, and how you can use it to create a robust evaluation framework.

Pairwise model evaluation to assess performance 

Pairwise model evaluation means comparing two models directly against each other to assess their relative performance on a specific task. There are three main benefits to pairwise model evaluation for LLMs:

1. Make informed decisions: The increasing number and variety of LLMs means you need to carefully evaluate and choose the best model for your specific task. Considering the strengths and weaknesses of each option is table stakes.

2. Define “better” quantitatively: Generated content from generative AI models, such as natural language texts or images, are usually unstructured, lengthy, and difficult to evaluate automatically without human intervention. Pairwise helps define ”better” response close to human responses to each prompt with human inspection.

3. Keep an eye out: LLMs should be continuously retrained and tuned with the new data to be enhanced compared with the previous versions of them and other latest models.

Vertex AI evaluation service 

The Gen AI evaluation service in Vertex AI lets you evaluate any generative model or application and benchmark the evaluation results against your own judgment, using your own evaluation criteria. It helps with: 

• Model selection among different models for specific use cases

• Model configuration optimization with different model parameters

• Prompt engineering for the preferred behavior and responses

• Fine-tuning LLMs for improved accuracy, fairness, and safety

• Optimizing RAG architectures

• Migration between different versions of a model

• Managing translation qualities between different languages 

• Evaluating agents

• Evaluating images and videos

It also supports model-based metrics for both pointwise and pairwise evaluations and computation-based metrics with ground-truth datasets of input and output pairs.

How to use Vertex AI evaluation service

The Vertex AI evaluation service can help you rigorously assess your generative AI models. You can define custom metrics, leveraging pre-built templates or your own expertise, to precisely measure performance against your specific goals. For standard NLP tasks, the service provides computation-based metrics like F1 scores for classification, BLEU for translation, and ROUGE-L for summarization.

For direct model comparison, pairwise evaluations allow you to quantify which model performs better. Metrics like candidate_model_win_rate and baseline_model_win_rate are automatically calculated, and judge models provide explanations for their scoring decisions, offering valuable insights. You can also perform pairwise comparisons using computation based metrics to compare against the ground truth data.

Beyond pre-built metrics, you have the flexibility to define your own, either through mathematical formulas or by using prompts to help “judge models” aligned with the context of the user-defined metrics. Embedding-based metrics are also available for evaluating semantic similarity.

Vertex AI Experiments and Metadata seamlessly integrate with the evaluation service, automatically organizing and tracking your datasets, results, and models. You can easily initiate evaluation jobs using the REST API or Python SDK and export results to Cloud Storage for further analysis and visualization.

In essence, the Vertex AI evaluation service provides a comprehensive framework for:

• Quantifying model performance: Using both standard and custom metrics.

• Comparing models directly: Through pairwise evaluations and judge model insights.

• Customizing evaluations: To meet your specific needs.

• Streamlining your workflow: With integrated tracking and easy API access.

It also provides guidance and templates to help you define your own metrics referring to those templates or from scratch with your experiences of prompt engineering and generative AI.

LLM Comparator: An open-source tool for human-in-the-loop LLM evaluation

LLM Comparator is an evaluation tool developed by PAIR (People + AI Research; PAIR) at Google, and is an active research project. 

LLM Comparator’s interface is highly intuitive for side-by-side comparisons of different model outputs, making it an excellent tool to augment automated LLM evaluation with human-in-the-loop processes. The tool provides useful features to help you evaluate the responses from two LLMs side-by-side using a range of informative metrics, such as the win rates of Model A or B, grouped by prompt category. It is also simple to extend the tool with user-defined metrics, via a feature called Custom Functions.

You can see the comparative performance of Model A and Model B across various metrics and prompt categories through ‘Score Distribution’ and ‘Metrics by Prompt Category’ visualizations. In addition, the ‘Rationale Summary visualization provides insights into why one model outperforms another by visually summarizing the key rationales influencing the evaluation results.

LLM Comparator is available as a Python package on PyPI, and can be installed on a local environment. Pairwise evaluation results from the Vertex AI Evaluation Service can also be loaded into LLM Comparator using provided libraries. To learn more about how you can transform the automated evaluation results to JSON files, please refer to the JSON data format and schema for LLM Comparator.

With features such as the Rationale Cluster visualization and Custom Functions, LLM Comparator can serve as an invaluable tool in the final stages of LLM evaluation where human-in-the-loop processes are needed to ensure overall quality.

Feedback from the field: How LLM Comparator adds value to Vertex AI evaluation service

By augmenting human evaluators with ready-to-use convenient visualizations and performance metrics calculated automatically, LLM Comparator reduces many chores of ML engineers to develop their own visualizations and quality monitoring tools. Thanks to the JSON data format and schema of LLM Comparator, Vertex AI evaluation service and LLM Comparator can be integrated conveniently without any serious amount of development work.

We’ve heard from our teams that the most useful feature of LLM Comparator is the visualization of “Rationale Summary”. “Rationale Summary” can be thought of as a kind of explainable AI (XAI) tool which is very useful to learn why a specific model among the two is better in the judge model’s view. Another important aspect of “Rationale Summary” visualization is that it can be used to understand how a specific language model is working differently from the other model, which is sometimes a very important support to infer why the model is more appropriate for specific tasks.  

A limitation of LLM Comparator is that it can be used just for pair-wise model evaluation, not for simultaneous multiple model evaluation. However, LLM Comparator already has basic components for comparative LLM evaluations and extending it to simultaneous multiple model evaluation may not be a big technical problem. This can be an excellent project for you to contribute to the LLM Comparator project.

Conclusion 

In this article, we learned and discussed how we can organize the evaluation process of LLMs with Vertex AI and LLM Comparator, an open source LLM evaluation tool by PAIR. By combining Vertex AI Evaluation Service and LLM Comparator, we’ve presented a semi-automated approach to systematically evaluate and compare the performance of diverse LLMs on Google Cloud. Get started with Vertex AI Evaluation Service today. 

^{We thank Rajesh Thallam, Skander Hannachi, and the Applied AI Engineering team for help with this blog post and guidance on overall best practices. We also thank Anant Nawalgaria for help with this blog post and technical guidance.}

Yassir is a super app, supporting the daily lives of users in more than 45 cities across Algeria, Morocco, Tunisia, South Africa, and Senegal who rely on our ride-hailing, last-mile delivery, and financial services solutions. These users are both consumers and vendors — including drivers, couriers, restaurants, and more — that use our platform to run their businesses. 

At Yassir, we process a wide variety of datasets to ensure we provide the best and most reliable solutions for our users across all of our offerings, and we depend on that data to continually improve those services. However, our previous infrastructure made unifying data and AI difficult. 

Previously, we had two separate data systems: one using Databricks for deploying and training machine learning models and another through Google Cloud and BigQuery for storing and analyzing data. This setup led to several issues, such as formatting incompatibilities that we could not resolve. In addition, retrieving data from Databricks for processing within Google Cloud wasn’t possible, and this disconnect directly impacted our application performance.

These siloed environments meant our teams often had to duplicate work to develop and maintain any data projects, paying to maintain separate environments, and, despite all of this, failing to get the information that teams needed at the desired pace. 

To address these issues, we decided to consolidate our data infrastructure with Google Cloud to bring all of these functions into one place. This migration would allow us to provide better access to data and more scalability, and create new opportunities to analyze, review, and improve performance.

Creating a more flexible, unified data platform

Our existing relationship with the Google Cloud team provided a strong foundation to not only resolve our data connectivity roadblocks but also implement new data processing workflows using BigQuery and deploy new AI and machine learning models with Vertex AI. Consolidating with a single data provider also gave us a centralized place to review and control expenses as well as simple, centralized data governance controls. As a growing company, being able to scale our cloud usage up or down to optimize costs allows us to test and iterate without a hard commitment to every project, and that flexibility is invaluable. 

We worked closely with the Google Cloud team to design a solution that aligns with our growth goals. This meant participating in technical and strategic workshops to help train our team on the ins and outs of BigQuery — and its real-time, governance, and open-source capabilities — empowering our engineers with the tools and resources they need to experiment. This collaborative approach allows us to nurture the type of engineering culture we want to promote at Yassir; rather than simply using out-of-the-box solutions, we can tackle more complex problems by adapting flexible, existing solutions to our specific use cases.

After conducting our internal compatibility reviews, we migrated individual models from our previous solution into Vertex AI to test their consistency, and now they’re up and running nearly autonomously. By migrating from Databricks to BigQuery and combining our own models with the models provided by Google Cloud, we’ve improved the performance and efficiency of our machine learning processes and better positioned ourselves for ongoing growth. We may not be processing petabytes of data yet, but we know that we have the capability to do so when needed.

Evolving from data processing to data insights

Our previously disconnected data solutions made it difficult to provide secure access to specific data for specific teams. Since we stored our data in BigQuery but deployed models with Databricks, granting access to information to a user or a team meant giving them the keys to everything. Now, we can implement role-based access controls as well as Infrastructure as Code (IaC) Terraform scripts to automatically grant and revoke access to datasets for individuals or teams. Sharing data through Looker Studio Pro and directly providing access to BigQuery tables for our more technical users also means we can ensure the required data reaches the right users.

With our data unified in BigQuery and connected to our machine learning models, we can better support everything from customer growth and retention to marketplace optimization by providing insights into product usage, customer data, and more. To ensure we’re hitting our internal and customer-related goals, we closely monitor and create dashboards for operational and analytical datasets. 

Our operational dashboards give our sales and marketing teams the insights they need to better target and reach merchants and consumers. They also include insights into our staffing processes, helping us to gradually reduce delivery times, complete more rides faster, and improve how we support specific markets. We also have product-level detection and monitoring that help us provide real-time dynamic pricing and identify fraudulent trips and orders. Each data point we collect gives us more opportunities to build a more personalized and consistent customer experience. 

Our leadership team relies on our rapidly available datasets to drive strategic decision-making, including regional investment decisions to grow the business, macro-level plans for growth trajectories and marketing budgets, and identification of the areas of the business that need the most support or attention. These roadmap decisions are core to our overall growth strategy, and they wouldn’t be possible without the flexibility and scalability we’ve been able to achieve with BigQuery.

Many organizations use Terraform to manage their cloud deployments through Infrastructure as Code. Today we are excited to share the general availability of Terraform providers for Oracle Database@Google Cloud. You can now deploy and manage Oracle Autonomous Database and Oracle Exadata Database Service resources using the Google Terraform provider. The release compliments the existing gcloud and Google Cloud console capabilities that we released with Oracle Database@Google Cloud at launch.

Provision Autonomous Database with Terraform

Creating an Oracle Autonomous Database on Google Cloud is as simple as defining a google_oracle_database_autonomous_database resource and running terraform init and terraform apply. The below code example does the following:

• Collects the details of VPC network where the ADB will be placed

• In this case, we are using an existing VPC named ora-vpc in the GCP project named project1 

• ID and display name of adbtf1

• Region of us-east4

• Project project1

• The admin password has been obfuscated

• VPC network defined above

• A client CIDR range of 172.16.44.0/25

• An ECPU count of 2

• Allocated storage of 100GB

• Autonomous workload type of OLTP

• License model using Oracle’s license-included SKU

Provision with ease using Google Cloud

Working with Terraform is simple on Google Cloud thanks to our native integration in Cloud Shell. Launch Cloud Shell from the Google Cloud console and create Terraform configuration files using our native editor or pull from your company’s source code repository. You can even run the terraform command directly in Cloud Shell. You’ll be authenticated using your Google Cloud IAM credentials and your resources will be immediately created and synchronized according to the infrastructure as code definitions. You can see the console, editor, and Cloud Shell in action below:

Why use Terraform for your Oracle Database@Google Cloud deployments?

• Simplified management: Define your entire Oracle Autonomous Databases or Exadata infrastructure and VM clusters in declarative configuration files.

• Integrated automation: Automate the provisioning, configuration, and management of your Oracle Database@Google Cloud resources.

• Improved consistency: Ensure consistent and repeatable deployments across different environments.

• Reduced errors: Minimize manual configuration and reduce the risk of human error.

• Enhanced collaboration: Enable teams to collaborate effectively on infrastructure management.

Ready to experience the benefits of Infrastructure as Code for Oracle Database@Google Cloud?

• Explore the google provider documentation: Autonomous Database Serverless, Exadata Infrastructure and Exadata VM Clusters

• Dive into the oci provider documentation: Exadata Database, Database homes, and Terraform on Oracle Cloud

The emergence of 5G networks opens a new frontier for connectivity, enabling advanced use cases that require ultra-low-latency, enhanced mobile broadband, and the Internet of Things (IoT) at scale. However, behind the promise of this hyper-connected future lies an equally critical challenge: managing the complexity of 5G networks. With dynamic demands on bandwidth, latency, and reliability, traditional static configurations and manual operations are no longer sufficient.

Enter intent-based service management, a pioneering approach that combines AI and automation with service orchestration and assurance. With intent-based service management, network resources in a 5G network scale and adjust dynamically according to real-time demand, based on the business intent, the 5G network resources are orchestrated dynamically and adjusted just as they are for cloud computing, storage, and networking and streaming services. ​

Together, Ericsson and Google Cloud are pushing the boundaries of this technology, exploring  technology that not only simplifies operations but redefines how telecommunications companies can deliver on the promise of 5G.

The power of intent-based autonomous operations in 5G

Intent-based networking has entered the spotlight as Communications Service Providers (CSPs) recognize that they must master substantial network complexity with sophisticated automation if they want to create, deliver, and sustain services, especially in cloud-native network environments like standalone 5G.​ Automation of networks and related operations cannot evolve and mature to full closed-loop autonomy without intent-based implementations.

Take for example a hospital that needs dedicated 5G connectivity, provided by a 5G network slice, in order to support emergency communications with ultra-low latency. The required configuration in such a scenario should not be a static configuration;  in order to provide optimal service experience it should evolve with the network conditions and the demand. The network layer needs to understand the intent behind every service request  e.g. “provide ultra-low-latency video services” — and translate it into actionable configurations across the network, which then are dynamically updated based on real-time factors such as changes in traffic conditions, underlying network status, and so on.

With Ericsson’s Service Orchestration and Assurance and Google Cloud’s Vertex AI, this process becomes far easier. The solution takes high-level intents, expressed in natural language, and maps them to precise technical requirements like bandwidth, latency, and throughput; then orchestrates the creation of a tailored network slice in real time. The quality of network-slice-based service provisioned is assured via the intelligent closed loop automation mechanism.

TMF standardization for intent-based operations​

To integrate different solution components with one another without expensive and fragile integration solutions, you need a standards-based interface and model.

The Telemanagement Forum (TMF) has specified APIs and a model with common intents to enable this sort of integration. TMF provides intent models and interfaces at the business, service, and resource layers, which derive from CSP-defined business objectives and follow the Autonomous Networks Reference Architecture.​ Ericsson and Google Cloud’s approach is fully compliant to these standards and intent models.

The Role of AI and automation

At the heart of this solution lies AI-driven closed-loop automation. Once a service is instantiated, AI agents continuously monitor its performance against defined KPIs such as quality of service (QoS). Going back to our hospital example, if latency in the hospital’s network slice exceeds the acceptable thresholds, the system automatically identifies the issue and proposes corrective actions, such as reallocating RAN resources or Core Network.

This closed-loop system isn’t just reactive — it’s predictive. By analyzing data from the network in real time, it can anticipate issues and resolve them before they impact users. For telcos, this means not just operational efficiency but also delivering a superior  quality of service.

Ericsson’s closed-loop system leverages Google Cloud’s AI agents for the intent translation, proposal and evaluation agents for slice assurance and remediation. The AI agents reason on how to best achieve a goal based on inputs and tools at their disposal, leveraging models, tools, and orchestration.

In addition to Gemini-based AI agents, Ericsson Service Orchestration and Assurance uses Vertex AI to implement predictive AI services for the remaining closed-loop agents (such as assurance, measurement, and actuation). 

Why Ericsson Service Orchestration and Assurance and Google Cloud

Ericsson Service Orchestration and Assurance is a multi-domain, multi-technology platform that enables cross-domain orchestration across access, transport, and core networks. It addresses CSPs’ critical need for simplification of open and programmable networks by automating the service lifecycle, ultimately boosting revenues and profitability.

The platform provides an open, standards-based architecture that fosters innovation and multi-vendor integration in the telecom ecosystem. It allows CSPs to design services with ease across both network and IT environments, reducing time-to-market and enabling collaboration across diverse teams and use cases with fewer manual errors. 

Furthermore, intent-driven orchestration accelerates the testing and launch of new services in multi-vendor environments, potentially increasing the number of services that can be launched. SLA commitments are enhanced through closed-loop automation, improving customer satisfaction and operational efficiency. Designed to be vendor-, technology-, and service-agnostic, Ericsson’s platform empowers CSPs to implement multi-domain service orchestration effectively, enabling service expansion and revenue growth.

To support this sophisticated framework, Google Cloud provides a foundation to meet the demands of modern telecom networks:

• Scalability for dynamic networks: As 5G networks expand, Google Cloud’s elastic compute power helps ensure even the most resource-intensive processes, such as intent translation and real-time assurance, run seamlessly without performance bottlenecks.

• AI-first infrastructure: Leveraging Vertex AI, intelligent agents translate intents into actionable configurations while optimizing network performance. With AI tools like BigQuery ML, telcos can move beyond basic analytics to actionable insights, enabling smarter decision-making.

• Vertex Model Garden for open innovation: Pre-trained models and open-source LLMs in Vertex AI Model Garden empower telcos to innovate and customize highly specialized 5G service management solutions.

• Hardware acceleration: Google Cloud’s Tensor Processing Units (TPUs) and Nvidia GPUs provide the computational power necessary to efficiently train and deploy complex AI models for intent-based service management.

• Hybrid and edge capabilities: 5G networks demand hybrid setups across cloud and edge environments. Google Cloud’s edge-to-cloud orchestration helps ensure operations, whether optimizing central data centers or delivering ultra-low-latency services at the edge.

• Security and compliance: Operating in a highly regulated environment, telcos benefit from Google Cloud’s secure, reliable infrastructure tailored to meet the stringent compliance requirements of telecom workloads.

• Gemini for advanced reasoning: Gemini’s multimodal models enhance intent interpretation by processing natural language and network telemetry data. This enables nuanced and precise intent translation, leading to more effective automated actions, higher efficiency, and improved reliability of 5G service management.

Together, Ericsson Service Orchestration and Assurance and Google Cloud are exploring technology to meet these challenges together, combining automation, intelligence, and scalability to empower CSPs to deliver innovative, efficient, and profitable 5G services.

Real-world impact: A hospital use case

Let’s return to the hospital scenario. A request for a network slice is received from hospital users through the system’s intent translator: “Provide video service for patient doctor interactions having ultra low latency.​”

The solution springs into action:

1. Translating intent: The request is converted into technical configurations for the RAN and core network.

2. Creating the slice: Using orchestration tools, the system allocates resources and configures the network slice with Quality of Service (QoS) goals across the 5G network.

3. Assuring performance: With closed-loop automation, AI agents continuously monitor quality of service elements such as latency or packet loss. If a deviation that violates the intent occurs or is forecasted to occur, corrective actions are proposed and executed automatically.

In this scenario, the hospital’s needs are met dynamically, helping to ensure critical operations proceed without disruption.

Beyond efficiency: Redefining 5G Innovation

Intent-based service management enables telcos to innovate at scale. By automating complex tasks, telcos can focus on creating new services, entering new markets, and unlocking revenue streams. For instance, they can offer differentiated slices tailored to industries like gaming, manufacturing, and healthcare, creating a competitive edge in a crowded market.

Moreover, the integration of custom gen AI models like Gemini helps these solutions be tailored to telcos’ unique needs. These LLMs, trained on telecom-specific data, enable intelligent automation that feels intuitive and responsive.

Ericsson and Google Cloud’s collaboration showcases the potential of intent-based service management. By combining Ericsson’s technical expertise with Google Cloud’s AI infrastructure, telcos can navigate the complexities of 5G with confidence.

As 5G adoption accelerates, solutions like this will become the cornerstone of telecom operations, enabling seamless, dynamic networks that deliver on the promise of connectivity.

The future of 5G isn’t just about faster speeds or lower latency — it’s about intelligent, intent-driven networks that adapt to the world’s needs in real time. And together, Ericsson and Google Cloud are making that future a reality.

Telecoms, like all businesses, are wondering how AI can transform their businesses. And there’s no better way to display how to build the AI-driven telecom than with demos. Join us at Mobile World Congress 2025, March 3-6 in Barcelona Hall 2 Booth #2H40, where we’ll be highlighting key agent use cases where AI is becoming an imperative:

• Customer experience

• Employee productivity

• Field operations

• Network operations

We know that the future of telecom isn’t just about gen AI — it’s also about data and security. We will show how to execute on cloud migrations and implementation of our core security practices.

We can’t do this alone. Across our 12 demonstrations, we have collaborations with key customers and partners on display including: Amdocs, Bell, Deutsche Telekom, Ericsson, MasOrange, Nokia, TELUS, and Vodafone. 

Lastly, we will showcase the power of cloud-native networks and the ability to monetize those networks. Discover how we harness the power of Open Gateway Initiative (OGI) APIs, expose them to Firebase developers, and enable a new way to implement electronic phone number verification (ePNV). You’ll see lots of amazing progress by the telecom community! 

Here are the demos that will be on display:

• Creating AI Agents: Share your requirements on the spot and we will use Gemini on Vertex AI and AI Studio to build your agent within minutes.

• Employee Productivity with Agentspace: Discover how Agentspace leverages Gemini’s advanced reasoning and Google-quality search to put all your enterprise knowledge at your employees’ fingertips, dramatically boosting productivity and unlocking expertise.

• CX with Amdocs: Discover how Google Cloud’s collaboration with Amdocs on amAIZ, powered by Gemini’s generative AI, is revolutionizing both customer experience management and autonomous network operations in telecom.

• Developing AI Apps: Discover how developers can leverage a suite of Google tools, including Flutter, Firebase, Gemini, and Vertex AI, to create intelligent app experiences that dramatically improve the customer journey, featuring AI-powered phone-plan selection.

• User Identity with Firebase and OGI: Experience a more secure and reliable user identity service built on Firebase and the Open Gateway Initiative, offering developers expanded reach and enhanced authentication capabilities.

• Ericsson Networks with AI: Ericsson has a long-standing relationship with Google Cloud. In this demonstration, we will showcase how the two companies are reimagining 5G networks with AI agents driven by intent-based service management using Vertex AI.

• Load Balance for Inferencing: Experience a gamified challenge against our AI-powered Cloud Load Balancer, then dive into the Google Cloud console to see how you can use custom metrics to optimize GPU utilization and achieve lightning-fast inference response times.

• AI for Cybersecurity: This demo showcases how Google Threat Intelligence provides visibility into threats and how it delivers detailed and timely threat intelligence to security teams around the world. Gemini in Threat Intelligence acts as a force multiplier, immediately surfacing threats most relevant to your unique risk profile and summarizing complex threat reports. 

• AI Network Operations: This AI network operations demo shows advanced AI/ML-powered tools and capabilities to help CSPs understand the impacts of network issues, detect them, and resolve them faster through automated end-to-end correlation and root-cause analysis. The solution leverages several native Google Cloud services.

• Nokia Network on Demand: This demo features a collaboration between Nokia AVA and Google Cloud to show customers how Nokia delivers an innovative approach to energy efficiency by combining AI-driven insights with gamification, engaging users and driving sustainability in telecom networks.

• Gen AI Field Operations: Last but not least, discover how Google Cloud is helping CSPs supercharge field-agent productivity, elevate service quality, and reduce costs through the power of multi-modal gen AI and democratized data access.

And if you are up for some fun, come by and get behind the wheel of the Formula E simulator, powered by Google Cloud AI. Experience the same cutting-edge technology that’s transforming motorsport through our partnership with Formula E. Feel firsthand how AI and data are revolutionizing racing, and the future of telecommunications.

Engage with the technologies reshaping the industry and discover how Google Cloud can help you drive innovation. Don’t just hear about the future of telecom — build it with us at Mobile World Congress 2025, March 3-6 in Barcelona Hall 2 Booth #2H40. The future of telecommunications awaits!

AI is driving unprecedented change and evolution across every industry, and the telecommunications sector is at a particularly interesting crossroads: The industry is standardizing on 5G; data is growing exponentially; and customer expectations are quickly changing. Leading communication service providers (CSPs) are working with Google Cloud to adopt AI and lead the industry forward in terms of innovation.

At Google Cloud, we believe that AI, harnessed boldly and responsibly, has the potential to transform how CSPs operate on planetary scale — and we’re deeply committed to delivering the technologies, tools, and resources that help them do so. From building a strong data foundation, to optimizing network operations, to enhancing customer experiences, our AI-powered solutions and strategic partnerships enable CSPs to innovate and grow their businesses.

Heading into Mobile World Congress 2025 next week, let’s look at the announcements we’ve made with leaders in this industry to build AI-driven telecoms. 

Data migrations are key to adopting AI 

CSPs typically need to move their data into one place to use AI effectively. Google Cloud helps with this by offering tools like BigQuery and Looker, which can store and organize vast amounts of data, creating scalable data lakes and data oceans that can be analyzed by data scientists and businesspeople alike. Our open approach allows integration with existing systems, streamlining data management and getting benefits faster. Combined with our AI infrastructure, CSPs can gain actionable insights, create personalized experiences, and build new services.

DNA, part of Telenor, for instance, announced it has selected Google Cloud to help accelerate its transition to the cloud and support the delivery of fast and reliable services to its customers, allowing it to gradually migrate its on-premises workloads to Google Cloud and pave the way for generative AI adoption. 

Vodafone Italy also announced it has modernized its data architecture by building a new AI-ready platform on Google Cloud called Nucleus. This gives the company enhanced process efficiency, scalability and real-time data processing. In partnership with Amdocs and Google Cloud, Vodafone re-engineered its data pipelines to optimize operational workflows and support business-critical functions. The migration to Nucleus was achieved in only 12 months without interruption or issues. The company’s modern architecture delivers key benefits, including enhanced agility, more efficient reporting, cost optimization, and improved real-time processing.

From reactive to proactive network operations with AI

Telecommunications providers face the constant challenge of maintaining optimal network performance and reliability in a dynamic and increasingly complex environment. Traditional reactive network-management approaches can lead to service disruptions, increased operational costs, and diminished customer satisfaction. Google Cloud’s AI-powered platform, which includes Vertex AI and Network Intelligence Center, provides intelligent automation for telecommunications providers, enabling them to analyze network data with BigQuery to identify anomalies, predict outages, and optimize traffic flow in real time. This shift from reactive to proactive management allows CSPs to not only improve reliability and reduce costs, but also create self-healing networks that dynamically adapt to changing conditions. 

Bell Canada, for example, announced its using Google Cloud’s AI technologies to automate network issue detection and resolution with its new AI operations (AI Ops) solution. Partnering with Google Cloud has enabled Bell to simultaneously boost software delivery productivity by 75%, accelerating time to market, and reduce customer-reported issues by 25%, significantly enhancing customer experience and solution reliability. And Deutsche Telekom is using Google Cloud’s Gemini 2.0 in Vertex AI to develop a network AI agent called RAN Guardian, which can analyze network behavior, detect performance issues, and implement corrective actions to improve network reliability and customer experiences. This collaboration aims to create autonomous, self-healing networks that proactively optimize RAN performance and reduce operational costs.

Then there’s Amdocs, a leading provider of software and services to communications and media companies, which in partnership with Google Cloud, announced recently that it has launched a new Network AIOps solution to help 5G network providers automate complex network operations, enhance service reliability, and improve customer experiences.

Through AI-powered insights, CSPs can deliver superior service quality while streamlining their operations, ultimately driving greater customer satisfaction and loyalty.

Empowering field technicians with AI

CSPs often face challenges in field service operations, such as inefficient dispatching and a lack of real-time diagnostics, leading to longer resolution times and increased customer downtime. Google Cloud’s platform, including a multimodal field-tech AI assistant and Customer Engagement Suite, empowers field technicians with real-time diagnostics, predictive maintenance, optimized routing, and agentic workflows powered by Gemini. This leads to fewer worker deployments and faster issue resolution. AI-driven diagnostics proactively identify and resolve potential issues. Predictive maintenance models anticipate equipment failures, enabling scheduled repairs. Agentic workflows and Gemini provide technicians with a comprehensive knowledge base for faster troubleshooting. Smart routing helps ensure efficient dispatch. Together, these AI-powered services help create a streamlined, customer-centric field service operation, providing faster and more effective support.As an example, TELUS’ NEO Assistant, an innovative multimodal AI copilot that leverages advanced machine learning and AI technologies like Google Cloud’s Gemini model, provides field technicians with instant access to critical information and streamlines workflows. Since its launch, NEO Assistant has been adopted by three-quarters of TELUS field technicians, contributed to the creation of nearly 7,000 jobs, and significantly increased efficiency by allowing tasks to be completed in under a minute.

Delighting customers with AI-driven insights

CSPs are facing increasing pressure to deliver exceptional customer service while managing costs and navigating complex customer journeys across multiple channels. Google Cloud’s AI platform, featuring the Customer Engagement Suite, Vertex AI, and Vertex AI Search, empowers CSPs to create exceptional customer experiences. By harnessing AI-driven insights about customer behavior, CSPs can anticipate needs, proactively address issues, and deliver personalized recommendations and seamless interactions, ultimately boosting satisfaction and revenue.

Chunghwa Telecom is using Google Cloud’s AI to enhance strategic decision-making and data analysis, exemplified by tools like NotebookLM Enterprise for strategic decision-making and data analysis, and a Customer Agent using Gemini 2.0 that slashes response times and is projected to reduce billing-related calls by 25% annually. Additionally, automating roaming charge verification with Document AI streamlines operations and reduces costs by eliminating manual invoice processing, leading to potential cost savings for both the company and subscribers.

Glance leverages Google’s suite of AI models, including Gemini and Imagen, to deliver visually captivating and AI-powered experiences that position consumers as the ‘heroes’ of their digital world. GlanceAI offers an immersive shopping experience from inferred user interests and images uploaded by consumers, drawing consumers in and helping them visualize themselves in various styles so they can make real-time purchase decisions. Here, AI transforms a quick glance at a device into an interactive shopping opportunity, whether on a smartphone lock screen or an ambient TV screen.

Unlocking new revenue streams through APIs

CSPs face high network deployment costs and pressure to innovate, leading them to explore APIs and engage developers to monetize their networks. Google Cloud’s developer-centric platform simplifies network asset monetization by providing necessary tools and infrastructure. Initiatives like GSMA Open Gateway foster collaboration and developer access to network capabilities.

Built on the Open Gateway vision, yesterday we announced a new Firebase phone number verification service, providing more than three million Firebase developers with access to critical network CSP APIs such as phone number acquisition and verification. We’re partnering with Deutsche Telekom, Orange, Telefónica, T-Mobile, Telenor, Verizon, Vodafone and others to offer this service to improve developer and end-user experiences, and strengthen security by mitigating fraud — creating new revenue opportunities for CSPs and developers alike. 

The future of AI in telecommunications

MWC 2025 marks a pivotal moment for AI-driven telecom innovation, and Google Cloud is at the forefront of this transformation. As CSPs continue to embrace AI in the coming months, our platform ensures they unlock new efficiencies, elevate customer experiences, and drive business growth. 

For more information, see our press page or visit Google Cloud at Mobile World Congress (MWC) from March 3-6 in Hall 2, Booth #2H40.

The telecommunications industry has always been on the leading edge of technology — but rarely has it been changing as quickly as it is today. Increased 5G adoption, the spread of edge computing, new monetization models, and growing consumer expectations are presenting opportunities and hurdles in equal measure. 

Since its inception, Google Cloud has made supporting telecommunications innovation a priority — and a key part of that has been fostering a partner ecosystem that can help meet the constant evolution that communication service providers (CSPs) are undergoing. 

The AI era has made such collaboration more important than ever, so ahead of Mobile World Congress (MWC), we’re showcasing the breadth of our ecosystem of telecom partners and the ways they are enabling customer success with Google Cloud. Our partners — spanning global system integrators (GSIs), specialized ISVs, and network equipment providers (NEPs) — are using AI to help CSPs optimize network performance, personalize customer experiences, automate operations, and identify new revenue streams. 

Improving network operations 

AI is enabling CSPs to move from reactive to proactive network management, with capabilities that span predictive maintenance, automated troubleshooting, and real-time optimization. Our partners are bringing the next-generation network to life by using Google Cloud’s Vertex AI and Network Intelligence Center. Together, we’re providing the necessary tools and technologies for building and deploying AI solutions across the modern network.

• Amdocs yesterday announced Network AIOps, an AI-powered solution built on Google Cloud’s Vertex AI and BigQuery, that can help telecommunications companies automate and optimize their 5G network operations, thus enhancing reliability and customer experiences.

• CARTO, a geospatial analytics platform provider, uses BigQuery to enable telecoms to visualize and analyze massive location-based datasets, leading to more effective network planning and optimized service delivery. 

• Ericsson, a leading telecommunications company, is building AI agent-driven network slice instantiation and assurance for telecom customers with Google Cloud’s AI. 

• Fortinet, a cybersecurity leader, collaborates with Google Cloud to enhance real-time threat detection and response capabilities with AI, providing network security and resilience for telecom providers.

• MATRIXX, a leading provider of cloud-native business support services (BSS) solutions, is using Google Cloud’s AI to simplify and automate tasks, helping telecoms reduce costs and deliver a better customer experience.

• Nokia is collaborating with Google Cloud to enhance its energy saving application, wherein telcos can include their subscribers as part of their energy savings goals.

Optimizing IT transformation and productivity 

Improving IT systems is important for telecom companies to work faster and smarter. Using tools like BigQuery, Looker, and Vertex AI, our partners are helping telecom providers automate tasks and optimize the value they get from data and AI projects while improving overall IT capabilities.

• BMC Helix Software uses Vertex AI to enhance its service assurance capabilities, helping telecoms streamline IT processes, predict and prevent outages, and automate resolutions.
• Datatonic is building solutions for telecoms with Vertex AI and Google Cloud’s generative AI technology, helping it better understand customer behavior, personalize service recommendations, enhance network analytics, and boost employee productivity.
• Zinkworks is using Vertex AI, Gemini models, and additional Google Cloud technology to develop and implement solutions that automate network operations, significantly reducing operating costs and enhancing efficiency for telecom workers in the field.

Enhancing the customer experience

AI helps CSPs better understand customer needs, personalize interactions, and provide proactive support. Partners are using Vertex AI, BigQuery, Customer Engagement Suite, and Telecom Subscriber Insights to develop solutions that deliver personalized and engaging experiences to telecom customers.

• commercetools is helping telco businesses build agile, scalable, and flexible commerce on top of Google Cloud.

• Pega is an AI decision-making and workflow automation platform that helps CSPs and other enterprises transform their operations, personalize customer engagement, and continuously adapt to change with the help of Google AI.

• Sprinklr offers a unified customer experience management platform powered by Google AI that helps front-office teams and contact center agents to make every customer experience extraordinary.

• Zeotap offers a secure customer data platform (CDP) on Google Cloud that empowers telco and other brands to integrate and orchestrate customer data, enabling targeted marketing while protecting customer privacy, all designed for now and for when cookies are no longer used.

Revamping monetization tools

AI plays a key role in optimizing pricing strategies, personalizing offers, and automating billing processes. Partners are using AI and network APIs to create more effective monetization models with solutions that can analyze customer data to identify preferences, predict churn, and recommend optimal pricing strategies. 

• Aduna, a venture between Ericsson and leading CSPs, is simplifying global access to network APIs, making it easier for Google Cloud developers to build new applications leveraging mobile networks to accelerate digital transformation across industries.

• Beyond Now offers an AI-powered digital marketplace for CSPs and technology providers to co-create and bundle partner solutions, helping support their customers’ digital transformation journeys. 

• Eureka.AI provides actionable market and risk intelligence products for industry verticals to help CSPs modernize their data monetization.

• Glide Identity provides a platform built on Google Cloud for identity and anti-fraud products, helping telecoms create new revenue opportunities and delight users. 

• Kloudville offers a marketplace powered by Google Cloud that enables telecoms to offer digital products and services to their SMB customers, creating new revenue opportunities and enhancing the customer experience. 

• Nokia’s Network-as-Code platform simplifies the use of advanced network capabilities by exposing them as developer-friendly interfaces, enabling applications to work seamlessly across multiple public and private networks, and fostering new value creation and network monetization.

Enabling customer success with services partners

Services partners play a critical role in providing customers with the expertise and support needed to plan, deploy, and optimize AI projects. Many of these partners have launched services specifically for telecoms and are continuing to demonstrate their proven ability to help customers transform with AI and other Google Cloud technology. 

• Accenture empowers CSPs to reinvent customer engagement, drive revenue growth, and enhance network operations through AI, machine learning, and gen AI agents, while prioritizing cost reduction and sustainability.

• Capgemini helps CSPs develop autonomous network capabilities and next-generation field operations, as well as strength in AI-powered marketing, customer care, and lifetime-value management. 

• Prodapt is helping the global telecom and technology industry use Google Cloud products, including generative AI tools and solutions, to modernize technology infrastructure, fast-track data migration, improve productivity, and make operations seamless and cost-efficient.

• TCS offers innovative end-to-end business solutions that encompass agentic AI frameworks for CSPs across customer experience, customer service, field service, and network operations; it also creates tailored offerings such as TCS CX Transformer for Telcos, which combines its vast industry expertise and Google Cloud’s data analytics and AI technology. 

• Tech Mahindra partners with CSPs to scale at speed by enabling them to simplify and modernize their business and technology, as well as to monetize their assets for greater revenue growth, and all of this is powered by AI.

Supporting sustainable practices

On top of navigating evolving customer demands and technology needs, CSPs are juggling pressure to decarbonize and comply with ever-changing global regulations. AI is helping telecom organizations optimize energy consumption, reduce waste, and minimize their carbon footprint. 

• Geotab provides vehicle and driver behavior data services for more than 55,000 fleets and 4.7 million vehicles around the world. Using Google Cloud’s data analytics and machine learning, Geotab helps customers improve fleet productivity, optimize fleet management, and achieve strong compliance.

• Watershed helps companies manage climate and ESG data, producing audit-ready metrics for voluntary and regulatory reporting using Google Cloud tools.

Google Cloud is committed to fostering a collaborative and mutually beneficial ecosystem where our partners can thrive and contribute to the success of our CSP customers. We believe that together, we can empower the telecom industry to embrace the full potential of digital transformation and shape the future of connectivity.

Visit Google Cloud at Hall 2, Booth #2H40 at Mobile World Congress (MWC) to learn more about how Google Cloud and our partners can help you transform your telecommunications business. We hope to see you in Barcelona!

Telecommunications companies face mounting pressure to reduce operational costs, enhance network resiliency, and deliver exceptional customer experiences. Earlier this week, Amdocs and Google Cloud announced a new network AI operations solution — Amdocs Network AIOps — that uses data to help communication service providers improve their networks and customer service. 

This solution is designed to make networks more self-sufficient and efficient, leading to a better experience for customers. This is the first step for telecoms in evolving to a fully autonomous network. In this blog, we wanted to explore more deeply how Amdocs achieved these autonomous innovations using their technology and Google Cloud’s AI and cloud solutions.

Network analytics and automation

Amdocs Network AIOps is a comprehensive platform designed to address the complexities of modern telecom networks. It’s built on massive quantities of telemetry data using Google Cloud’s data and AI capabilities, including BigQuery and Vertex AI, to provide a comprehensive observabilities and automated AI solution. 

The solution is a network operations AI framework that consists of three layers — Observe, Decide, and Act — that leverages AI and ML to automate network operations, optimize performance, and enhance service reliability.

Key capabilities behind the framework of the Amdocs Network AIOps Solution include:

• Data ingestion and mediation (Observe layer): The platform ingests data from diverse sources across the network, including network elements, operations support systems (OSS) and business support systems (BSS), and probes. 

• AI-driven insights (Decide layer): With tight integration into Google Cloud’s BigQuery, Vertex AI, and Gemini services, Amdocs Network AIOps provides actionable insights in support of predictive analytics and root-cause analysis. This integration empowers operators to harness the power of cloud-based AI/ML capabilities for enhanced network automation enabling proactive network management, thereby reducing the mean-time-to-resolution (MTTR) for incidents.

• Automated workflows (Act layer): The platform automates routine tasks such as network configuration, performance optimization, and incident remediation. This frees up valuable resources and reduces the risk of human error.

• Closed-loop automation (Act layer): Amdocs Network AIOps employs a closed-loop approach, where the system continuously learns from previous actions and data patterns to improve its accuracy and efficiency over time.

Actionable network insights built on AI

Telecom operators are already using Amdocs Network AIOps to drive insights in support of a variety of networking use cases such as anomaly detection, root cause analysis, and predictive maintenance. 

Network downtime is a major concern for telecom operators, leading to lost revenue and dissatisfied customers. The Network AIOps solution significantly reduces the downtime by correlating data from a variety of network sources and using the predictive power of Vertex AI’s no-code or low-code models. 

By training ML models on historical network data, including performance metrics, fault logs, and environmental factors, this solution can accurately predict the likelihood of future network failures. This allows operators to proactively schedule maintenance, replace aging equipment, and optimize network configurations to minimize downtime and ensure uninterrupted service.

Google Cloud services such as BigQuery and Vertex AI are foundational to Amdocs Network AIOps. These services enable transformation and ingestion of petabytes of data, and near-real-time predictive analytics, anomaly detection and correlation, helping telecommunications businesses in meeting the goals of greater network efficiency and reliability and improved customer satisfaction.

Your intelligent network copilot

Managing the complexity of 5G networks can be overwhelming for even the most skilled network engineers. The Amdocs Network Platform for Operations (Act layer), integrated with a multimodal gen-AI-powered live network assistant, provides a single interface for comprehensive network monitoring and management across the full stack of hardware infrastructure, containers, virtualization software, and network applications (RAN, 5G core, IMS core) in a multi-vendor environment.

This gen-AI-powered assistant acts as an intelligent agent for network engineers, offering:

• Proactive alerts and insights: The assistant continuously monitors the network, analyzing data from BigQuery data lake to identify potential issues and correlate events from the infrastructure stack and multi-vendor workload, and use it to proactively alert engineers.

• Automated troubleshooting and remediation: By leveraging its deep knowledge base and AI capabilities, the gen AI assistant can automatically diagnose common network problems and even take corrective actions, such as restarting services or rerouting traffic

• Multimodal and natural language interaction: Network engineers can interact with the assistant using natural language phrases and sentences, as well as audio, video, and images — making it easier to ask questions, seek recommendations, and receive clear, concise answers.

Embrace the AI-Powered future of telecom

The joint Network AIOps solution from Amdocs and Google Cloud delivers a wide range of benefits for telecom operators:

• Reduced operational costs: Automation, predictive maintenance, and intelligent resource allocation lead to significant cost savings.

• Improved network resiliency: Proactive issue identification and automated remediation help minimize downtime and enhance network stability.

• Enhanced customer experience: AI-powered optimization provides an integrated and reliable user experience, fostering customer satisfaction and loyalty.

• Increased efficiency: The gen AI Network Assistant helps network engineers work more efficiently, freeing up their time for strategic initiatives.

The collaboration between Amdocs and Google Cloud brings together the best of both worlds: deep telecom expertise and cutting-edge AI capabilities. The joint Network AIOps solution lets telecom operators embrace the AI-powered future, optimize their 5G networks, and deliver exceptional customer experiences.

To address the challenges of AI-driven 5G networks, please visit Amdocs Network AIOps Solution website. And you can learn more about how Google Cloud is partnering with communication service providers to deliver AI-driven telecom at Google Cloud’s industry site.