Google Cloud

Cloud SQL, Google Cloud’s fully managed database service for PostgreSQL, MySQL, and SQL Server workloads, offers strong availability SLAs, depending on which edition you choose: a 99.95% SLA, excluding maintenance for Enterprise edition; and a 99.99% SLA, including maintenance for Enterprise Plus. In addition, Cloud SQL offers numerous high availability and scalability features that are crucial for maintaining business continuity and minimizing downtime, especially for mission-critical databases. 

These features can help address some common database deployment challenges:

1. Combined read/write instances: Using a single instance for both reads and writes creates a single point of failure. If the primary instance goes down, both read and write operations are impacted. In the event that your storage is full and auto-scaling is disabled, even a failover would not help. 

2. Downtime during maintenance: Planned maintenance can disrupt business operations.

3. Time-consuming scaling: Manually scaling instance size for planned workload spikes is a lengthy process that requires significant planning.

4. Complex cross-region disaster recovery: Setting up and managing cross-region DR requires manual configuration and connection string updates after a failover.

In this blog, we show you how to maximize your business continuity efforts with Cloud SQL’s high availability and scalability features, as well as how to use Cloud SQL Enterprise Plus features to build resilient database architectures that can handle workload spikes, unexpected outages, and read scaling needs.

Architecting a highly available and robust database 

Using the Cloud SQL high availability feature, which automatically fails over to a standby instance, is a good starting point but not sufficient: scenarios such as storage full issues, regional outages, or failover problems can still cause disruptions. Separating read workloads from write workloads is essential for a more robust architecture.

A best-practice approach involves implementing Cloud SQL read replicas alongside high availability. Read traffic should be directed to dedicated read-replica instances, while write operations are handled by the primary instance. You can enable high availability either on the primary, the read replica(s), or both, depending on your specific requirements. This separation helps ensure that the primary can serve production traffic predictably, and that read operations can continue uninterrupted via the read replicas even when there is downtime. 

Below is a sample regional architecture with high availability and read-replica enabled.

You can deploy this architecture regionally across multiple zones or extend it cross-regionally for disaster recovery and geographically-distributed read access. A regional deployment with a highly available primary and a highly available read replica that spans three availability zones provides resilience against zonal failures: Even if two zones fail, the database remains accessible for both read and write operations after failover. Cross-region read replicas enhance this further, providing regional DR capabilities.

Cloud SQL Enterprise Plus features

Cloud SQL Enterprise Plus offers significant advantages for performance and availability:

1. Enhanced hardware: Run databases on high-performance hardware with up to 128 vCPUs and 824GB of RAM.

2. Data cache: Enable data caching for faster read performance.

3. Near-zero downtime operations: Experience near-zero downtime maintenance and sub-second (<1s) downtime for instance scaling.

4. Advanced disaster recovery: Streamline disaster recovery with failover to cross-region DR-Replica and automatic reinstatement of the old primary. The application can still connect using the same write endpoint, which is automatically assigned to the new primary after failover. 

Enterprise Plus edition addresses the previously mentioned challenges:

1. Improved performance: Benefit from higher core-to-memory ratios for better database performance.

2. Faster reads: Data caching improves read performance for read-heavy workloads. Read-cache can be enabled in the primary, the read-replica, or both as needed. 

3. Easy scaling: Scale instances quickly with minimal downtime (sub-second) to handle traffic spikes or planned events. Scale the instance down when traffic is low with sub-second downtime. 

4. Minimized maintenance downtime: Reduce downtime during maintenance to less than a second and provide better business continuity. 

5. Handle regional failures: Easily fail over to a cross-region DR replica, and Cloud SQL automatically rebuilds your architecture as the original region recovers. This lessens the hassle of DR drills and helps ensure application availability. 

6. Automatic IP address re-pointing: Leverage the write endpoint to automatically connect to the current primary after a switchover or failover and you don’t need to make any IP address changes on the application end. 

To test out these benefits quickly, there’s an easy, near-zero downtime upgrade option from Cloud SQL Enterprise edition to Enterprise Plus edition.

Planned maintenance best practices

While Cloud SQL Enterprise Plus offers near-zero downtime planned maintenance, following these best practices can further enhance business continuity: 

1. Staging environment testing: To identify potential issues, use the maintenance timing feature to deploy maintenance to test/staging environments at least a week before production.

2. Read-replica maintenance: Apply self-service maintenance to one of the read replicas before the primary instance to avoid simultaneous downtime for read and write operations. Make sure that the primary and other replicas are updated shortly afterwards, as we recommend maintaining the same maintenance version in the primary as well as all the other replicas. 

3. Maintenance window: Always configure a maintenance window during off-peak hours to control when maintenance is performed. 

4. Maintenance notifications: Opt in to maintenance notifications to make sure you receive an email at least one week before scheduled maintenance. 

5. Reschedule maintenance: Use the reschedule maintenance feature if a maintenance activity conflicts with a critical business period.

6. Deny maintenance period: Use the deny maintenance period feature to postpone maintenance for up to 90 days during sensitive periods. 

By combining these strategies, you can build highly available and scalable database solutions in Cloud SQL, helping to ensure your business continuity and minimize downtime. Refer to the maintenance FAQ for more detailed information.

What is the true cost of enterprise AI?

As a technology leader and a steward of company resources, understanding these costs isn’t just prudent – it’s essential for sustainable AI adoption. To help, we’ll unveil a comprehensive approach to understanding and managing your AI costs on Google Cloud, ensuring your organization captures maximum value from its AI investments.

Whether you’re just beginning your AI journey or scaling existing solutions, this approach will equip you with the insights needed to make informed decisions about your AI strategy.

Why understanding AI costs matters now

Google Cloud offers a vast and ever-expanding array of AI services, each with its own pricing structure. Without a clear understanding of these costs, you risk budget overruns, stalled projects, and ultimately, a failure to realize the full potential of your AI investments. This isn’t just about saving money; it’s about responsible AI development – building solutions that are both innovative and financially sustainable.

Breaking down the Total Cost of Ownership (TCO) for AI on Google Cloud

Let’s dissect the major cost components of running AI workloads on Google Cloud:

Let’s estimate costs with an example

Let’s illustrate this with a hypothetical, yet realistic, generative AI use case: Imagine you’re a retail customer with an automated customer support chatbot.

Scenario: A medium-sized e-commerce company wants to deploy a chatbot on their website to handle common customer inquiries (order status, returns, product information and more). They plan to use a pre-trained language model (like one available through Vertex AI Model Garden) and fine-tune it on their own customer support data.

Assumptions:

• Model: Fine-tuning a low latency language model (in this case we will use Gemini 1.5 Flash).

• Training data: 1 million customer support conversations (text data).

• Traffic: 100K chatbot interactions per day.

• Hosting: Vertex AI Prediction for serving the model.

• Fine-tuning frequency: Monthly.

Cost estimation

As the retail customer in this example, here’s how you might approach this. 

1. First, discover your model serving cost:

• Vertex AI Prediction (Gemini 1.5 Flash for Chat) pricing is modality-based pricing so in this case since our input and output is text, the usage unit will be characters. Let’s assume an average of 1000 input characters and 500 output characters per interaction.
• Cost per 1M characters input: $0.0375.
• Cost per 1M characters output: $0.15
• Input cost per day: 100,000 interactions * 1000 characters * $0.0375 / 1000000 = $3.75
• Output cost per day: 100,000 interactions * 500 characters * $0.15 / 1000000 characters = $7.5
• Total model serving cost per day: $11.25
• Total model serving cost per month (~30 days): ~$337

2. Second, identify your training and tuning costs:

In this scenario, we aim to enhance the model’s accuracy and relevance to our specific use case through fine-tuning. This involves inputting a million past chat interactions, enabling the model to deliver more precise and customized interactions.

• Cost per training tokens: $8 / M tokens
• Cost per training characters: $2 / M characters (where each token approximately equates to 4 characters)
• Tuning cost (first month): 1,000,000 conversation (training data) * 1500 characters (input + output) * 2 /1,000,000 = $3,000
• Tuning cost (subsequent month): 100,000 conversation (new training data) * 1500 characters (input + output) * 2 /1,000,000 = $300

3. Third, understand the cloud hosting costs:

Since we’re using Vertex AI Prediction, the underlying infrastructure is managed by Google Cloud. The cost is included in the per-request pricing. However, if we are self-managing the model on GKE or Compute Engine, we’d need to factor in VM costs, GPU/TPU costs (if applicable), and networking costs. For this example, we assume this is $0, as it is part of Vertex AI cost.

4. Fourth, define the training data storage and adapter layers costs:

The infrastructure costs for deploying machine learning models often raise concerns, but the data storage components can be economical at moderate scales. When implementing a conversational AI system, storing both the training data and the specialized model adapters represents a minor fraction of the overall costs. Let’s break down these storage requirements and their associated expenses.

• 1M conversations, assuming an average size of 5KB per conversation, would be roughly 5GB of data.
• Cloud Storage cost for 5GB is negligible: $0.1 per month.
• Adapter layers (fine-tuned model weights) might add another 1GB of storage. This would still be very inexpensive: $0.02 per month.
• Total storage cost per month: < $1/month

5. Fifth, consider the application layer and setup costs: 

This depends heavily on the specific application. In this case we are using Cloud Run Functions and Logging. Cloud Run to handle pre- and post-processing of chatbot requests (e.g., formatting, database lookups). In this case let’s assume we use request-based billing so we are only charged when it processes the request. In this example we are processing 3M requests per month (100K * 30) and assuming 1 sec for average execution time: $14.30

• Cloud Logging and Monitoring for tracking chatbot performance and debugging issues. Let’s estimate 100GB of logging volume (which is on higher end) and retaining the logs for 3 months: $28

Total application layer cost per month:~ $40

6. Finally, incorporate the Operational support cost:

This is the hardest to estimate, as it depends on the internal team’s size and responsibilities. Let’s assume a conservative estimate of 5 hours per week of an engineer’s time dedicated to monitoring and maintaining the chatbot, at an hourly rate of $100.

• Total operational support cost per month: 5 hours/week * 4 weeks/month * $100/hour = $2000
• Total estimated monthly cost (First month):
• $ 340 (Serving) + $3000 (Training) + $1 (Storage) + $40 (Application) + $2000 (Operational) = $5,381
• Total estimated monthly cost (Subsequent months):
• $340 (Serving) + $300 (Training) + $1 (Storage) + $40 (Application) + $2000 (Operational) = $2,681

You can find the full estimate of cost here. Note that this does not include tuning and operational cost as it is not available in pricing export yet. 

Once you have a good understanding of your AI costs, it is important to develop an optimization strategy that encompasses infrastructure choices, resource utilization, and monitoring practices to maintain performance while controlling expenses. By understanding the various cost components and leveraging Google Cloud’s tools and resources, you can confidently embark on your AI journey. Cost management isn’t a barrier; it’s an enabler. It allows you to experiment, innovate, and build transformative AI solutions in a financially responsible way. 

Get started

• Start understanding your AI costs today: Explore the Google Cloud Pricing Calculator and the Vertex AI Pricing Page.

• Learn more at Google Cloud Next: Register for the Google Next session on AI Investment to Impact: Unlocking Sustainable ROI with Google Cloud.

• Engage Google Cloud for expert guidance: Get expert help to design cost effect AI architectures, contact Google Cloud Consulting or PSO.

Written by: Joshua Goddard

Executive Summary

• Rosetta 2 is Apple’s translation technology for running x86-64 binaries on Apple Silicon (ARM64) macOS systems.

• Rosetta 2 translation creates a cache of Ahead-Of-Time (AOT) files that can serve as valuable forensic artifacts.

• Mandiant has observed sophisticated threat actors leveraging x86-64 compiled macOS malware, likely due to broader compatibility and relaxed execution policies compared to ARM64 binaries.

• Analysis of AOT files, combined with FSEvents and Unified Logs (with a custom profile), can assist in investigating macOS intrusions.

Introduction

Rosetta 2 (internally known on macOS as OAH) was introduced in macOS 11 (Big Sur) in 2020 to enable binaries compiled for x86-64 architectures to run on Apple Silicon (ARM64) architectures. Rosetta 2 translates signed and unsigned x86-64 binaries just-in-time or ahead-of-time at the point of execution. Mandiant has identified several new highly sophisticated macOS malware variants over the past year, notably compiled for x86-64 architecture. Mandiant assessed that this choice of architecture was most likely due to increased chances of compatibility on victim systems and more relaxed execution policies. Notably, macOS enforces stricter code signing requirements for ARM64 binaries compared to x86-64 binaries running under Rosetta 2, making unsigned ARM64 binaries more difficult to execute. Despite this, in the newly identified APT malware families observed by Mandiant over the past year, all were self-signed, likely to avoid other compensating security controls in place on macOS.

The Rosetta 2 Cache

When a x86-64 binary is executed on a system with Rosetta 2 installed, the Rosetta 2 Daemon process (oahd) checks if an ahead-of-time (AOT) file already exists for the binary within the Rosetta 2 cache directory on the Data volume at /var/db/oah/<UUID>/. The UUID value in this file path appears to be randomly generated on install or update. If an AOT file does not exist, one will be created by writing translation code to a .in_progress file and then renaming it to a .aot file of the same name as the original binary. The Rosetta 2 Daemon process then runs the translated binary.

The /var/db/oah directory and its children are protected and owned by the OAH Daemon user account _oahd. Interaction with these files by other user accounts is only possible if System Integrity Protection (SIP) is disabled, which requires booting into recovery mode.

The directories under /var/db/oah/<UUID>/ are binary UUID values that correspond to translated binaries. Specifically, these binary UUID values are SHA-256 hashes generated from a combination of the binary file path, the Mach-O header, timestamps (created, modified, and changed), size, and ownership information. If the same binary is executed with any of these attributes changed, a new Rosetta AOT cache directory and file is created. While the content of the binaries is not part of this hashing function, changing the content of a file on an APFS file system will update the changed timestamp, which effectively means content changes can cause the creation of a new binary UUID and AOT file. Ultimately, the mechanism is designed to be extremely sensitive to any changes to x86-64 binaries at the byte and file system levels to reduce the risk of AOT poisoning.

The Rosetta 2 cache binary UUID directories and the AOT files they contain appear to persist until macOS system updates. System updates have been found to cause the deletion of the cache directory (the Random UUID directory). After the upgrade, a directory with a different UUID value is created, and new Binary UUID directories and AOT files are created upon first launch of x86-64 binaries thereafter.

Translation and Universal Binaries

When universal binaries (containing both x86-64 and ARM64 code) are executed by a x86-64 process running through Rosetta 2 translation, the x86-64 version of these binaries is executed, resulting in the creation of AOT files.

In a Democratic People’s Republic of Korea (DPRK) crypto heist investigation, Mandiant observed a x86-64 variant of the POOLRAT macOS backdoor being deployed and the attacker proceeding to execute universal system binaries including ping, chmod, sudo, id, and cat through the backdoor. This resulted in AOT files being created and provided evidence of attacker interaction on the system through the malware (Figure 5).

In some cases, the initial infection vector in macOS intrusions has involved legitimate x86-64 code that executes malware distributed as universal binaries. Because the initial x86-64 code runs under Rosetta 2, the x86-64 versions of malicious universal binaries are executed, leaving behind Rosetta 2 artifacts, including AOT files. In one case, a malicious Python 2 script led to the downloading and execution of a malicious universal binary. The Python 2 interpreter ran under Rosetta 2 since no ARM64 version was available, so the system executed the x86-64 version of the malicious universal binary, resulting in the creation of AOT files. Despite the attacker deleting the malicious binary later, we were able to analyze the AOT file to understand its functionality.

Unified Logs

The Rosetta 2 Daemon emits logs to the macOS Unified Log; however, the binary name values are marked as private. These values can be configured to be shown in the logs with a custom profile installed. Informational logs are recorded for AOT file lookups, when cached AOT files are available and utilized, and when translation occurs and completes. For binaries that are not configured to log to the Unified Log and are not launched interactively, in some cases this was found to be the only evidence of execution within the Unified Logs. Execution may be correlated with other supporting artifacts; however, this is not always possible.

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Aot lookup request for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Translating image <private> -> <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Translation finished for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Aot lookup request for <private>

0x21b1afc Info 0x0 1596 0 oahd: <private>(1880): 
Using cached aot <private> -> <private>

Figure 3: macOS Unified Logs showing Rosetta lookups, using cached files, and translating with private data disabled (default)

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Aot lookup request for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Translating image /Users/Mandiant/my_binary -> 
/var/db/oah/237823680d6bdb1e9663d60cca5851b63e79f6c
8e884ebacc5f285253c3826b8/1c65adbef01f45a7a07379621
b5800fc337fc9db90d8eb08baf84e5c533191d9/my_binary.in_progress

0x2ec304 Info 0x0 668 0 oahd: my_binary (Re(34180): 
Translation finished for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary(34180): 
Aot lookup request for /Users/Mandiant/my_binary

0x2ec304 Info 0x0 668 0 oahd: my_binary(34180): 
Using cached aot /Users/Mandiant/my_binary -> 
/var/db/oah/237823680d6bdb1e9663d60cca5851b63e
79f6c8e884ebacc5f285253c3826b8/1c65adbef01f45a7
a07379621b5800fc337fc9db90d8eb08baf84e5c533191d9/my_binary.aot

Figure 4: macOS Unified Logs showing Rosetta lookups, using cached files, and translating with private data enabled (with custom profile installed)

FSEvents

FSEvents can be used to identify historical execution of x86-64 binaries even if Unified Logs or files in the Rosetta 2 Cache are not available or have been cleared. These records will show the creation of directories within the Rosetta 2 cache directory, the creation of .in_progress files, and then the renaming of the file to the AOT file, which will be named after the original binary.

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba3
5cff449a366ea59b26af7d54c59779cdfac161 
FolderEvent; FolderCreated; 35102230

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba35
cff449a366ea59b26af7d54c59779cdfac161/
com.apple.systemsettings.cache.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102231

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/5660060629e3493074db75fba35
cff449a366ea59b26af7d54c59779cdfac161/com.apple.systemsettings.cache.aot 
FileEvent; Renamed; 35102231

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599f
73f25fab17744eb0c51b8d0071c53bb878471 
FolderEvent; FolderCreated; 35102223

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599
f73f25fab17744eb0c51b8d0071c53bb878471/sudo.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102224

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/31a32b61c112dae22363556599
f73f25fab17744eb0c51b8d0071c53bb878471/sudo.aot
FileEvent; Renamed; 35102224

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629 
FolderEvent; FolderCreated; 35102259

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629/id.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102260

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/9d8b46ee31e24e4988f923ac2e5
23171b7868f20b671cbaeb39d8db6199f4629/id.aot 
FileEvent; Renamed; 35102260
private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9 
FolderEvent; FolderCreated; 35102355

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9/chmod.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102356

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/fc9130581b8efed813de3826cfd
4bb34586b0872a0977efaa1d51f0861f564c9/chmod.aot 
FileEvent; Renamed; 35102356

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde 
FolderEvent; FolderCreated; 35102671

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde/cat.aot.in_progress 
FileEvent; Created;Renamed;Modified; 35102672

private/var/db/oah/433955637247d22a05957b32fa7f08e0b2f022
ed40311775d461444ce17beadb/c38ceae510d3b2a96bfa6f040fdf
7587121eb388f508c5d50f53efc40cf35dde/cat.aot 
FileEvent; Renamed; 35102672

Figure 5: Decoded FSEvents records showing the translation of a x86-64 POOLRAT variant on macOS, and subsequent universal system binaries executed by the malware as x86-64

AOT File Analysis

The AOT files within the Rosetta 2 cache can provide valuable insight into historical evidence of execution of x86-64 binaries. In multiple cases over the past year, Mandiant identified macOS systems being the initial entry vector by APT groups targeting cryptocurrency organizations. In the majority of these cases, Mandiant identified evidence of the attackers deleting the malware on these systems within a few minutes of a cryptocurrency heist being perpetrated. However, the AOT files were left in place, likely due to the protection by SIP and the relative obscurity of this forensic artifact.

From a forensic perspective, the creation and modification timestamps on these AOT files provide evidence of the first time a specified binary was executed on the system with a unique combination of the attributes used to generate the SHA-256 hash. These timestamps can be corroborated with other artifacts related to binary execution where available (for example, Unified Logs or ExecPolicy, XProtect, and TCC Databases), and file system activity through FSEvents records, to build a more complete picture of infection and possible attacker activity if child processes were executed.

Where multiple AOT files exist for the same origin binary under different Binary UUID directories in the Rosetta 2 cache, and the content (file hashes) of those AOT files is the same, this is typically indicative of a change in file data sections, or more commonly, file system metadata only.

Mandiant has previously shown that AOT files can be analyzed and used for malware identification through correlation of symbols. AOT files are Mach-O binaries that contain x86-64 instructions that have been translated from the original ARM64 code. They contain jump-backs into the original binary and contain no API calls to reference. Certain functionality can be determined through reverse engineering of AOT files; however, no static data, including network-based indicators or configuration data, are typically recoverable. In one macOS downloader observed in a notable DPRK cryptocurrency heist, Mandiant observed developer file path strings as part of the basic Mach-O information contained within the AOT file. The original binary was not recovered due to the attacker deleting it after the heist, so this provided useful data points to support threat actor attribution and malware family assessment.

/Users/crown/Library/Developer/Xcode/DerivedData/
DownAndMemload-becawjfobisdcocirecqedzcixcf/Build/Intermediates.noindex/
DownAndMemload.build/Release/DownAndMemload.build/
Objects-normal/x86_64/main.o
/Users/crown/Library/Developer/Xcode/DerivedData/
DownAndMemload-becawjfobisdcocirecqedzcixcf/Build/Intermediates.noindex/
DownAndMemload.build/Release/DownAndMemload.build/Objects-normal/
x86_64/queue.o
/Volumes/Data/Development/DownAndMemload/DownAndMemload/
_g_szServerUrl
_szgetpwuid
_szsleep

Figure 6: Interesting strings from an AOT file related to a malicious DPRK downloader that was unrecoverable

In any case, determining malware functionality is more effective using the original complete binary instead of the AOT file, because the AOT file lacks much of the contextual information present in the original binary. This includes static data and complete Mach-O headers.

Poisoning AOT Files

Much has been written within the industry about the potential for the poisoning of the Rosetta 2 cache through modification or introduction of AOT files. Where SIP is disabled, this is a valid attack vector. Mandiant has not yet seen this technique in the wild; however, during hunting or investigation activities, it is advisable to be on the lookout for evidence of AOT poisoning. The best way to do this is by comparing the contents of the ARM64 AOT files with what would be expected based on the original x86-64 executable. This can be achieved by taking the original x86-64 executable and using it to generate a known-good AOT file, then comparing this to the AOT file in the cache. Discrepancies, particularly the presence of injected shellcode, could indicate AOT poisoning.

Conclusion

There are several forensic artifacts on macOS that may record historical evidence of binary execution. However, in cases of advanced intrusions with forensically aware attackers, original binaries being deleted, and no further security monitoring solutions, combining FSEvents, Unified Logs, and, crucially, residual AOT files on disk has provided the residual evidence of intrusion on a macOS system.

Whilst signed macOS ARM64 binaries may be the future, for now AOT files and the artifacts surrounding them should be reviewed in analysis of any suspected macOS intrusion and leveraged for hunting opportunities wherever possible.

The behavior identified in the cases presented here was identified on various versions of macOS between 13.5 and 14.7.2. Future or previous versions of macOS and Rosetta 2 may behave differently.

Acknowledgements

Special thanks to Matt Holley, Mohamed El-Banna, Robert Wallace, and Adrian Hernandez.

It’s a persistent question: How do you know which generative AI model is the best choice for your needs? It all comes down to smart evaluation.

In this post, we’ll share how to perform pairwise model evaluations – a way of comparing two models directly against each other – using Vertex AI evaluation service and LLM Comparator. We’ll introduce each tool’s useful features, why the tools help us evaluate performance of LLMs, and how you can use it to create a robust evaluation framework.

Pairwise model evaluation to assess performance 

Pairwise model evaluation means comparing two models directly against each other to assess their relative performance on a specific task. There are three main benefits to pairwise model evaluation for LLMs:

1. Make informed decisions: The increasing number and variety of LLMs means you need to carefully evaluate and choose the best model for your specific task. Considering the strengths and weaknesses of each option is table stakes.

2. Define “better” quantitatively: Generated content from generative AI models, such as natural language texts or images, are usually unstructured, lengthy, and difficult to evaluate automatically without human intervention. Pairwise helps define ”better” response close to human responses to each prompt with human inspection.

3. Keep an eye out: LLMs should be continuously retrained and tuned with the new data to be enhanced compared with the previous versions of them and other latest models.

Vertex AI evaluation service 

The Gen AI evaluation service in Vertex AI lets you evaluate any generative model or application and benchmark the evaluation results against your own judgment, using your own evaluation criteria. It helps with: 

• Model selection among different models for specific use cases

• Model configuration optimization with different model parameters

• Prompt engineering for the preferred behavior and responses

• Fine-tuning LLMs for improved accuracy, fairness, and safety

• Optimizing RAG architectures

• Migration between different versions of a model

• Managing translation qualities between different languages 

• Evaluating agents

• Evaluating images and videos

It also supports model-based metrics for both pointwise and pairwise evaluations and computation-based metrics with ground-truth datasets of input and output pairs.

How to use Vertex AI evaluation service

The Vertex AI evaluation service can help you rigorously assess your generative AI models. You can define custom metrics, leveraging pre-built templates or your own expertise, to precisely measure performance against your specific goals. For standard NLP tasks, the service provides computation-based metrics like F1 scores for classification, BLEU for translation, and ROUGE-L for summarization.

For direct model comparison, pairwise evaluations allow you to quantify which model performs better. Metrics like candidate_model_win_rate and baseline_model_win_rate are automatically calculated, and judge models provide explanations for their scoring decisions, offering valuable insights. You can also perform pairwise comparisons using computation based metrics to compare against the ground truth data.

Beyond pre-built metrics, you have the flexibility to define your own, either through mathematical formulas or by using prompts to help “judge models” aligned with the context of the user-defined metrics. Embedding-based metrics are also available for evaluating semantic similarity.

Vertex AI Experiments and Metadata seamlessly integrate with the evaluation service, automatically organizing and tracking your datasets, results, and models. You can easily initiate evaluation jobs using the REST API or Python SDK and export results to Cloud Storage for further analysis and visualization.

In essence, the Vertex AI evaluation service provides a comprehensive framework for:

• Quantifying model performance: Using both standard and custom metrics.

• Comparing models directly: Through pairwise evaluations and judge model insights.

• Customizing evaluations: To meet your specific needs.

• Streamlining your workflow: With integrated tracking and easy API access.

It also provides guidance and templates to help you define your own metrics referring to those templates or from scratch with your experiences of prompt engineering and generative AI.

LLM Comparator: An open-source tool for human-in-the-loop LLM evaluation

LLM Comparator is an evaluation tool developed by PAIR (People + AI Research; PAIR) at Google, and is an active research project. 

LLM Comparator’s interface is highly intuitive for side-by-side comparisons of different model outputs, making it an excellent tool to augment automated LLM evaluation with human-in-the-loop processes. The tool provides useful features to help you evaluate the responses from two LLMs side-by-side using a range of informative metrics, such as the win rates of Model A or B, grouped by prompt category. It is also simple to extend the tool with user-defined metrics, via a feature called Custom Functions.

You can see the comparative performance of Model A and Model B across various metrics and prompt categories through ‘Score Distribution’ and ‘Metrics by Prompt Category’ visualizations. In addition, the ‘Rationale Summary visualization provides insights into why one model outperforms another by visually summarizing the key rationales influencing the evaluation results.

LLM Comparator is available as a Python package on PyPI, and can be installed on a local environment. Pairwise evaluation results from the Vertex AI Evaluation Service can also be loaded into LLM Comparator using provided libraries. To learn more about how you can transform the automated evaluation results to JSON files, please refer to the JSON data format and schema for LLM Comparator.

With features such as the Rationale Cluster visualization and Custom Functions, LLM Comparator can serve as an invaluable tool in the final stages of LLM evaluation where human-in-the-loop processes are needed to ensure overall quality.

Feedback from the field: How LLM Comparator adds value to Vertex AI evaluation service

By augmenting human evaluators with ready-to-use convenient visualizations and performance metrics calculated automatically, LLM Comparator reduces many chores of ML engineers to develop their own visualizations and quality monitoring tools. Thanks to the JSON data format and schema of LLM Comparator, Vertex AI evaluation service and LLM Comparator can be integrated conveniently without any serious amount of development work.

We’ve heard from our teams that the most useful feature of LLM Comparator is the visualization of “Rationale Summary”. “Rationale Summary” can be thought of as a kind of explainable AI (XAI) tool which is very useful to learn why a specific model among the two is better in the judge model’s view. Another important aspect of “Rationale Summary” visualization is that it can be used to understand how a specific language model is working differently from the other model, which is sometimes a very important support to infer why the model is more appropriate for specific tasks.  

A limitation of LLM Comparator is that it can be used just for pair-wise model evaluation, not for simultaneous multiple model evaluation. However, LLM Comparator already has basic components for comparative LLM evaluations and extending it to simultaneous multiple model evaluation may not be a big technical problem. This can be an excellent project for you to contribute to the LLM Comparator project.

Conclusion 

In this article, we learned and discussed how we can organize the evaluation process of LLMs with Vertex AI and LLM Comparator, an open source LLM evaluation tool by PAIR. By combining Vertex AI Evaluation Service and LLM Comparator, we’ve presented a semi-automated approach to systematically evaluate and compare the performance of diverse LLMs on Google Cloud. Get started with Vertex AI Evaluation Service today. 

^{We thank Rajesh Thallam, Skander Hannachi, and the Applied AI Engineering team for help with this blog post and guidance on overall best practices. We also thank Anant Nawalgaria for help with this blog post and technical guidance.}

Yassir is a super app, supporting the daily lives of users in more than 45 cities across Algeria, Morocco, Tunisia, South Africa, and Senegal who rely on our ride-hailing, last-mile delivery, and financial services solutions. These users are both consumers and vendors — including drivers, couriers, restaurants, and more — that use our platform to run their businesses. 

At Yassir, we process a wide variety of datasets to ensure we provide the best and most reliable solutions for our users across all of our offerings, and we depend on that data to continually improve those services. However, our previous infrastructure made unifying data and AI difficult. 

Previously, we had two separate data systems: one using Databricks for deploying and training machine learning models and another through Google Cloud and BigQuery for storing and analyzing data. This setup led to several issues, such as formatting incompatibilities that we could not resolve. In addition, retrieving data from Databricks for processing within Google Cloud wasn’t possible, and this disconnect directly impacted our application performance.

These siloed environments meant our teams often had to duplicate work to develop and maintain any data projects, paying to maintain separate environments, and, despite all of this, failing to get the information that teams needed at the desired pace. 

To address these issues, we decided to consolidate our data infrastructure with Google Cloud to bring all of these functions into one place. This migration would allow us to provide better access to data and more scalability, and create new opportunities to analyze, review, and improve performance.

Creating a more flexible, unified data platform

Our existing relationship with the Google Cloud team provided a strong foundation to not only resolve our data connectivity roadblocks but also implement new data processing workflows using BigQuery and deploy new AI and machine learning models with Vertex AI. Consolidating with a single data provider also gave us a centralized place to review and control expenses as well as simple, centralized data governance controls. As a growing company, being able to scale our cloud usage up or down to optimize costs allows us to test and iterate without a hard commitment to every project, and that flexibility is invaluable. 

We worked closely with the Google Cloud team to design a solution that aligns with our growth goals. This meant participating in technical and strategic workshops to help train our team on the ins and outs of BigQuery — and its real-time, governance, and open-source capabilities — empowering our engineers with the tools and resources they need to experiment. This collaborative approach allows us to nurture the type of engineering culture we want to promote at Yassir; rather than simply using out-of-the-box solutions, we can tackle more complex problems by adapting flexible, existing solutions to our specific use cases.

After conducting our internal compatibility reviews, we migrated individual models from our previous solution into Vertex AI to test their consistency, and now they’re up and running nearly autonomously. By migrating from Databricks to BigQuery and combining our own models with the models provided by Google Cloud, we’ve improved the performance and efficiency of our machine learning processes and better positioned ourselves for ongoing growth. We may not be processing petabytes of data yet, but we know that we have the capability to do so when needed.

Evolving from data processing to data insights

Our previously disconnected data solutions made it difficult to provide secure access to specific data for specific teams. Since we stored our data in BigQuery but deployed models with Databricks, granting access to information to a user or a team meant giving them the keys to everything. Now, we can implement role-based access controls as well as Infrastructure as Code (IaC) Terraform scripts to automatically grant and revoke access to datasets for individuals or teams. Sharing data through Looker Studio Pro and directly providing access to BigQuery tables for our more technical users also means we can ensure the required data reaches the right users.

With our data unified in BigQuery and connected to our machine learning models, we can better support everything from customer growth and retention to marketplace optimization by providing insights into product usage, customer data, and more. To ensure we’re hitting our internal and customer-related goals, we closely monitor and create dashboards for operational and analytical datasets. 

Our operational dashboards give our sales and marketing teams the insights they need to better target and reach merchants and consumers. They also include insights into our staffing processes, helping us to gradually reduce delivery times, complete more rides faster, and improve how we support specific markets. We also have product-level detection and monitoring that help us provide real-time dynamic pricing and identify fraudulent trips and orders. Each data point we collect gives us more opportunities to build a more personalized and consistent customer experience. 

Our leadership team relies on our rapidly available datasets to drive strategic decision-making, including regional investment decisions to grow the business, macro-level plans for growth trajectories and marketing budgets, and identification of the areas of the business that need the most support or attention. These roadmap decisions are core to our overall growth strategy, and they wouldn’t be possible without the flexibility and scalability we’ve been able to achieve with BigQuery.

Many organizations use Terraform to manage their cloud deployments through Infrastructure as Code. Today we are excited to share the general availability of Terraform providers for Oracle Database@Google Cloud. You can now deploy and manage Oracle Autonomous Database and Oracle Exadata Database Service resources using the Google Terraform provider. The release compliments the existing gcloud and Google Cloud console capabilities that we released with Oracle Database@Google Cloud at launch.

Provision Autonomous Database with Terraform

Creating an Oracle Autonomous Database on Google Cloud is as simple as defining a google_oracle_database_autonomous_database resource and running terraform init and terraform apply. The below code example does the following:

• Collects the details of VPC network where the ADB will be placed

• In this case, we are using an existing VPC named ora-vpc in the GCP project named project1 

• ID and display name of adbtf1

• Region of us-east4

• Project project1

• The admin password has been obfuscated

• VPC network defined above

• A client CIDR range of 172.16.44.0/25

• An ECPU count of 2

• Allocated storage of 100GB

• Autonomous workload type of OLTP

• License model using Oracle’s license-included SKU

Provision with ease using Google Cloud

Working with Terraform is simple on Google Cloud thanks to our native integration in Cloud Shell. Launch Cloud Shell from the Google Cloud console and create Terraform configuration files using our native editor or pull from your company’s source code repository. You can even run the terraform command directly in Cloud Shell. You’ll be authenticated using your Google Cloud IAM credentials and your resources will be immediately created and synchronized according to the infrastructure as code definitions. You can see the console, editor, and Cloud Shell in action below:

Why use Terraform for your Oracle Database@Google Cloud deployments?

• Simplified management: Define your entire Oracle Autonomous Databases or Exadata infrastructure and VM clusters in declarative configuration files.

• Integrated automation: Automate the provisioning, configuration, and management of your Oracle Database@Google Cloud resources.

• Improved consistency: Ensure consistent and repeatable deployments across different environments.

• Reduced errors: Minimize manual configuration and reduce the risk of human error.

• Enhanced collaboration: Enable teams to collaborate effectively on infrastructure management.

Ready to experience the benefits of Infrastructure as Code for Oracle Database@Google Cloud?

• Explore the google provider documentation: Autonomous Database Serverless, Exadata Infrastructure and Exadata VM Clusters

• Dive into the oci provider documentation: Exadata Database, Database homes, and Terraform on Oracle Cloud

The emergence of 5G networks opens a new frontier for connectivity, enabling advanced use cases that require ultra-low-latency, enhanced mobile broadband, and the Internet of Things (IoT) at scale. However, behind the promise of this hyper-connected future lies an equally critical challenge: managing the complexity of 5G networks. With dynamic demands on bandwidth, latency, and reliability, traditional static configurations and manual operations are no longer sufficient.

Enter intent-based service management, a pioneering approach that combines AI and automation with service orchestration and assurance. With intent-based service management, network resources in a 5G network scale and adjust dynamically according to real-time demand, based on the business intent, the 5G network resources are orchestrated dynamically and adjusted just as they are for cloud computing, storage, and networking and streaming services. ​

Together, Ericsson and Google Cloud are pushing the boundaries of this technology, exploring  technology that not only simplifies operations but redefines how telecommunications companies can deliver on the promise of 5G.

The power of intent-based autonomous operations in 5G

Intent-based networking has entered the spotlight as Communications Service Providers (CSPs) recognize that they must master substantial network complexity with sophisticated automation if they want to create, deliver, and sustain services, especially in cloud-native network environments like standalone 5G.​ Automation of networks and related operations cannot evolve and mature to full closed-loop autonomy without intent-based implementations.

Take for example a hospital that needs dedicated 5G connectivity, provided by a 5G network slice, in order to support emergency communications with ultra-low latency. The required configuration in such a scenario should not be a static configuration;  in order to provide optimal service experience it should evolve with the network conditions and the demand. The network layer needs to understand the intent behind every service request  e.g. “provide ultra-low-latency video services” — and translate it into actionable configurations across the network, which then are dynamically updated based on real-time factors such as changes in traffic conditions, underlying network status, and so on.

With Ericsson’s Service Orchestration and Assurance and Google Cloud’s Vertex AI, this process becomes far easier. The solution takes high-level intents, expressed in natural language, and maps them to precise technical requirements like bandwidth, latency, and throughput; then orchestrates the creation of a tailored network slice in real time. The quality of network-slice-based service provisioned is assured via the intelligent closed loop automation mechanism.

TMF standardization for intent-based operations​

To integrate different solution components with one another without expensive and fragile integration solutions, you need a standards-based interface and model.

The Telemanagement Forum (TMF) has specified APIs and a model with common intents to enable this sort of integration. TMF provides intent models and interfaces at the business, service, and resource layers, which derive from CSP-defined business objectives and follow the Autonomous Networks Reference Architecture.​ Ericsson and Google Cloud’s approach is fully compliant to these standards and intent models.

The Role of AI and automation

At the heart of this solution lies AI-driven closed-loop automation. Once a service is instantiated, AI agents continuously monitor its performance against defined KPIs such as quality of service (QoS). Going back to our hospital example, if latency in the hospital’s network slice exceeds the acceptable thresholds, the system automatically identifies the issue and proposes corrective actions, such as reallocating RAN resources or Core Network.

This closed-loop system isn’t just reactive — it’s predictive. By analyzing data from the network in real time, it can anticipate issues and resolve them before they impact users. For telcos, this means not just operational efficiency but also delivering a superior  quality of service.

Ericsson’s closed-loop system leverages Google Cloud’s AI agents for the intent translation, proposal and evaluation agents for slice assurance and remediation. The AI agents reason on how to best achieve a goal based on inputs and tools at their disposal, leveraging models, tools, and orchestration.

In addition to Gemini-based AI agents, Ericsson Service Orchestration and Assurance uses Vertex AI to implement predictive AI services for the remaining closed-loop agents (such as assurance, measurement, and actuation). 

Why Ericsson Service Orchestration and Assurance and Google Cloud

Ericsson Service Orchestration and Assurance is a multi-domain, multi-technology platform that enables cross-domain orchestration across access, transport, and core networks. It addresses CSPs’ critical need for simplification of open and programmable networks by automating the service lifecycle, ultimately boosting revenues and profitability.

The platform provides an open, standards-based architecture that fosters innovation and multi-vendor integration in the telecom ecosystem. It allows CSPs to design services with ease across both network and IT environments, reducing time-to-market and enabling collaboration across diverse teams and use cases with fewer manual errors. 

Furthermore, intent-driven orchestration accelerates the testing and launch of new services in multi-vendor environments, potentially increasing the number of services that can be launched. SLA commitments are enhanced through closed-loop automation, improving customer satisfaction and operational efficiency. Designed to be vendor-, technology-, and service-agnostic, Ericsson’s platform empowers CSPs to implement multi-domain service orchestration effectively, enabling service expansion and revenue growth.

To support this sophisticated framework, Google Cloud provides a foundation to meet the demands of modern telecom networks:

• Scalability for dynamic networks: As 5G networks expand, Google Cloud’s elastic compute power helps ensure even the most resource-intensive processes, such as intent translation and real-time assurance, run seamlessly without performance bottlenecks.

• AI-first infrastructure: Leveraging Vertex AI, intelligent agents translate intents into actionable configurations while optimizing network performance. With AI tools like BigQuery ML, telcos can move beyond basic analytics to actionable insights, enabling smarter decision-making.

• Vertex Model Garden for open innovation: Pre-trained models and open-source LLMs in Vertex AI Model Garden empower telcos to innovate and customize highly specialized 5G service management solutions.

• Hardware acceleration: Google Cloud’s Tensor Processing Units (TPUs) and Nvidia GPUs provide the computational power necessary to efficiently train and deploy complex AI models for intent-based service management.

• Hybrid and edge capabilities: 5G networks demand hybrid setups across cloud and edge environments. Google Cloud’s edge-to-cloud orchestration helps ensure operations, whether optimizing central data centers or delivering ultra-low-latency services at the edge.

• Security and compliance: Operating in a highly regulated environment, telcos benefit from Google Cloud’s secure, reliable infrastructure tailored to meet the stringent compliance requirements of telecom workloads.

• Gemini for advanced reasoning: Gemini’s multimodal models enhance intent interpretation by processing natural language and network telemetry data. This enables nuanced and precise intent translation, leading to more effective automated actions, higher efficiency, and improved reliability of 5G service management.

Together, Ericsson Service Orchestration and Assurance and Google Cloud are exploring technology to meet these challenges together, combining automation, intelligence, and scalability to empower CSPs to deliver innovative, efficient, and profitable 5G services.

Real-world impact: A hospital use case

Let’s return to the hospital scenario. A request for a network slice is received from hospital users through the system’s intent translator: “Provide video service for patient doctor interactions having ultra low latency.​”

The solution springs into action:

1. Translating intent: The request is converted into technical configurations for the RAN and core network.

2. Creating the slice: Using orchestration tools, the system allocates resources and configures the network slice with Quality of Service (QoS) goals across the 5G network.

3. Assuring performance: With closed-loop automation, AI agents continuously monitor quality of service elements such as latency or packet loss. If a deviation that violates the intent occurs or is forecasted to occur, corrective actions are proposed and executed automatically.

In this scenario, the hospital’s needs are met dynamically, helping to ensure critical operations proceed without disruption.

Beyond efficiency: Redefining 5G Innovation

Intent-based service management enables telcos to innovate at scale. By automating complex tasks, telcos can focus on creating new services, entering new markets, and unlocking revenue streams. For instance, they can offer differentiated slices tailored to industries like gaming, manufacturing, and healthcare, creating a competitive edge in a crowded market.

Moreover, the integration of custom gen AI models like Gemini helps these solutions be tailored to telcos’ unique needs. These LLMs, trained on telecom-specific data, enable intelligent automation that feels intuitive and responsive.

Ericsson and Google Cloud’s collaboration showcases the potential of intent-based service management. By combining Ericsson’s technical expertise with Google Cloud’s AI infrastructure, telcos can navigate the complexities of 5G with confidence.

As 5G adoption accelerates, solutions like this will become the cornerstone of telecom operations, enabling seamless, dynamic networks that deliver on the promise of connectivity.

The future of 5G isn’t just about faster speeds or lower latency — it’s about intelligent, intent-driven networks that adapt to the world’s needs in real time. And together, Ericsson and Google Cloud are making that future a reality.

Telecoms, like all businesses, are wondering how AI can transform their businesses. And there’s no better way to display how to build the AI-driven telecom than with demos. Join us at Mobile World Congress 2025, March 3-6 in Barcelona Hall 2 Booth #2H40, where we’ll be highlighting key agent use cases where AI is becoming an imperative:

• Customer experience

• Employee productivity

• Field operations

• Network operations

We know that the future of telecom isn’t just about gen AI — it’s also about data and security. We will show how to execute on cloud migrations and implementation of our core security practices.

We can’t do this alone. Across our 12 demonstrations, we have collaborations with key customers and partners on display including: Amdocs, Bell, Deutsche Telekom, Ericsson, MasOrange, Nokia, TELUS, and Vodafone. 

Lastly, we will showcase the power of cloud-native networks and the ability to monetize those networks. Discover how we harness the power of Open Gateway Initiative (OGI) APIs, expose them to Firebase developers, and enable a new way to implement electronic phone number verification (ePNV). You’ll see lots of amazing progress by the telecom community! 

Here are the demos that will be on display:

• Creating AI Agents: Share your requirements on the spot and we will use Gemini on Vertex AI and AI Studio to build your agent within minutes.

• Employee Productivity with Agentspace: Discover how Agentspace leverages Gemini’s advanced reasoning and Google-quality search to put all your enterprise knowledge at your employees’ fingertips, dramatically boosting productivity and unlocking expertise.

• CX with Amdocs: Discover how Google Cloud’s collaboration with Amdocs on amAIZ, powered by Gemini’s generative AI, is revolutionizing both customer experience management and autonomous network operations in telecom.

• Developing AI Apps: Discover how developers can leverage a suite of Google tools, including Flutter, Firebase, Gemini, and Vertex AI, to create intelligent app experiences that dramatically improve the customer journey, featuring AI-powered phone-plan selection.

• User Identity with Firebase and OGI: Experience a more secure and reliable user identity service built on Firebase and the Open Gateway Initiative, offering developers expanded reach and enhanced authentication capabilities.

• Ericsson Networks with AI: Ericsson has a long-standing relationship with Google Cloud. In this demonstration, we will showcase how the two companies are reimagining 5G networks with AI agents driven by intent-based service management using Vertex AI.

• Load Balance for Inferencing: Experience a gamified challenge against our AI-powered Cloud Load Balancer, then dive into the Google Cloud console to see how you can use custom metrics to optimize GPU utilization and achieve lightning-fast inference response times.

• AI for Cybersecurity: This demo showcases how Google Threat Intelligence provides visibility into threats and how it delivers detailed and timely threat intelligence to security teams around the world. Gemini in Threat Intelligence acts as a force multiplier, immediately surfacing threats most relevant to your unique risk profile and summarizing complex threat reports. 

• AI Network Operations: This AI network operations demo shows advanced AI/ML-powered tools and capabilities to help CSPs understand the impacts of network issues, detect them, and resolve them faster through automated end-to-end correlation and root-cause analysis. The solution leverages several native Google Cloud services.

• Nokia Network on Demand: This demo features a collaboration between Nokia AVA and Google Cloud to show customers how Nokia delivers an innovative approach to energy efficiency by combining AI-driven insights with gamification, engaging users and driving sustainability in telecom networks.

• Gen AI Field Operations: Last but not least, discover how Google Cloud is helping CSPs supercharge field-agent productivity, elevate service quality, and reduce costs through the power of multi-modal gen AI and democratized data access.

And if you are up for some fun, come by and get behind the wheel of the Formula E simulator, powered by Google Cloud AI. Experience the same cutting-edge technology that’s transforming motorsport through our partnership with Formula E. Feel firsthand how AI and data are revolutionizing racing, and the future of telecommunications.

Engage with the technologies reshaping the industry and discover how Google Cloud can help you drive innovation. Don’t just hear about the future of telecom — build it with us at Mobile World Congress 2025, March 3-6 in Barcelona Hall 2 Booth #2H40. The future of telecommunications awaits!

AI is driving unprecedented change and evolution across every industry, and the telecommunications sector is at a particularly interesting crossroads: The industry is standardizing on 5G; data is growing exponentially; and customer expectations are quickly changing. Leading communication service providers (CSPs) are working with Google Cloud to adopt AI and lead the industry forward in terms of innovation.

At Google Cloud, we believe that AI, harnessed boldly and responsibly, has the potential to transform how CSPs operate on planetary scale — and we’re deeply committed to delivering the technologies, tools, and resources that help them do so. From building a strong data foundation, to optimizing network operations, to enhancing customer experiences, our AI-powered solutions and strategic partnerships enable CSPs to innovate and grow their businesses.

Heading into Mobile World Congress 2025 next week, let’s look at the announcements we’ve made with leaders in this industry to build AI-driven telecoms. 

Data migrations are key to adopting AI 

CSPs typically need to move their data into one place to use AI effectively. Google Cloud helps with this by offering tools like BigQuery and Looker, which can store and organize vast amounts of data, creating scalable data lakes and data oceans that can be analyzed by data scientists and businesspeople alike. Our open approach allows integration with existing systems, streamlining data management and getting benefits faster. Combined with our AI infrastructure, CSPs can gain actionable insights, create personalized experiences, and build new services.

DNA, part of Telenor, for instance, announced it has selected Google Cloud to help accelerate its transition to the cloud and support the delivery of fast and reliable services to its customers, allowing it to gradually migrate its on-premises workloads to Google Cloud and pave the way for generative AI adoption. 

Vodafone Italy also announced it has modernized its data architecture by building a new AI-ready platform on Google Cloud called Nucleus. This gives the company enhanced process efficiency, scalability and real-time data processing. In partnership with Amdocs and Google Cloud, Vodafone re-engineered its data pipelines to optimize operational workflows and support business-critical functions. The migration to Nucleus was achieved in only 12 months without interruption or issues. The company’s modern architecture delivers key benefits, including enhanced agility, more efficient reporting, cost optimization, and improved real-time processing.

From reactive to proactive network operations with AI

Telecommunications providers face the constant challenge of maintaining optimal network performance and reliability in a dynamic and increasingly complex environment. Traditional reactive network-management approaches can lead to service disruptions, increased operational costs, and diminished customer satisfaction. Google Cloud’s AI-powered platform, which includes Vertex AI and Network Intelligence Center, provides intelligent automation for telecommunications providers, enabling them to analyze network data with BigQuery to identify anomalies, predict outages, and optimize traffic flow in real time. This shift from reactive to proactive management allows CSPs to not only improve reliability and reduce costs, but also create self-healing networks that dynamically adapt to changing conditions. 

Bell Canada, for example, announced its using Google Cloud’s AI technologies to automate network issue detection and resolution with its new AI operations (AI Ops) solution. Partnering with Google Cloud has enabled Bell to simultaneously boost software delivery productivity by 75%, accelerating time to market, and reduce customer-reported issues by 25%, significantly enhancing customer experience and solution reliability. And Deutsche Telekom is using Google Cloud’s Gemini 2.0 in Vertex AI to develop a network AI agent called RAN Guardian, which can analyze network behavior, detect performance issues, and implement corrective actions to improve network reliability and customer experiences. This collaboration aims to create autonomous, self-healing networks that proactively optimize RAN performance and reduce operational costs.

Then there’s Amdocs, a leading provider of software and services to communications and media companies, which in partnership with Google Cloud, announced recently that it has launched a new Network AIOps solution to help 5G network providers automate complex network operations, enhance service reliability, and improve customer experiences.

Through AI-powered insights, CSPs can deliver superior service quality while streamlining their operations, ultimately driving greater customer satisfaction and loyalty.

Empowering field technicians with AI

CSPs often face challenges in field service operations, such as inefficient dispatching and a lack of real-time diagnostics, leading to longer resolution times and increased customer downtime. Google Cloud’s platform, including a multimodal field-tech AI assistant and Customer Engagement Suite, empowers field technicians with real-time diagnostics, predictive maintenance, optimized routing, and agentic workflows powered by Gemini. This leads to fewer worker deployments and faster issue resolution. AI-driven diagnostics proactively identify and resolve potential issues. Predictive maintenance models anticipate equipment failures, enabling scheduled repairs. Agentic workflows and Gemini provide technicians with a comprehensive knowledge base for faster troubleshooting. Smart routing helps ensure efficient dispatch. Together, these AI-powered services help create a streamlined, customer-centric field service operation, providing faster and more effective support.As an example, TELUS’ NEO Assistant, an innovative multimodal AI copilot that leverages advanced machine learning and AI technologies like Google Cloud’s Gemini model, provides field technicians with instant access to critical information and streamlines workflows. Since its launch, NEO Assistant has been adopted by three-quarters of TELUS field technicians, contributed to the creation of nearly 7,000 jobs, and significantly increased efficiency by allowing tasks to be completed in under a minute.

Delighting customers with AI-driven insights

CSPs are facing increasing pressure to deliver exceptional customer service while managing costs and navigating complex customer journeys across multiple channels. Google Cloud’s AI platform, featuring the Customer Engagement Suite, Vertex AI, and Vertex AI Search, empowers CSPs to create exceptional customer experiences. By harnessing AI-driven insights about customer behavior, CSPs can anticipate needs, proactively address issues, and deliver personalized recommendations and seamless interactions, ultimately boosting satisfaction and revenue.

Chunghwa Telecom is using Google Cloud’s AI to enhance strategic decision-making and data analysis, exemplified by tools like NotebookLM Enterprise for strategic decision-making and data analysis, and a Customer Agent using Gemini 2.0 that slashes response times and is projected to reduce billing-related calls by 25% annually. Additionally, automating roaming charge verification with Document AI streamlines operations and reduces costs by eliminating manual invoice processing, leading to potential cost savings for both the company and subscribers.

Glance leverages Google’s suite of AI models, including Gemini and Imagen, to deliver visually captivating and AI-powered experiences that position consumers as the ‘heroes’ of their digital world. GlanceAI offers an immersive shopping experience from inferred user interests and images uploaded by consumers, drawing consumers in and helping them visualize themselves in various styles so they can make real-time purchase decisions. Here, AI transforms a quick glance at a device into an interactive shopping opportunity, whether on a smartphone lock screen or an ambient TV screen.

Unlocking new revenue streams through APIs

CSPs face high network deployment costs and pressure to innovate, leading them to explore APIs and engage developers to monetize their networks. Google Cloud’s developer-centric platform simplifies network asset monetization by providing necessary tools and infrastructure. Initiatives like GSMA Open Gateway foster collaboration and developer access to network capabilities.

Built on the Open Gateway vision, yesterday we announced a new Firebase phone number verification service, providing more than three million Firebase developers with access to critical network CSP APIs such as phone number acquisition and verification. We’re partnering with Deutsche Telekom, Orange, Telefónica, T-Mobile, Telenor, Verizon, Vodafone and others to offer this service to improve developer and end-user experiences, and strengthen security by mitigating fraud — creating new revenue opportunities for CSPs and developers alike. 

The future of AI in telecommunications

MWC 2025 marks a pivotal moment for AI-driven telecom innovation, and Google Cloud is at the forefront of this transformation. As CSPs continue to embrace AI in the coming months, our platform ensures they unlock new efficiencies, elevate customer experiences, and drive business growth. 

For more information, see our press page or visit Google Cloud at Mobile World Congress (MWC) from March 3-6 in Hall 2, Booth #2H40.

The telecommunications industry has always been on the leading edge of technology — but rarely has it been changing as quickly as it is today. Increased 5G adoption, the spread of edge computing, new monetization models, and growing consumer expectations are presenting opportunities and hurdles in equal measure. 

Since its inception, Google Cloud has made supporting telecommunications innovation a priority — and a key part of that has been fostering a partner ecosystem that can help meet the constant evolution that communication service providers (CSPs) are undergoing. 

The AI era has made such collaboration more important than ever, so ahead of Mobile World Congress (MWC), we’re showcasing the breadth of our ecosystem of telecom partners and the ways they are enabling customer success with Google Cloud. Our partners — spanning global system integrators (GSIs), specialized ISVs, and network equipment providers (NEPs) — are using AI to help CSPs optimize network performance, personalize customer experiences, automate operations, and identify new revenue streams. 

Improving network operations 

AI is enabling CSPs to move from reactive to proactive network management, with capabilities that span predictive maintenance, automated troubleshooting, and real-time optimization. Our partners are bringing the next-generation network to life by using Google Cloud’s Vertex AI and Network Intelligence Center. Together, we’re providing the necessary tools and technologies for building and deploying AI solutions across the modern network.

• Amdocs yesterday announced Network AIOps, an AI-powered solution built on Google Cloud’s Vertex AI and BigQuery, that can help telecommunications companies automate and optimize their 5G network operations, thus enhancing reliability and customer experiences.

• CARTO, a geospatial analytics platform provider, uses BigQuery to enable telecoms to visualize and analyze massive location-based datasets, leading to more effective network planning and optimized service delivery. 

• Ericsson, a leading telecommunications company, is building AI agent-driven network slice instantiation and assurance for telecom customers with Google Cloud’s AI. 

• Fortinet, a cybersecurity leader, collaborates with Google Cloud to enhance real-time threat detection and response capabilities with AI, providing network security and resilience for telecom providers.

• MATRIXX, a leading provider of cloud-native business support services (BSS) solutions, is using Google Cloud’s AI to simplify and automate tasks, helping telecoms reduce costs and deliver a better customer experience.

• Nokia is collaborating with Google Cloud to enhance its energy saving application, wherein telcos can include their subscribers as part of their energy savings goals.

Optimizing IT transformation and productivity 

Improving IT systems is important for telecom companies to work faster and smarter. Using tools like BigQuery, Looker, and Vertex AI, our partners are helping telecom providers automate tasks and optimize the value they get from data and AI projects while improving overall IT capabilities.

• BMC Helix Software uses Vertex AI to enhance its service assurance capabilities, helping telecoms streamline IT processes, predict and prevent outages, and automate resolutions.
• Datatonic is building solutions for telecoms with Vertex AI and Google Cloud’s generative AI technology, helping it better understand customer behavior, personalize service recommendations, enhance network analytics, and boost employee productivity.
• Zinkworks is using Vertex AI, Gemini models, and additional Google Cloud technology to develop and implement solutions that automate network operations, significantly reducing operating costs and enhancing efficiency for telecom workers in the field.

Enhancing the customer experience

AI helps CSPs better understand customer needs, personalize interactions, and provide proactive support. Partners are using Vertex AI, BigQuery, Customer Engagement Suite, and Telecom Subscriber Insights to develop solutions that deliver personalized and engaging experiences to telecom customers.

• commercetools is helping telco businesses build agile, scalable, and flexible commerce on top of Google Cloud.

• Pega is an AI decision-making and workflow automation platform that helps CSPs and other enterprises transform their operations, personalize customer engagement, and continuously adapt to change with the help of Google AI.

• Sprinklr offers a unified customer experience management platform powered by Google AI that helps front-office teams and contact center agents to make every customer experience extraordinary.

• Zeotap offers a secure customer data platform (CDP) on Google Cloud that empowers telco and other brands to integrate and orchestrate customer data, enabling targeted marketing while protecting customer privacy, all designed for now and for when cookies are no longer used.

Revamping monetization tools

AI plays a key role in optimizing pricing strategies, personalizing offers, and automating billing processes. Partners are using AI and network APIs to create more effective monetization models with solutions that can analyze customer data to identify preferences, predict churn, and recommend optimal pricing strategies. 

• Aduna, a venture between Ericsson and leading CSPs, is simplifying global access to network APIs, making it easier for Google Cloud developers to build new applications leveraging mobile networks to accelerate digital transformation across industries.

• Beyond Now offers an AI-powered digital marketplace for CSPs and technology providers to co-create and bundle partner solutions, helping support their customers’ digital transformation journeys. 

• Eureka.AI provides actionable market and risk intelligence products for industry verticals to help CSPs modernize their data monetization.

• Glide Identity provides a platform built on Google Cloud for identity and anti-fraud products, helping telecoms create new revenue opportunities and delight users. 

• Kloudville offers a marketplace powered by Google Cloud that enables telecoms to offer digital products and services to their SMB customers, creating new revenue opportunities and enhancing the customer experience. 

• Nokia’s Network-as-Code platform simplifies the use of advanced network capabilities by exposing them as developer-friendly interfaces, enabling applications to work seamlessly across multiple public and private networks, and fostering new value creation and network monetization.

Enabling customer success with services partners

Services partners play a critical role in providing customers with the expertise and support needed to plan, deploy, and optimize AI projects. Many of these partners have launched services specifically for telecoms and are continuing to demonstrate their proven ability to help customers transform with AI and other Google Cloud technology. 

• Accenture empowers CSPs to reinvent customer engagement, drive revenue growth, and enhance network operations through AI, machine learning, and gen AI agents, while prioritizing cost reduction and sustainability.

• Capgemini helps CSPs develop autonomous network capabilities and next-generation field operations, as well as strength in AI-powered marketing, customer care, and lifetime-value management. 

• Prodapt is helping the global telecom and technology industry use Google Cloud products, including generative AI tools and solutions, to modernize technology infrastructure, fast-track data migration, improve productivity, and make operations seamless and cost-efficient.

• TCS offers innovative end-to-end business solutions that encompass agentic AI frameworks for CSPs across customer experience, customer service, field service, and network operations; it also creates tailored offerings such as TCS CX Transformer for Telcos, which combines its vast industry expertise and Google Cloud’s data analytics and AI technology. 

• Tech Mahindra partners with CSPs to scale at speed by enabling them to simplify and modernize their business and technology, as well as to monetize their assets for greater revenue growth, and all of this is powered by AI.

Supporting sustainable practices

On top of navigating evolving customer demands and technology needs, CSPs are juggling pressure to decarbonize and comply with ever-changing global regulations. AI is helping telecom organizations optimize energy consumption, reduce waste, and minimize their carbon footprint. 

• Geotab provides vehicle and driver behavior data services for more than 55,000 fleets and 4.7 million vehicles around the world. Using Google Cloud’s data analytics and machine learning, Geotab helps customers improve fleet productivity, optimize fleet management, and achieve strong compliance.

• Watershed helps companies manage climate and ESG data, producing audit-ready metrics for voluntary and regulatory reporting using Google Cloud tools.

Google Cloud is committed to fostering a collaborative and mutually beneficial ecosystem where our partners can thrive and contribute to the success of our CSP customers. We believe that together, we can empower the telecom industry to embrace the full potential of digital transformation and shape the future of connectivity.

Visit Google Cloud at Hall 2, Booth #2H40 at Mobile World Congress (MWC) to learn more about how Google Cloud and our partners can help you transform your telecommunications business. We hope to see you in Barcelona!

Telecommunications companies face mounting pressure to reduce operational costs, enhance network resiliency, and deliver exceptional customer experiences. Earlier this week, Amdocs and Google Cloud announced a new network AI operations solution — Amdocs Network AIOps — that uses data to help communication service providers improve their networks and customer service. 

This solution is designed to make networks more self-sufficient and efficient, leading to a better experience for customers. This is the first step for telecoms in evolving to a fully autonomous network. In this blog, we wanted to explore more deeply how Amdocs achieved these autonomous innovations using their technology and Google Cloud’s AI and cloud solutions.

Network analytics and automation

Amdocs Network AIOps is a comprehensive platform designed to address the complexities of modern telecom networks. It’s built on massive quantities of telemetry data using Google Cloud’s data and AI capabilities, including BigQuery and Vertex AI, to provide a comprehensive observabilities and automated AI solution. 

The solution is a network operations AI framework that consists of three layers — Observe, Decide, and Act — that leverages AI and ML to automate network operations, optimize performance, and enhance service reliability.

Key capabilities behind the framework of the Amdocs Network AIOps Solution include:

• Data ingestion and mediation (Observe layer): The platform ingests data from diverse sources across the network, including network elements, operations support systems (OSS) and business support systems (BSS), and probes. 

• AI-driven insights (Decide layer): With tight integration into Google Cloud’s BigQuery, Vertex AI, and Gemini services, Amdocs Network AIOps provides actionable insights in support of predictive analytics and root-cause analysis. This integration empowers operators to harness the power of cloud-based AI/ML capabilities for enhanced network automation enabling proactive network management, thereby reducing the mean-time-to-resolution (MTTR) for incidents.

• Automated workflows (Act layer): The platform automates routine tasks such as network configuration, performance optimization, and incident remediation. This frees up valuable resources and reduces the risk of human error.

• Closed-loop automation (Act layer): Amdocs Network AIOps employs a closed-loop approach, where the system continuously learns from previous actions and data patterns to improve its accuracy and efficiency over time.

Actionable network insights built on AI

Telecom operators are already using Amdocs Network AIOps to drive insights in support of a variety of networking use cases such as anomaly detection, root cause analysis, and predictive maintenance. 

Network downtime is a major concern for telecom operators, leading to lost revenue and dissatisfied customers. The Network AIOps solution significantly reduces the downtime by correlating data from a variety of network sources and using the predictive power of Vertex AI’s no-code or low-code models. 

By training ML models on historical network data, including performance metrics, fault logs, and environmental factors, this solution can accurately predict the likelihood of future network failures. This allows operators to proactively schedule maintenance, replace aging equipment, and optimize network configurations to minimize downtime and ensure uninterrupted service.

Google Cloud services such as BigQuery and Vertex AI are foundational to Amdocs Network AIOps. These services enable transformation and ingestion of petabytes of data, and near-real-time predictive analytics, anomaly detection and correlation, helping telecommunications businesses in meeting the goals of greater network efficiency and reliability and improved customer satisfaction.

Your intelligent network copilot

Managing the complexity of 5G networks can be overwhelming for even the most skilled network engineers. The Amdocs Network Platform for Operations (Act layer), integrated with a multimodal gen-AI-powered live network assistant, provides a single interface for comprehensive network monitoring and management across the full stack of hardware infrastructure, containers, virtualization software, and network applications (RAN, 5G core, IMS core) in a multi-vendor environment.

This gen-AI-powered assistant acts as an intelligent agent for network engineers, offering:

• Proactive alerts and insights: The assistant continuously monitors the network, analyzing data from BigQuery data lake to identify potential issues and correlate events from the infrastructure stack and multi-vendor workload, and use it to proactively alert engineers.

• Automated troubleshooting and remediation: By leveraging its deep knowledge base and AI capabilities, the gen AI assistant can automatically diagnose common network problems and even take corrective actions, such as restarting services or rerouting traffic

• Multimodal and natural language interaction: Network engineers can interact with the assistant using natural language phrases and sentences, as well as audio, video, and images — making it easier to ask questions, seek recommendations, and receive clear, concise answers.

Embrace the AI-Powered future of telecom

The joint Network AIOps solution from Amdocs and Google Cloud delivers a wide range of benefits for telecom operators:

• Reduced operational costs: Automation, predictive maintenance, and intelligent resource allocation lead to significant cost savings.

• Improved network resiliency: Proactive issue identification and automated remediation help minimize downtime and enhance network stability.

• Enhanced customer experience: AI-powered optimization provides an integrated and reliable user experience, fostering customer satisfaction and loyalty.

• Increased efficiency: The gen AI Network Assistant helps network engineers work more efficiently, freeing up their time for strategic initiatives.

The collaboration between Amdocs and Google Cloud brings together the best of both worlds: deep telecom expertise and cutting-edge AI capabilities. The joint Network AIOps solution lets telecom operators embrace the AI-powered future, optimize their 5G networks, and deliver exceptional customer experiences.

To address the challenges of AI-driven 5G networks, please visit Amdocs Network AIOps Solution website. And you can learn more about how Google Cloud is partnering with communication service providers to deliver AI-driven telecom at Google Cloud’s industry site.

Vodafone Italy is reshaping its operations by building a modernized, AI-ready data architecture on Google Cloud, designed to enhance process efficiency, scalability, and real-time data processing. This transformation, powered by Vodafone Italy’s cloud-based platform called Nucleus, aims to unlock new AI-driven capabilities and streamline data management.

Nucleus, developed by Vodafone Italy’s engineering team, leverages Google’s AI infrastructure and BigQuery, along with a robust data movement application and a comprehensive ETL framework, to bring all analytical use cases into a cloud-native environment. This modern architecture enhances data agility, scalability, and AI-driven insights, while enabling Vodafone to consolidate fragmented data silos into a centralized, real-time data ecosystem. In partnership with Amdocs and Google Cloud, Vodafone Italy re-engineered its data pipelines within this flexible ecosystem to optimize operational workflows and support business-critical functions. By integrating its operational data store (ODS) and enterprise data platform(VID) on Nucleus, Vodafone Italy created a scalable foundation for analytics, AI, and machine learning.

Undertaken in only 12 months of careful planning and design, the migration to Nucleus was achieved without interruption or issue for consumers or the business.

This modernized architecture delivers key benefits, including enhanced agility, more efficient regulatory reporting, cost optimization, and improved real-time processing across core functions like finance, operations, and marketing. Moreover, Nucleus provides a blueprint for Vodafone Group’s broader cloud modernization strategy, helping to ensure scalability and future-proofing the organization’s data ecosystem.

Laying the foundation for data modernization: The road to Nucleus

In 2017, Vodafone Italy launched “NEXT”, a bold, company-wide initiative aimed at modernizing its business support systems (BSS) and data management as part of a large-scale digital transformation.

At the heart of this journey was a data management transformation, designed to eliminate inefficiencies caused by fragmented legacy systems, including Teradata, SAP, and BSS workloads, and establish a streamlined, AI-ready data ecosystem. The project focused on harmonizing data processes across finance, commercial, and operations functions to ensure greater efficiency in reporting, real-time data accessibility, and regulatory compliance.

This transformation was pivotal in enhancing scalability, governance, and agility, enabling Vodafone Italy to accelerate time-to-market for new products and leverage data-driven decision-making across the organization. Recognizing the complexity of modern telecom operations, Vodafone Italy partnered with Amdocs as its primary systems integrator, capitalizing on its expertise in BSS and data management to navigate the challenges of a highly regulated industry.

The foundation laid through NEXT set the stage for Vodafone Italy’s next leap in innovation — Nucleus — a cloud-first data platform designed to unlock new AI-driven capabilities and future-proof its digital infrastructure.

Transforming the data architecture: A unified, intelligent approach

The NEXT program marks a breakthrough in data architecture, shifting from fragmented data silos to a centralized, two-tier business intelligence platform designed for scalability, governance, and real-time insights.

Tier 1: Operational Data Store

At the foundation, the Operational Data Store (ODS) is built on the Amdocs Logical Data Model (aLDM) — a TM-Forum-certified model tailored for telecommunications. Implemented via the Amdocs Data Hub, this tier seamlessly integrates data from diverse applications in near-real-time, delivering a cohesive, 360-degree view of customer behavior and interactions across all touchpoints.

Tier 2: Vodafone Integrated Dimensional Data model

Serving as the enterprise data warehouse, the Vodafone Integrated Dimensional Data (VID) model was co-developed by Amdocs and Vodafone’s business and IT teams. It acts as a single source of truth for financial and analytical reporting, aligning with Vodafone’s governance and compliance standards. By consolidating hundreds of terabytes of data into a streamlined model with a few hundred entities, VID simplifies data governance, accelerates decision-making, and enhances operational efficiency.

Together, these tiers power mission-critical applications across the organization — enabling advanced AI and analytics, financial reporting, campaign management, intelligent customer engagement, and martech solutions. With near real-time data access and self-service capabilities, Vodafone’s teams can derive deeper insights, drive innovation, and enhance customer experiences at scale.

Nucleus: Advancing Vodafone Italy’s data modernization

With a modern data foundation already in place, Vodafone Italy sought to further enhance flexibility and interoperability within its data ecosystem. To achieve this, the company embarked on the modernization of VID, integrating it seamlessly into a broader, cloud-first architecture designed for greater efficiency, scalability, and AI readiness.

In June 2023, Vodafone Italy initiated this transformation project, setting a 12-month roadmap focused on planning, testing, and executing the evolution of this mission-critical asset. The modernization process was strategically designed to optimize VID’s integration with Nucleus, providing integrated data processing and enhanced analytics capabilities.

To drive this success, Vodafone Italy partnered again with Amdocs, leveraging its deep expertise in VID and aLDM. Together, Vodafone Italy’s engineering team and Amdocs collaborated within the Nucleus environment to accelerate the shift towards a more agile, scalable, and future-ready data architecture.

Seamless data modernization for business continuity

Vodafone Italy successfully modernized its data infrastructure by migrating to Google Cloud, leveraging BigQuery and a “clone and shift” approach in collaboration with Google Cloud and Amdocs. This meticulously planned 12-month transformation helped ensure a smooth transition without disrupting business operations, maintaining full data integrity and user experience. By replicating their existing data platform within BigQuery, Vodafone unlocked the benefits of cloud-native capabilities, including enhanced scalability, agility, and real-time insights, while gaining access to a comprehensive suite of data management tools within the Nucleus framework.

“This program has unlocked features and capabilities that were once difficult to imagine. We now have a unified data platform, and a comprehensive data ocean that natively integrates AI and generative AI. This empowers us to better serve our customers and understand their needs, protect their data, deliver real-time reporting to our sales, customer operations, and marketing teams, and significantly improve the productivity and time-to-market of our IT department.” – Massimo Guarino, Head of Digital & IT, Data & Analytics and Always-on Marketing Systems, Vodafone Italy

“The Nucleus framework we developed fully harnesses the power of Google Cloud to create new value and transform our operations. This achievement was made possible through our exceptional partnership with Amdocs, whose expertise was instrumental in delivering a seamless and successful migration. Together, we have greatly improved operational speed and adaptability, setting the stage to seize new opportunities, deepen our collaboration, and drive significant growth.” – Michele Bertoni, BI Engineering and Delivery Manager, Vodafone Italy

“Implementing a new data management solution with an aLDM operational data store and enterprise data platform layer gives Vodafone a competitive edge — improving efficiency, enhancing business teams’ experience, and increasing customer and agent satisfaction, while reducing operational costs. It’s a privilege to support Vodafone Italy’s data-driven journey as their dedicated partner.” – Eran Katz, BI Consulting and Data Architecture Unit Lead, Amdocs

In short, Vodafone Italy’s transformation journey exemplifies the power of the cloud in driving data unification, AI integration, and operational efficiency at scale. This partnership, alongside Amdocs, showcases how organizations can unlock unprecedented value through cloud-powered innovation, optimizing both customer experience and IT efficiency. We are proud to support Vodafone and Amdocs in this journey and look forward to further enabling their data-driven growth. Clear here to learn more about Amdocs and Google Cloud’s collaboration on AI and data services.

Connecting hybrid environments to the cloud is a very important aspect of cloud architecture. In addition to connecting from on-premises environments, you also have multicloud environments that all need to communicate. In this blog we will look at some reference architectures for hub-and-spoke communication using Cross-Cloud Network.

The power of Cross-Cloud Network

As your cloud projects grow and you add additional networks, you need inter-network communication. Cross-Cloud Network provides a set of functionality and architectures for any-to-any connectivity leveraging Google’s software-defined global scaled backbone to connect your distributed applications.

Let’s look at two architectural patterns: one based on VPC Network Peering, and the other on Network Connectivity Center.

#1 – Inter-VPC communication with VPC Network Peering example pattern

To understand how to think about designing your network, let’s look at the flow of a packet from an external network to an application located in workload VPC network 1 located in Google Cloud. This design is focused on the use of VPC Network Peering. The network is composed of an external network (on-prem and other clouds), and the Google Cloud network (transit VPC, services access VPC, managed services VPC, workloads VPC).

This design uses the following services for its end-to-end solution:

• Cloud Interconnect (Direct, Partner, Cross-Cloud) – To connect connect from your on-prem or other clouds to the transit VPC

• Cloud VPN – To connect from service-access VPC to transit VPC and export custom routes from private services access network

• VPC Network Peering – To connect from workload VPC to transit VPC

• Private services access – To connect to managed services privately in the services access VPC

• Private Service Connect – To expose services in the managed services VPC network to be consumed in the services access VPC

• Network Connectivity Center VPC spokes – To allow communication between workload VPCs if necessary

To understand more specific details like route exchange and packet flow, please read the full architecture document reference guide: Cross-Cloud Network inter-VPC connectivity using VPC Network Peering.

#2 – Inter-VPC communication with Network Connectivity Center

In this more modern design, we use Network Connectivity Center with a star configuration and interconnect spokes. To understand how to think about designing your network in this configuration, let’s look at the flow of a packet from an external network to an application located in the workload VPC 1.

The network consists of an external network (on-prem and other clouds), and the Google Cloud network (transit VPC, service access VPC, managed services VPC, Private Service Connect consumer VPC, and workload VPC).

This design uses the following services to provide an end-to-end solution.

• Cloud Interconnect (Direct, Partner, Cross-Cloud) – To connect from your on-prem or other clouds to the transit VPC. In this case multiple external locations are connecting in different regions.

• Cloud VPN – To connect from service access VPC to transit VPC and export custom routes from private services access network

• VPC Network Peering – To connect from workload VPC to transit VPC

• Private services access – To connect to managed services privately in the services access VPC

• Private Service Connect – To expose services in the managed services VPC network to be consumed in the services access VPC and Private Service Connect consumer VPC with endpoints to service made available to connected peers.

• Network Connectivity Center VPC spokes – To allow communication between workload VPCs if necessary

• Network Connectivity Center topology – Utilizes preset topologies (choose mesh or star depending on your requirements)

To understand the specific details such as the Network Connectivity Center star topology, route exchange and packet flow, please read the full architecture document reference guide: Cross-Cloud Network inter-VPC connectivity using Network Connectivity Center.

Next steps

Take a deeper dive into network migration support and Cross-Cloud Network.

• Document: Cross-Cloud Network for distributed applications

• Document: Designing networks for migrating enterprise workloads: Architectural approaches

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

For communication service providers (CSPs), a major hurdle in monetizing their networks is engaging the developer community effectively. Historically, complex, non-standardized APIs and a lack of developer-friendly resources have limited access to valuable network capabilities, preventing CSPs from fully capitalizing on the potential of their infrastructure.

However, by using platforms like Firebase, Google’s web and mobile application development platform, and embracing initiatives like the Open Gateway Initiative (OGI), CSPs can gain access to a vast pool of developers who are eager to build innovative applications with standardized APIs.

Today, we’re announcing a new Firebase phone number verification service, providing more than three million Firebase developers with access to critical network CSP APIs such as phone number acquisition and verification. We’re also partnering with Deutsche Telekom, Orange, Telefónica, T-Mobile, Telenor, Verizon, Vodafone and others to improve developer and end-user experiences, and strengthen security by mitigating fraud — creating new revenue opportunities for CSPs and developers alike. 

This work builds on the OGI vision to enable developers to build innovative applications and services through a global, standardized API framework that exposes network services. Google Cloud is proud to be part of this transformative movement, and we’re excited for the future it enables.

The challenge of traditional authentication methods

User authentication is critical for mobile and web app developers to build secure, personalized, and functional apps that protect user data, enable monetization, and foster trust. However, traditional methods, including SMS One-Time Passwords (OTPs), often create friction for users with limitations like slow delivery, interception risks, and unreliable cellular service. This can lead to frustrated users, abandoned sign-ups, and security vulnerabilities. Furthermore, bad actors increasingly exploit the SMS channel for fraudulent activities, such as artificially inflating traffic for profit. Developers and CSPs need more reliable, secure, and user-friendly verification alternatives.

OGI: A standardized solution for secure authentication

OGI, backed by the GSMA and global CSPs, has already set the stage for a more collaborative and innovative telecom ecosystem. The initiative currently provides a standardized framework for CSPs to expose their network capabilities through APIs, enabling developers to access core network functionalities, including secure user identity verification, in a consistent and predictable manner, regardless of the user’s mobile operator. By connecting the Firebase developer ecosystem to the broader OGI, we’re helping address some long-standing challenges, like complex APIs, consent models, fragmented billing experiences, and more. 

Benefits for users, developers, and CSPs alike

Firebase’s standardized approach to authentication offers a multitude of benefits:

• Enhanced security: By directly verifying user identity with CSPs, APIs significantly reduce the risk of fraud and unauthorized access.
• Improved privacy: The operating system manages the consent experience, providing intuitive and transparent controls in line with regional requirements.
• Improved user experience: Firebase enables smooth verification experiences, minimizing the friction of traditional methods like SMS one-time passwords (OTPs).
• Increased accessibility: Firebase works across both Wi-Fi and cellular networks, providing reliable authentication even in areas with poor cellular connectivity.
• Simplified development: Developers can access a global network of CSPs through a single, standardized API framework, reducing development time and complexity.
• New revenue streams for CSPs: Firebase helps CSPs monetize their network capabilities and become key players in the digital ecosystem.
• Simplified billing: Billing is centrally managed through Google Cloud, and CSPs get paid by Google.
• Access to millions of developers: By leveraging Google’s relationship with over three million developers, CSPs can scale the business quickly.

“Digital identity APIs are the future of safe online transactions as they provide secure and reliable evidence of users’ identities in a world of ever-increasing cyber threats. The partnership with Google enables us to bring next-generation authentication services to our customers, strengthening our overall efforts to provide unparalleled cyber security now and in the future.” – Nicholas Nikrouyan, VP Voice & Mobile Services, Deutsche Telekom.” 

“Privacy and security are key for Telefónica in the development of digital services. We are thrilled to partner with Google to accelerate the launch of more secure and user-friendly phone number verification solutions through Open Gateway’s APIs, seamlessly integrated with Cloud Firebase.” – Chema Alonso, Chief Digital Officer, Telefónica.

“We at Telenor Linx believe that simplicity is the key to unlocking secure digital experiences. By providing user-friendly verification and authentication solutions built on top of our core connectivity, we ensure that mobile operators remain central to protecting customers’ identities. Partnering with global development communities to drive open standards and reduce fraud opportunities is a natural next step for our industry, and we look forward to delivering safe, seamless access to services for businesses and consumers alike together with Google.” – Stig Waagbø, CEO, Telenor Linx

“Vodafone is collaborating with Google and other mobile operators to enable frictionless and cryptographically secure authentication. We believe this is key to enabling developers and stimulating the next wave of digital services,” – Johanna Wood, Director of Network APIs, Vodafone.

Privacy at the forefront

Both Google Cloud and the Open Gateway Initiative are committed to user privacy. OGI APIs are designed with strong privacy protections, ensuring that user data is handled responsibly and securely. Users maintain control over their data and provide explicit consent for any data sharing, consistent with established privacy best practices. Firebase expands on these promises with a unified, simple and consistent user experience independent of the market the user is operating in.

If you’re a CSP wanting to offer Firebase phone number verification on your network, please register your interest here. Going to Mobile World Congress (MWC), March 3-6, 2025 in Barcelona? Visit Google Cloud at Hall 2, Booth #2H40 to see experiences in action.

When it comes to data center power systems, batteries play an important role. The applications that run in our data centers require nearly continuous uptime. And while utility power is highly reliable, power outages are unavoidable. 

When an outage happens, batteries can supply short-duration power, allowing servers to operate continuously when the facility switches between AC power sources, or to ride through transient power disturbances. Or, if a facility loses both primary and alternate power sources for an extended period of time, batteries can supply sufficient power to allow machines to execute a clean shutdown procedure. This is helpful in expediting machine restarts after the power outage. More importantly, it helps ensure that critical user data is safely stored to disk and not lost in the power disruption. 

At Google, we rely on a 48Vdc rack power system with integrated battery backup units (BBUs), and in 2015, we became one of the first hyperscale data center providers to deploy Lithium-ion BBUs. These Li-ion batteries had twice the life, twice the power and half the volume of previous-generation lead-acid batteries. Switching from lead-acid batteries to Li-ion means we deploy only one-quarter the number of batteries, greatly reducing the battery waste generated by our data centers. 

We recently reached an important milestone: Google has more than 100 million cells deployed in battery packs across our global data center fleet. This is remarkable, and only possible thanks to the safety-first approach we take to deploy Li-ion batteries at scale.

The main safety risk associated with Li-ion batteries is the battery going into thermal runaway if it’s accidentally mishandled or exposed to excessive temperatures or overcharging. While a rare event, the resulting fire is extremely difficult to extinguish due to the large amount of heat generated, driving a thermal runaway chain reaction to nearby cells. 

To deploy this large fleet of Li-ion cells, we have had to make safety a core principle of our battery design. Specifically, as an early adopter of the UL9540A thermal runaway test method, we subject our Li-ion BBU designs to rigorous flame safety testing that demonstrates their ability to limit thermal runaway. As a result, Google has successfully been granted permits to deploy BBUs in some of the world’s most stringent jurisdictions, in the APAC region. 

In addition, our Li-ion BBUs benefit from our distributed UPS architecture that offers significant availability and TCO benefits compared to traditional monolithic UPS systems. The distributed UPS architecture improves machine availability by: 1) reducing the failure-domain blast radius to a single rack, and 2) locating the batteries in the rack to eliminate intermediate points of failure between the UPS and machines. This architecture also provides TCO benefits by scaling the UPS with the deployment, i.e., reducing day-1 UPS cost. Additionally, locating the batteries in the rack on the same DC bus as the machines eliminates intermediate AC/DC power conversion steps that cause efficiency losses. In 2016 we shared the 48V rack power system spec with the Open Compute Project, including specs for the Li-ion BBUs. 

Li-ion batteries have been crucial to ensuring the uninterrupted operation of Google Cloud data centers. By transitioning from lead-acid to Li-ion BBUs, we’ve significantly improved power availability, efficiency, and lifespan, even as we simultaneously address their critical safety risks. Our commitment to rigorous safety testing and adherence to standards and test methods like UL9540A has enabled us to deploy millions of Li-ion BBUs globally, providing our customers with the high level of reliability they expect from Google Cloud. 

Getting to 100 million Li-ion batteries is just one of many examples of how we are building a reliable cloud and power-efficient AI. As data center power systems evolve to include new technologies including large battery energy storage systems (BESS) and new workload requirements (AI workloads), we remain dedicated to exploring and implementing innovative solutions to build the most efficient and safest cloud data centers.

The authors would like to acknowledge Vijay Boovaragavan, Matt Tamashiro, Sandeep Sebastian, Thibault Pelloux-Gervais, Ken Wong, Mike Meakins, Stanley Fung, and Scott Sharp for their contributions.

Many specialized vector databases today require you to create complex pipelines and applications in order to get the data you need. AlloyDB for PostgreSQL offers Google Research’s, state-of-the-art vector search index, ScaNN, enabling you to optimize the end-to-end retrieval of the most fresh, relevant data with a single SQL statement.

Today, we are introducing a set of new enhancements to help you get even more out of vector search in AlloyDB. First, we are launching inline filtering, a major performance enhancement to filtered vector search in AlloyDB. One of the most powerful features in AlloyDB is the ability to perform filtered vector search directly in the database, instead of post-processing on the application side. Inline filtering helps ensure that these types of searches are fast, accurate, and efficient — automatically combining the best of vector indexes and traditional indexes on metadata columns to achieve better query performance.

Second, we are launching enterprise-grade observability and management tooling for vector indexes to help you ensure stable performance and the highest quality search results. This includes a new recall evaluator, or built-in tooling for evaluating recall, a key metric of vector search quality. That means you no longer have to build your own measurement pipelines and processes for your applications to deliver good results. We’re also introducing vector index distribution statistics, allowing customers with rapidly changing real-time data to achieve more stable, consistent performance.

Together, these launches further strengthen our mission of providing performant, flexible, high-quality end-to-end solutions for vector search that enterprises can rely on.

A review of filtered vector search in AlloyDB

Many customers start their journey with vector search trying simple search on a single column. For example, a retailer might want to perform a semantic search on product descriptions to surface the right products to match end-user queries.

However, very quickly, as you look to productionize these solutions and improve the quality of your results, you may find that the queries themselves get more interesting. You might iterate — add filters, perform joins with other tables, and aggregate your data. For example, the retailer might want to allow users to filter by size, price, and more.

AlloyDB’s PostgreSQL interface provides a strong developer experience for these types of workloads. Because vector search is integrated into the SQL interface, developers can very easily query structured and unstructured data together in a single SQL statement, as opposed to writing complex application code that pulls data from multiple sources. 

Moreover, changing requirements such as adding new query filters typically don’t require schema or index updates. If our retailer, for example, wants to only show in-stock items at the end user’s local store, they can very easily join their products table with an existing store inventory table via the SQL interface.

All of this, and more, is possible in AlloyDB! 

Inline filtering

But as a developer, you don’t just want to execute the query — you also want excellent performance and recall. To deliver the best performance, the AlloyDB query optimizer makes choices on how to execute a query with filters. Inline filtering, is a new query optimization technique that allows the AlloyDB query optimizer to evaluate both the metadata filtering conditions and the vector search in tandem, leveraging both vector indexes and indexes on the metadata columns. Inline filtering is now available for the ScaNN index in AlloyDB, a search technology based on over a decade of Google research into semantic search algorithms.

AlloyDB intelligently and automatically employs this technique when it’s most beneficial. Depending on the query and the distribution of the underlying data, the query planner automatically chooses the execution plan with the best performance. When filters are very selective, i.e., when a very small number of rows matches the filter, the query planner typically executes a pre-filter. This can leverage an index on a metadata column to find the small subset of rows that match the filter, and then perform a nearest-neighbor search on only those rows. Alternatively, the query planner may decide to execute a post-filter in cases of low selectivity — i.e., if a large percentage of rows match the filtered condition. Here, the query planner starts with the vector index to come up with a list of relevant candidates, and then removes results that do not match the predicates on the metadata columns. 

Inline filtering, on the other hand, is best for cases with medium selectivity. As AlloyDB searches through the vector index, it only computes distances for vectors that match the metadata filtering conditions. This massively improves performance for these queries complementing the advantages of post-filter or pre-filter. With this feature, AlloyDB provides great performance across the whole gamut of selectivities of filters when combined with vector search.

Enterprise-grade observability

If you’re running similarity search or generative AI workloads in production, you need stable performance and quality of results, just as you do for any other database workload. Observability and manageability tooling are key to achieving that.

With the new recall evaluator, built directly into the database, you can now more systematically measure, and ultimately tune, search quality with a single stored procedure in the database rather than build custom evaluation pipelines.

Recall in similarity search is the fraction of relevant instances that were retrieved from a search, and is the most common metric used for measuring search quality. One source of recall loss comes from the difference between approximate nearest neighbor search, or aNN, and k (exact) nearest neighbor search, or kNN. Vector indexes like AlloyDB’s ScaNN implement aNN algorithms, allowing you to speed up vector search on large datasets in exchange for a small tradeoff in recall. Now, AlloyDB provides you with the ability to measure this tradeoff directly in the database for individual queries and ensure that it is stable over time. You can update query and index parameters in response to this information to achieve better results and performance. This management tooling is critical if you care deeply about stable, high-quality results.

In addition to recall improvements, we’re also introducing vector index distribution statistics for the ScaNN index, allowing developers to see the distribution of vectors within the index. This is particularly useful for workloads with high write throughput or data change rates. In these scenarios, new real-time data is automatically added to the index and is ready for querying right away. Now, you can monitor any changes in vector-index distribution, and ensure that performance stays robust through these data changes.

Getting started

ScaNN for AlloyDB is generally available in AlloyDB. To get started with it, follow our quickstart guide to creating an AlloyDB instance, then read the documentation for some fast and easy vector queries. You can also now try AlloyDB for free with our 30-day free trials.

You can also tune into our live Data Cloud Innovation Webcast: Data Analytics and Databases to learn more about these new technologies.

To learn more about the ScaNN for AlloyDB index, check out our introduction to the ScaNN for AlloyDB index, or read our ScaNN for AlloyDB whitepaper for an introduction to vector search at large, and then a deep dive into the ScaNN algorithm and how we implemented it in PostgreSQL and AlloyDB.

The cloud is evolving fast — and that means you need to evolve fast. With the explosion of AI, it’s not enough to build skills; you have to be able to prove you have them. 

As more companies pursue digital transformation and AI adoption, validating skills quickly and effectively is more critical than ever. That’s not just the message everyone is hearing from recruiters and executives — we also have new research demonstrating the impact of getting Google Cloud certified. And recognizing just how valuable certification is, we’ve got three new ones to help drive your success in 2025. 

A recent Ipsos study commissioned by Google Cloud surveyed more than 3,000 cloud practitioners, students, and decision-makers and confirmed that certifications not only increase career opportunities, they also drive efficiencies for digital businesses (which is pretty much every business these days). From seasoned professionals to those just starting out, as well as the organizations for which they work, everyone benefits from validating their cloud skills with trusted certifications. 

Certifications: A catalyst for confidence and career advancement 

Certifications are a key driver of career growth for engineers, data scientists, and other cloud professionals. For Google Cloud learners, certifications are considered the most valuable part of their learning journey. This is supported by the Ipsos data: eight in 10 Google Cloud learners report that certification equips them with the skills needed for in-demand roles, accelerates their promotion potential, and contributes to their overall professional success when they share their credentials online.

The Ipsos research also reveals the significant impact of Google Cloud certifications on students.  Empowered by certifications, students report higher salaries and faster time-to-hire. An impressive nine in 10 Google Cloud certified students say their training made them more competitive in the job market, leading to better career opportunities.

Helping cloud leaders find and build more efficient teams 

Furthermore, certifications build confidence and efficiency for cloud leaders and decision-makers. Leaders from organizations using Google Cloud report that certifications significantly improve the efficiency of their cloud operations. They cite increased confidence in on-time project completion, accelerated onboarding to roles and projects, and greater confidence in a candidate’s knowledge during the hiring process. In fact, more than six in 10 leaders say one of the most important resources for cloud learners is getting certified, and approximately 70% believe certified employees are more productive.

Explore—and prepare for—the latest certifications from Google Cloud, integrated with AI concepts

To get started out take your training to the next level, you can explore the full catalog of Google Cloud certifications, which now include these newly launched certifications: 

1. Associate Data Practitioner Certification: This certification is a great fit for data scientists who want to validate their Google Cloud data skills and knowledge, like ensuring data is clean, secure, and usable for AI and machine learning models. Follow this learning path to prepare for the exam.

2. Associate Google Workspace Administrator Certification: Validate your proficiency in the core skills required to successfully manage Google Workspace environments, including effectively managing the AI-powered assistant. Follow this learning path to prepare for the exam.

3. Professional Cloud Architect Certification [Renewal]: Prove your skills as a professional cloud architect with this new, streamlined recertification exam, focused on the application of generative AI solutions to solve real-world business challenges. Check out the exam guide to prepare for the exam. 

How certification moves the needle: Hear from certified professionals

People are already transforming their careers and supercharging their teams with the help of certifications. Hear from a Chief Technology Officer, a senior cloud architect, a risk manager and a student in information systems about the difference a certification makes in their day-to-day:

The rapid pace of AI innovation has made skills validation an imperative — for cloud professionals and the companies they call home. Learn more about all your cloud credentialing options here or explore our full suite of Google Cloud learning tools at skills.google.