Google Cloud

SQL has been the cornerstone of data management for over 50 years, valued for its declarative nature and robust ecosystem. However, traditional SQL has limitations, including rigid clause structures, verbose syntax, and complex nested queries that can hinder readability and maintenance. To address these challenges, Google introduced pipe syntax, an extension to GoogleSQL that reimagines how queries are written and processed. A pipe-structured data flow makes SQL more easier to read and write than ever before.

As we discussed in a previous blog post, pipe syntax introduces a linear, top-down approach to SQL queries, using the pipe operator (|>) to chain operations like filtering, aggregating, and joining in a logical sequence. This structure aligns with the natural flow of data transformations, making queries more intuitive, readable, and maintainable. Unlike standard SQL, which enforces a strict order (SELECT, FROM, WHERE, etc.), pipe syntax allows operations to be applied in any order, reducing the need for cumbersome subqueries or Common Table Expressions (CTEs). 

Since pipe syntax became generally available in April, we have seen you embrace it across a myriad of use cases — from streamlining data transformations, building insightful reports to efficiently analyzing logs, and many more. In this blog, we are highlighting three different use cases from customers about how they are using pipe syntax to simplify data transformations and log data analysis.

1.Supercharging data analysis with a linear data flow 

Imagine you want to build a recommendation report to suggest whether a logical or physical pricing model is more cost effective for long-term data storage in BigQuery. While physical storage is generally cheaper, costs can vary based on factors like data compressibility.

You might write a query like this using standard SQL:

Using pipe syntax, the query can be written as:

Pipe syntax eliminates the need for CTEs, allowing users to chain operations linearly. This reduces the query’s complexity and makes it easier to follow, especially for cost analysis tasks requiring multiple transformations. Now you can build a report faster, and share the queries across teams that are easier for them to understand. 

2. Simplifying data transformations in data pipelines 

Suppose you are building a data pipeline for a dashboard. You want to process customer order data to calculate total revenue per customer, filter high-value customers, and then rank them.

In standard SQL:

Using pipe syntax, the query can be written as:

Building a data pipeline is a multi-step process involving data cleaning, transformation and aggregation. Pipe syntax simplifies developers’ workflows by allowing them to chain operations like aggregation and filtering in a logical and sequential way. The simplicity of the code allows developers to focus more time on solving business problems instead of writing and debugging cumbersome SQL code. Fewer lines of code and higher productivity eventually translate to faster business outcomes. 

3. Simpler log analysis in Google Cloud Logging 

Log Analytics in Google Cloud Logging is a powerful tool for storing, searching, analyzing, and monitoring application and system log data through SQL. When you need to perform complex analysis, aggregations, or transformations on their logs, the Log Analytics feature gives you the full strength of SQL.

When helping enterprises to migrate their workloads and logging tooling to Cloud Logging, translating pipe-like structure into standard SQL queries for typical log analysis tasks can sometimes feel cumbersome; it involves subqueries and common table expressions (CTEs) that might not feel immediately intuitive. Pipe syntax offers a more linear, pipe-based syntax, often making common log analysis patterns easier to read and write.

Let’s look at a concrete example.

The challenge: Analyzing log frequency by time, severity, and resource attribute

Imagine you want to analyze your logs to find patterns where specific log messages (identified by severity and tags) occur frequently (more than 5 times per minute), excluding certain messages (like those containing “CookieIncluded=False”).

Using standard SQL in Log Analytics, you might write a query like this:

Using pipe syntax, the query can be written as:

Beyond looking different, pipe syntax fundamentally changes how you build and read queries for sequential log processing. It tackles two common pain points of standard SQL: nested subqueries and “inside-out” logic.

PipeSQL eliminates this nesting for common sequential tasks. Each pipe (|>) feeds the previous command’s output into the next. Aggregation results from the summary are immediately available for filtering in the subsequent WHERE clause without needing an explicit nested structure. The query remains flat and linear. By tackling nesting and promoting a linear flow, pipe syntax significantly lowers the cognitive load for many common log analysis tasks, making powerful analytics more accessible.

Customer love for pipe syntax 

“I was very excited when BigQuery first introduced pipe syntax. As someone who is familiar with R, it is very natural for myself and our team of analysts to use pipe-like syntax for more streamlined data transformation via Dataform. I am happy to share that I have seen >30% reduction in my code compared to standard SQL. This eventually transferred to time saving to read, write and debug queries. I am looking forward to onboarding more analysts to use pipe syntax across my organization.” – Shanker Venkatachalam, Senior Manager, Business Strategy, Northwell.edu

“The pipe syntax has proven to be a revelation in my analytical work. Although I initially had concerns about the learning curve, it was surprisingly smooth. The linear flow facilitates step-by-step query construction, ensuring each transformation is correct before moving on. This reduces complexity and improves long-term maintainability. Notably, it also naturally encourages a ‘micro-transformations’ mindset, resulting in significantly more organized and understandable queries.”– Axel Thevenot, Google Developer Expert, Venom Engineering

“At Bindplane, we specialize in accelerating migrations to Cloud Observability with our telemetry pipeline built on OpenTelemetry. Once the data arrives, using pipe syntax makes it much easier to build key observability assets like dashboards, alerts, and queries. Once pipe syntax is available, I’m done writing regular SQL. It’s just so much easier for helping enterprises get up and running with Cloud Logging/Log Analytics.” – Keith Schmitt, Senior Software Engineer, Bindplane 

What’s next for pipe syntax?

As shared at Cloud Next, BigQuery supports Gemini-powered Data Preparation Agents, which can analyze your data and auto-generate data pipelines to clean and transform your data. You will soon be able to use pipe queries to validate generated queries, which simplifies your pipelines and makes it easier to review and verify code.

In short, pipe syntax is more than a syntactic upgrade. It’s a paradigm shift that makes SQL more accessible and productive for analysts, engineers, and data teams. Its modular, pipeline-based approach aligns with modern data workflows, saving development time and improving efficiency. Whether you’re building reports, developing data pipelines, or analyzing logs, pipe syntax delivers shorter, clearer, and more efficient queries, empowering you to focus on insights rather than syntax. 

Ready to simplify your SQL workflows? Learn how to use pipe syntax from this tutorial, and check out the following resources: 

• Pipe syntax documentation and detailed reference guide 

• Demo video, tutorial 

• Blog post on using pipe syntax for log data analysis

• Research publication (VLDB 2024) with more background and language design details

• Open-source GoogleSQL implementation, ZetaSQL

As your operational needs change, sometimes you need to move data residing within Google’s Cloud Storage to a new location, to improve resilience, optimize performance, meet compliance needs, or simply to reorganize your infrastructure. Yet moving buckets can be a daunting, complex, risky endeavor that involves manual scripting, painstaking coordination, and the risk of data loss, or worse yet, extended downtime. This can discourage organizations from making the changes they need to their storage environments.

We recently introduced Cloud Storage bucket relocation, a unique feature among leading hyperscalers that makes it easy  to change your bucket’s location. Bucket relocation eliminates the need for complex manual planning and helps prevent extended downtime, for an easy transition with minimal application disruption, and strong data integrity. Your bucket’s name, and all the object metadata within it, remain identical throughout the relocation, so there are no path changes, and your applications experience minimal downtime while the underlying storage is moved. Furthermore, your objects retain their original storage class (e.g., Standard, Nearline, Coldline, Archive) and time-in-class in the new location. This is key for many cost efficiency strategies, helping ensure capabilities such as Autoclass continue to operate intelligently to optimize your storage costs post-migration.

Bucket relocation is a key capability within the Storage Intelligence suite, alongside tools like Storage Insights, which  provides deep visibility into your storage landscape and identifies optimization opportunities. Bucket relocation then lets you act on these insights, and move your data between diverse Cloud Storage locations — regional locations for low latency, dual-regions for high availability and disaster recovery, or multi-regions for global accessibility — to meet your  business, performance, and compliance objectives.

Bucket relocation under the hood

Bucket relocation relies on two critical techniques. 

• Asynchronous data copy: Bucket relocation leverages a unique and optimized asynchronous data transfer mechanism that copies data in the background to  minimize impact to ongoing operations. Existing operations like writing, reading, and updating objects continue while the entire dataset is being copied.

• Metadata preservation: Historically, Google Cloud customers moved data with the Storage Transfer Service, which copied the objects to a new bucket and deleted existing ones. Bucket relocation, on the other hand, automatically and meticulously moves all your bucket’s and objects’ associated metadata, thereby preserving state. This includes information like:

• Storage class: Your objects retain their original storage class (e.g., Standard, Nearline, Coldline, Archive) in the new location.

• Bucket and object names: The naming structure of your buckets and objects remains identical.

• Creation and update timestamps: These markers are preserved, so that features like object lifecycle management (OLM) rules continue to operate.

• Access Control Lists (ACLs) and IAM policies: Bucket- and object-level permissions are transferred to help maintain your security posture.

• Custom metadata: Any user-defined metadata associated with your objects is also migrated.

By handling the complexities of asynchronous data transfer and automatic metadata migration, bucket relocation minimizes the risks and overhead associated with a manual bucket migration. Crucially, because the bucket name is preserved throughout the relocation process, applications accessing the bucket don’t need to be modified.

Relocate your bucket in a few simple steps

With bucket relocation, you can move your Cloud Storage buckets in three simple steps. Here’s a breakdown:

1. Initiate a dry run:

• Before starting the actual relocation, it’s highly recommended to perform a dry run. This simulates the process without moving any data, allowing you to identify potential issues early on, such as incompatible configurations.
• The dry run checks for incompatibilities like customer-managed encryption keys (CMEK), locked retention policies, objects with temporary holds, and bucket tags, without you having to manually validate each of them.
• Make sure to add the --dry-run flag!

Replace BUCKET_NAME with the name of your bucket and LOCATION with the desired destination.

2. Start the relocation process:

• This step initiates the actual data transfer from the source bucket to the destination bucket. During this phase, you can still read, modify, and delete objects in the bucket. However, the bucket metadata (i.e., bucket-level  parameters and configurations) is write-locked to prevent changes that could affect the relocation.
  Note: Removing the --dry-run flag from the dry-run command initiates the relocation.

3. Finalize the relocation process:

• Once the incremental data copy is complete, you’re ready to trigger the final synchronization step (except when moving between multi-region and configurable dual-region). This involves a brief period where writes to the bucket are disabled to help ensure their data integrity; any last-second changes made to the objects within the bucket while the incremental copy was in progress are copied to the destination. After the data’s integrity is verified, the bucket’s location is updated, and all requests are automatically redirected to the new location. During the final synchronization step, attempts to update objects in the bucket will result in an HTTP 412 error.
• Do not initiate the final synchronization process until the relocation process progress reaches ~99%. This helps you minimize downtime because most of the data has already been synchronized in the background. 

Note: If you’re moving between multi-regions and configurable dual-regions within the same multi-region code, you’re all set — bucket relocation handles the transition in the background, no finalization or downtime required!

The OPERATION_ID is provided as output from Step-2. The OPERATION_ID is listed with the keyword name. For instance:

name: projects/_/buckets/my-bucket/operations/AbCJYd8jKT1n-Ciw1LCNXIcubwvij_TdqO-ZFjuF2YntK0r74

And there you have it — In just three steps, you’ven moved your entire bucket, its data, and metadata, to its new location.

Early users of bucket relocation have had great success with the new feature. 

“With Storage Intelligence and bucket relocation, we effortlessly transitioned to dual-region buckets. The seamless process, powered by the bucket relocation, minimized downtime and ensured data integrity. We migrated the buckets with peace of mind and without the manual headaches.” – Adam Steele, Product Manager, Spotify

“We recently utilized the bucket relocation feature of Storage Intelligence to successfully complete a ~300 bucket migration and PBs of data project from multi-region to regional storage, to optimize network data transfer costs. Without bucket relocation, this process would have required extensive automation and scripting, resulting in increased downtime and effort.” – Deepak Mahato, Data Platform Infrastructure Manager, GroupOn

Experience the ease and efficiency of managing your Cloud Storage buckets with bucket relocation in Storage Intelligence. To learn more, visit the bucket relocation documentation and the Storage Intelligence overview.

In today’s hyperconnected world, delivering personalized content at scale requires more than just aggregating information – it demands deep understanding of context, relationships, and user preferences. Glance, a leading content discovery platform that delivers personalized, real-time content experiences on mobile lock screens across the globe, serves over 300 million users worldwide across 450 million devices. Beyond news aggregation, Glance curates diverse content including entertainment, sports, gaming, shopping, and lifestyle content, making every glance at the phone screen meaningful and engaging.

However, with the exponential growth of digital content, particularly the overwhelming volume of daily news articles, Glance faced a critical challenge: how to effectively navigate this information while maintaining the personalized, contextual experiences users expect. The existing search and content discovery capabilities needed significant enhancement to uncover emerging trends, improve search relevance, provide deeper context, and most importantly, deliver truly personalized content recommendations that resonate with individual user preferences and behaviors.

Glance partnered with Google Cloud Consulting team to build a sophisticated Content Knowledge Graph (CKG) that addresses these challenges head-on. This solution leverages Google Cloud’s advanced AI and data processing capabilities, including Gemini models, BigQuery, Vertex AI, and Google Cloud partner Neo4j, to ingest, process, extract, standardize, classify, and structure news data into a dynamic network of interconnected entities and relationships. The Content Knowledge Graph has dramatically improved search relevance, enhanced personalized content discovery, provided deeper contextual insights, increased user engagement, and improved scalability and efficiency.

Tackling the content discovery and personalization challenge  

Glance’s mission extends far beyond traditional content aggregation. As a platform serving hundreds of millions of users globally, Glance must deliver hyper-personalized content experiences that anticipate user interests and adapt to evolving preferences in real-time. The challenge was multifaceted:

• Uncovering trends at scale – Identifying emerging topics, viral content, and the complex relationships between different content categories across news, entertainment, sports, and lifestyle domains

• Enhancing personalized search – Improving the accuracy and relevance of search results based on individual user behavior patterns, preferences, and contextual signals

• Providing intelligent context – Offering users deeper understanding not just of individual pieces of content, but how different stories, events, and topics connect across the broader content ecosystem

• Scaling personalization – Delivering these sophisticated experiences across 300+ million users while maintaining real-time responsiveness and relevance

Manually analyzing and connecting the dots within this vast, multi-dimensional content landscape was simply not scalable. Glance needed an intelligent, automated approach that could understand content at both granular and contextual levels while powering personalized recommendations at unprecedented scale.

Building a Gemini-powered content intelligence engine

Glance and Google Cloud Consulting team collaborated to architect and implement a comprehensive Content Knowledge Graph that transforms how content is understood, connected, and delivered. This sophisticated system leverages the full spectrum of Google Cloud’s AI and data processing capabilities:

• Intelligent content ingestion and processing – The system ingests content from diverse sources beyond news, including entertainment articles, sports updates, lifestyle content, and trending topics, storing them in BigQuery for efficient, scalable processing.

• Advanced entity extraction and relationship mapping – Using Gemini foundational models, the system extracts key entities (people, organizations, locations, events, brands) and identifies complex relationships between them across different content categories.

• Entity standardization and knowledge linking – Extracted entities are normalized using Gemini’s advanced language understanding and linked to authoritative knowledge sources like Wikipedia, ensuring consistency and enabling sophisticated cross-content analysis.

• Multi-dimensional content classification – Gemini foundation models classify content into granular categories using the IAB content taxonomy while also identifying sentiment, urgency, and relevance signals for personalization.

• Intelligent content summarization and tagging – Gemini generates contextual tags, compelling short headlines, and category labels, enabling users to quickly grasp content essence while powering recommendation algorithms.

• Dynamic Knowledge Graph construction – The extracted information is structured into a Neo4j graph database, creating a living, breathing network of interconnected entities, topics, relationships, and user interaction patterns.

• Real-time trend analysis and prediction – The system integrates with external trend APIs and analyzes user engagement patterns to identify and predict trending topics, providing actionable insights for content curation.

• Interactive analytics dashboard – NeoDash powers an interactive dashboard for monitoring trending content, analyzing entity relationships, and visualizing content performance across different user segments.

Diagram 2:

Engineering for Global Scale and Performance

Glance enhanced the Content Knowledge Graph solution to handle 50,000+ daily articles across multiple content categories. The engineering team implemented several critical optimizations:

• Event-driven architecture transformation – Migrating to a Kafka-based event-driven architecture with intelligent retries and asynchronous operations resulted in 4x throughput improvement, enabling real-time content processing at massive scale.

• Graph database optimization – Neo4j query optimization and indexing strategies drastically reduced query response times from seconds to milliseconds, enabling real-time content recommendations.

• Kubernetes-native deployment – Moving to managed Google Kubernetes Engine (GKE) with auto-scaling capabilities improved system reliability and resource utilization.

• Strategic performance optimization – Applying the 80-20 principle to focus on high-impact optimizations, coupled with Redis caching and Cloud Spanner for critical data, reduced processing latency by 80% and boosted recommendation coverage from under 60% to over 85%.

• Intelligent load balancing – Implementing smart load distribution across processing pipelines ensured consistent performance even during viral content spikes and peak traffic periods.

 Business impact

The Content Knowledge Graph has delivered measurable improvements across key business metrics:

• Enhanced content discovery performance – CKG-powered content recommendations boosted Cards per Session (CPS) by 24%, directly improving user engagement and platform stickiness.

• Significantly increased user engagement – More relevant and contextually aware content delivery resulted in higher click-through rates and a 5% increase in swiping sessions, particularly in the related content sections.

• Real-time trend intelligence – Users now discover trending topics instantly across news, entertainment, sports, and lifestyle categories, with faster trend detection compared to previous systems.

• Data-driven content strategy – The CKG provides comprehensive, actionable insights into content performance, user preferences, and emerging trends, enabling data-driven editorial and curation decisions.

• Global scalability and efficiency – The cloud-native architecture seamlessly handles Glance’s ever growing global content pool while maintaining cost efficiency and performance.

Shaping the Future of content discovery

The Glance Content Knowledge Graph transforms  raw, unstructured content into a sophisticated, interconnected knowledge network, and has  has empowered Glance to deliver truly engaging content experiences that anticipate user needs.The solution’s success lies not just in its technical sophistication, but in its ability to enhance the human experience of content discovery – making every interaction with Glance more meaningful, relevant, and engaging.

As content continues to proliferate and user expectations for personalization grow, the principles and technologies demonstrated in this project provide a blueprint for the future of intelligent content platforms. We’re excited to see how Glance leverages this powerful foundation to further innovate in personalized content discovery and set new standards for user experience in the digital content ecosystem.

At Google Cloud, we are committed to making it as seamless as possible for you to build and deploy the next generation of AI and agentic applications. Today, we’re thrilled to announce that we are collaborating with Docker to drastically simplify your deployment workflows, enabling you to bring your sophisticated AI applications from local development to Cloud Run with ease. 

Deploy your compose.yaml directly to Cloud Run

Previously, bridging the gap between your development environment and managed platforms like Cloud Run required you to manually translate and configure your infrastructure. Agentic applications that use MCP servers and self-hosted models added additional complexity. 

The open-source Compose Specification is one of the most popular ways for developers to iterate on complex applications in their local environment, and is the basis of Docker Compose. And now, gcloud run compose up brings the simplicity of Docker Compose to Cloud Run, automating this entire process. Now in private preview, you can deploy your existing compose.yaml file to Cloud Run with a single command, including building containers from source and leveraging Cloud Run’s volume mounts for data persistence. 

Supporting the Compose Specification with Cloud Run makes for easy transitions across your local and cloud deployments, where you can keep the same configuration format, ensuring consistency and accelerating your dev cycle.

“We’ve recently evolved Docker Compose to support agentic applications, and we’re excited to see that innovation extend to Google Cloud Run with support for GPU-backed execution. Using Docker and Cloud Run, developers can now iterate locally and deploy intelligent agents to production at scale with a single command. It’s a major step forward in making AI-native development accessible and composable. We’re looking forward to continuing our close collaboration with Google Cloud to simplify how developers build and run the next generation of intelligent applications.” – Tushar Jain, EVP Engineering and Product, Docker

Cloud Run, your home for AI applications

Support for the compose spec isn’t the only AI-friendly innovation you’ll find in Cloud Run. We recently announced general availability of Cloud Run GPUs, removing a significant barrier to entry for developers who want access to GPUs for AI workloads. With its pay-per-second billing, scale to zero, and rapid scaling (which takes approximately 19 seconds for a gemma3:4b model for time-to-first-token), Cloud Run is a great hosting solution for deploying and serving LLMs. 

This also makes Cloud Run a strong solution for Docker’s recently announced OSS MCP Gateway and Model Runner, making it easy for developers to take the AI applications locally to production in the cloud seamlessly. By supporting Docker’s recent addition of ‘models’ to the open Compose Spec, you can deploy these complex solutions to the cloud with a single command.  

Bringing it all together

Let’s review the compose file for the above demo. It consists of a multi-container application (defined in services) built from sources and leveraging a storage volume (defined in volumes). It also uses the new models attribute to define AI models and a Cloud Run-extension defining the runtime image to use:

Building the future of AI

We’re committed to offering developers maximum flexibility and choice by adopting open standards and supporting various agent frameworks. This collaboration on Cloud Run and Docker is another example of how we aim to simplify the process for developers to build and deploy intelligent applications. 

Compose Specification support is available for our trusted users — sign up here for the private preview.

AI is making it easier than ever to get clear, reliable answers from your data. With intelligent tools like the Conversational Analytics API, powered by Gemini, you no longer need intricate systems to get insights.

The Conversational Analytics API lets you use everyday language to ask questions of your data in BigQuery or Looker, with more sources to come. This brings powerful business intelligence to your entire organization through the apps you already use, like Slack. Developers can also build custom data agents that understand your business and provide answers where they’re needed. 

Now in preview, this API transforms unstructured conversation and structured data into actionable insights, accessible through natural language. You can create custom data agents, map business terms, and define calculations to empower your users and analysts.

Conversational Analytics API’s intelligent toolkit

The Conversational Analytics API integrates multiple AI-powered tools to process user requests, including Natural Language to Query (NL2Query) and a Python code interpreter for generating responses. A critical context retrieval tool is also utilized to ensure the API’s answers are accurate and relevant by incorporating details about your specific datasets. 

Let’s take a peek under the hood:

Context retrieval: The key to unlocking relevant answers

For intelligent conversational data interactions, leveraging the context retrieval tool is essential. For BigQuery, the tool meticulously pulls schema information, detailed column, table descriptions from Dataplex. When interacting with Looker, it accesses the LookML model, retrieving field definitions, labels, and defined measures. This deep, accurate understanding of your data’s structure, relationships, and inherent business logic is critical as it enables the agent to become an expert in your data landscape. This ensures every response is firmly grounded, highly relevant, and ultimately, a trusted answer for your business, providing reliable and trusted answers.

NL2Query engine: Turning questions into queries

At the core of the Conversational Analytics API lies a robust NL2Query engine, designed to support both BigQuery and Looker data sources. This engine translates user-provided natural language questions into semantically equivalent and syntactically correct queries appropriate for the specified data source. 

For instance, a user could pose a question such as, “What is the average sales value per order item broken down by payment method?” and the engine would process this to generate and execute the necessary query, providing a precise answer without requiring the user to write any SQL. The NL2Query engine’s capability includes handling ambiguity and inferring the customer’s intent from natural language input, ensuring accurate mapping to the underlying dataset structure.

Python code interpreter: Unleash advanced analytics

Building upon basic query capabilities, the code interpreter functionality within the Conversational Analytics API leverages Python to facilitate advanced analytical tasks. This allows the API to generate and execute Python code for complex calculations and statistical analyses, going beyond what standard query languages can achieve. Users can address intricate scenarios that would be difficult or impossible with SQL alone, accessing sophisticated analyses like statistical modeling and data transformations through Python libraries, all via a conversational interface without needing direct Python coding.

For instance, a user could request to “Calculate churn rate by customer segment, and show me the distribution.” In response, the code interpreter would automatically generate and execute the necessary Python code to perform this analysis and deliver insightful visualizations and statistics.

If you are Interested in testing out the code interpreter tool, currently in preview. Learn more here.

Intelligent visualization engine for data stories that pop

Data in its raw form can often be challenging to interpret. The visualization engine within the Conversational Analytics API addresses this by transforming the results of your queries into compelling visual representations. This capability streamlines the data visualization process, providing benefits such as instant chart generation without the manual steps of selecting data, labeling axes, or choosing chart types and colors. The result is the transformation of numerical data into visually engaging and easily digestible insights that effectively highlight hidden patterns and trends.

For example, a user could request to “Plot total orders by month,” and the system would instantly generate a relevant visualization, such as a line chart, effectively illustrating sales performance over time, ready for sharing and analysis.

Start having intelligent conversations with your data

Stop struggling with data complexity. Request a demo today or turn on Conversational Analytics in Looker to begin chatting with your data. You can even bring Conversational Analytics to your own application with this API. Get started in minutes with our quickstart application here.

When it comes to large-scale data analysis, BigQuery is a powerhouse, with fundamental aggregate functions like SUM, AVG, and COUNT allowing you to extract meaningful insights for all types of workloads. And today, we’re excited to take your data analytics to the next level with a suite of advanced aggregation features that unlock more complex and sophisticated use cases. These aggregate functions fall into three categories: 

• Group by extension (grouping sets/cube, group by struct, array, group by all)

• User-defined aggregate functions (Javascript/SQL UDAFs) 

• Approximate aggregate functions (KLL quantiles, Apache DataSketches)

We built these functions as they were top feature requests from our customer council group. Here is what New York Times had to say:

“I just want to say thank you to BigQuery’s team who launched GROUP BY ROLLUP. We had a daily query taking more than 2 hours to run that now takes 10 minutes using this instead, and many other teams that want to use it now because of it. We saw slot consumption drop by about 96%!” – Edward Podojil, consultant, The New York Times

Let’s take a deeper look at these new aggregate functions, and how to use them in your data analytics workflows. 

Group-by extensions

Grouping sets provide flexibility in calculating aggregations in multiple dimensions in one single statement without having to use UNION ALL. Group by struct, array allows you to easily group by commonly used data types in BigQuery. Group by lets you group by all non-aggregate columns in the Select statement without having to repeat each column twice. 

GROUP BY GROUPING SETS, CUBE (GA) 

Users often need to slice and dice their data in multiple dimensions. Currently you have to rely on repeated UNION ALL/CROSS Join to group your data, which can make queries cumbersome and hard to understand. GROUP BY GROUPING SETS allow you to group different dimensions in a single statement. For example, the following query give you the sum of amount grouped by different combinations: 

• Date: total sales per day

• Region: total sales per region

• Product: total sales per product

GROUP BY CUBE(x, y) is a shorthand syntax for GROUP BY GROUPING SETS ((x,y), x, y, ()), so you can GROUP BY all combinations of different dimensions.

GROUP BY STRUCT, ARRAY (GA)
STRUCT and ARRAY are among the most commonly used data types in BigQuery today. Working with STRUCT and ARRAY data in BigQuery just got easier! You can now use GROUP BY and SELECT DISTINCT for these semi-structured data types directly. This means no more time-consuming workarounds like converting STRUCT/ARRAY to JSON strings. This will simplify your queries and boost performance, making complex analysis more efficient (documentation).

GROUP BY ALL (GA)
GROUP BY ALL deduces non-aggregated columns from the SELECT clause, eliminating the need to list the same columns twice in SELECT and GROUP BY. It’s often used in queries with many dimensions and few aggregations. Listing all columns twice would be long and tedious. For example, you can group Chicago taxi trips by company, payment_type and taxi_id with a simple GROUP BY ALL query.

User-defined aggregate functions (Javascript/SQL UDAF)

User-defined aggregate functions, or UDAFs, let you define custom aggregations once, and reuse them across projects and teams. You no longer need to write repeated logic again and again to unlock advanced functionality like weighted average, merging JSON data, or even simulating geospatial functions.

Javascript UDAF (GA)

JavaScript user-defined aggregate functions (JS UDAFs) let you create custom aggregation logic beyond built-in functions, so you can calculate metrics tailored precisely to your needs. For instance, you can craft UDAFs to compute weighted averages, specialized statistics, or even construct data sketches. Here’s an example of a JavaScript UDAF that simulates the mode() function, returning the most frequent value within a group.

SQL UDAF (GA)

SQL user-defined aggregate functions (SQL UDAFs) allow users to encapsulate complex aggregate expressions into a UDF, for composability and reusability without having to write the same code again and again. For example, you can wrap multiple aggregate function calls into a UDAF through a struct constructor.

Approximate aggregate functions 

Companies across ad-tech, retail, fin-tech usually have massive amounts of multi-dimensional data. However, extracting data insights from millions or billions of rows of user-behavior data can be very expensive and time-consuming. Many companies are willing to calculate approximate aggregated results within defined error bounds, to get faster responses. Sketches enable approximate estimates of distinct counts, quantiles, histograms, and other statistical measures — all with minimal memory and computational overhead, and with a single pass through the data at scale. 

KLL quantile functions (Preview)

BigQuery supports quantile calculations using native KLL quantile functions. For example, you could estimate the median trip duration and trip distance for all taxi rides on a given day or month. 

First, create daily KLL quantile sketches over trip_seconds and trip_miles from the `chicago_taxi_trips` data table. 

Then, with EXTRACT_POINT_INT64/FLOAT64 functions, you can get the median trip_seconds and trip_miles for all trips on a specific day.

You can also leverage the MERGE_POINT_INT64/FLOAT64 functions to estimate the median trip_seconds and trip_miles for all trips in a month. The query merges daily sketches at first, and then calculates quantile values over a month. 

Apache DataSketches (GA) 
Besides native support for sketches, BigQuery also supports open-source Apache DataSketches, a high-performance library of stochastic streaming algorithms. Initially developed by Yahoo, you can directly use them though the public UDF bigquery-utils repo powered by JS UDAFs. Here are some examples, just name a few (see more details in this blog post): 

1. Theta Sketch: designed for cardinality estimation and set operations (union, intersection, difference)

2. KLL Sketch: designed for quantile estimation

3. Tuple Sketch: an extension of the Theta Sketch that supports associating values with the estimated unique items

The following examples use Theta Sketch over a public dataset, Chicago taxi trips, to demonstrate the power of the UDAF sketches library. 

First, create a daily Theta Sketch to estimate the unique number of taxis running per day, using theta_sketch_agg_string.

You can estimate the unique number of taxis that run on both days, using theta_sketch_intersection.

You can estimate the unique number of taxis that run on either of these two days, using theta_sketch_union.

Do more with advanced aggregation

Using BigQuery’s advanced aggregation capabilities allows you to perform more complex analysis efficiently. Grouping sets provide flexibility in calculating aggregations at multiple dimensions. UDAFs empower you to define custom aggregations. Approximate aggregate functions provide speed, scalability and performance when approximate results are acceptable. By leveraging these features, you can unlock deeper insights from your data and help your team make timely business decisions. We can’t wait to hear your use cases and how you plan to use them in your day to day analysis!

Accurate time-series forecasting is essential for many business scenarios such as planning, supply chain management, and resource allocation. BigQuery now embeds TimesFM, a state-of-the-art pre-trained model from Google Research, enabling powerful forecasting via the simple AI.FORECAST function.

Time-series analysis is used across a wide range of fields including retail, healthcare, finance, manufacturing, and the sciences. Through the use of forecasting algorithms, users can have a more thorough understanding of their data including the recognition of trends, seasonal variations, cyclical patterns, and stationarity. 

BigQuery already natively supports the well-known ARIMA_PLUS and ARIMA_PLUS_XREG models for time-series analysis. More recently, with the rapid progress and success of large pre-trained LLM models, the Google Research team developed TimesFM, a foundational model specifically for the time series domain.

The Time Series foundation model

TimesFM is a forecasting model that’s pre-trained on a large time-series corpus of 400 billion real-world time-points. A big advantage of this model is its ability to perform “zero-shot” forecasting. This means that it can make accurate predictions on unseen datasets without any training. In terms of the architecture, TimesFM is built as a decoder-only transformer model, which outputs batches of contiguous time-point segments at a time. This model has been featured on the GIFT-Eval benchmark and Monash public dataset, with a variety of public benchmarks from different domains and granularities. While ARIMA_PLUS offers customizability and explainability, the TimesFM model provides high ease-of-use and delivers good generalizability across many business domains, and often beats custom trained statistical and deep learning models.

How BigQuery supports TimesFM

The latest TimesFM 2.0 is now a native model in BigQuery. With 500 million parameters, TimesFM model inference runs directly on BigQuery infrastructure so there are no models to train, endpoints to manage, connections to set up, or quotas to adjust. TimesFM in BigQuery is also fast and scalable — you can forecast millions of univariate time series in a few minutes with a single SQL query.

Examples of the new AI.FORECAST function

To demonstrate, consider a use case that relies on the public bigquery-public-data.san_francisco_bikeshare.bikeshare_trips table. This dataset contains information about individual bicycle trips taken using the San Francisco Bay Area’s bike-share program. 

Example 1: Single time series

The following query aggregates the total number of bike-share trips on each day and forecasts the number of trips for the next 10 days (the default horizon).

The results look similar to:

The output includes the forecast timestamp and values through the columns forecast_timestamp and forecast_value. The confidence_level is default as 0.95. The prediction_interval_lower_bound and prediction_interval_upper_bound show the bounds for each forecasted point.

Example 2: Multiple time series

The AI.FORECAST function also lets you forecast multiple time series at a time as shown in the following example. The following query forecasts the number of bike share trips per subscriber type and per hour for the next month (approximately 720 hours), based on the previous four months of historical data.

The results look similar to the following:

In addition to the columns used by the single time series, there’s the time series identifier column, which we defined earlier as subscriber_type.Visualize the results 

You can merge the history data and forecasted data and visualize these results. The following graph visualizes the ‘Subscriber’ time series with its the lower and upper bounds of the prediction interval as follows:

You can see the detailed queries we used to generate this in the tutorial.

When to use TimesFM vs ARIMA_PLUS

For quick, out-of-the-box forecasts, establishing baselines, or identifying general trends with minimal setup, use TimesFM. If you need to model specific patterns, fine-tune forecasts for seasonality or holidays, multivariate (ARIMA_PLUS_XREG), require explainable results, or want to leverage a longer historical context, ARIMA_PLUS is the more suitable choice.

Take the next step

The TimesFM 2.0 model is now available in BigQuery in preview. For more details, please see the tutorial and the documentation.

Today, we’re announcing Sol, a new transatlantic subsea cable system to connect the U.S., Bermuda, the Azores, and Spain. Sol translates to “sun” in Spanish and Portuguese, a nod to the cable system’s landing points in warmer climates. Alongside the Nuvem subsea cable, Sol completes our unique investment in transatlantic resiliency — with the two systems interconnecting terrestrially in the U.S. and in Iberia, as well as in Bermuda and the Azores. 

The Sol cable will be manufactured in the U.S. and, once operational, bolster capacity and reliability for our growing network of 42 Google Cloud regions around the world, helping meet growing customer demand for Google Cloud and AI services across the U.S., Europe, and beyond.  

Palm Coast, Florida, will anchor the Sol cable in the U.S. We’ll partner with DC BLOX to land the cable and establish a new connectivity hub in the Sunshine State. We’ll also develop a terrestrial route linking Palm Coast to our South Carolina cloud region.

When it’s complete, Sol will be the only in-service fiber-optic cable between Florida and Europe. In Spain, we’ll partner with Telxius to provide the necessary infrastructure to land the Sol cable in Santander, further integrating the Google Cloud region in Madrid into our global network. 

“Florida is once again at the forefront of strengthening our nation’s digital infrastructure. The anchoring of the Sol subsea cable in our state is a testament to our state’s commitment to fostering an environment that is ripe for technological investment and innovation. We are proud to partner with companies like Google and DC BLOX on strategic infrastructure projects that will support the future of AI innovation and bring tangible benefits to our communities and the entire Sunshine State.” – J. Alex Kelly, Florida Secretary of Commerce

“This is a landmark moment for Palm Coast, Flagler Beach, and Flagler County, and it’s a clear signal that we are a community of the future, investing in our economic development and vitality. The Sol subsea cable is more than just infrastructure; it’s a gateway to unprecedented global connectivity that will attract further high-caliber industries that our residents deserve. We are not just putting Palm Coast and our community on the map; we are building a direct route to the world’s digital economy, ensuring a prosperous and dynamic future for our community.” – Vice Mayor Theresa Carli Pontieri, Palm Coast, Florida 

 “DC BLOX is proud and honored to expand the foundational digital infrastructure that is vital to Florida’s growing economy. Supporting Google’s Sol cable, along with the capacity for additional cables, the new Palm Coast Cable Landing Station campus enhances Florida’s position as a nexus for global communications.” – Chris Gatch, Chief Revenue Officer, DC BLOX 

“Digital connectivity is today a strategic factor for the economic and social development of any city. At Santander City Council, we are committed to facilitating this type of infrastructure because its implementation attracts investment, talent, and technology companies, in addition to generating employment and positioning us as a city prepared for remote work, advanced services, and the Smart City model. That’s why we appreciate the interest that Google has shown in our coast and for choosing Santander as a transatlantic connection point!” – City Council Mayor Gema Igual, Santander, Spain, 

“Telxius is delighted to support Google in bringing the Sol cable to Spain, a key milestone that will boost transatlantic connectivity with enhanced capacity, reliability, and resilience. Through our submarine landing services, we accelerate project implementation and help expand global networks for our customers.” – Mario Martín, CEO, Telxius

“The Government of the Azores welcomes this important investment in the region with great enthusiasm and congratulates Google on its visionary, pioneering, and strategic approach, which fully aligns with our own vision: that the Azores are an important and strategic hub for digital connectivity in Europe and the North Atlantic.” – Artur Lima, Vice President, Government of the Azores

“Promoting the expansion and modernization of submarine cable infrastructure, with a view to strengthening the country’s international connectivity and digital competitiveness, is an essential goal for the government to the development of our country. The project for Google’s new “Sol” cable, with 16 fiber optic cable pairs and the potential to provide redundancy to the “Nuvem” cable, also from that company, will increase resilience and respond to the growing demand for digital infrastructure in Portugal and Europe.”– Miguel Pinto Luz, Minister of Infrastructure and Housing, Portugal

“Bermuda welcomes Google’s continued investment with the announcement of a second cable ‘Sol,’ further establishing Bermuda as a digital hub in the Atlantic. Bermuda greatly values its positive relationship with Google. This second cable highlights the benefits of this Government’s strategy to develop Bermuda as a progressive world leading jurisdiction for technology companies.” – Alexa Lightbourne, Minister of Home Affairs, Bermuda

Sol will add capacity, increase reliability, and decrease latency for Google users and Google Cloud customers around the world. Alongside cable systems like Nuvem, Firmina, Equiano, and Grace Hopper, Sol further establishes key locations across the Atlantic as connectivity hubs, strengthening local economies and bringing AI’s benefits to people and businesses around the world.

There’s a buzz of excitement here at Tobacco Dock as we welcome our customers and partners to the Google Cloud Summit London. Together, we’re exploring the essential role Google Cloud is playing in driving AI innovation and adoption across the UK. Today is about shining a spotlight on our customers and partners, focusing on the real-world stories of how they are accelerating innovation, realising value, and reshaping entire industries with the powerful combination of AI agents and robust cloud infrastructure.

The opportunity ahead is immense. AI is poised to inject more than £400 billion into the UK economy by 2030, and this year’s summit reaffirms our long-term commitment to helping the UK realise its potential and shape its AI future.

Investing in Britain’s digital future

To lead in the AI era, a nation needs a strong digital foundation. That’s why we’re proud to announce a series of strategic commitments to the UK’s digital future.

First, we are launching a new partnership with the UK Government to help modernise public services. This collaboration marks a significant shift, exploring the use of advanced tech and AI to move beyond decades-old legacy contracts that can leave essential services vulnerable. It supports the UK Government’s “blueprint for a modern digital government” by aiming to streamline services, reduce costs, and lay the digital foundations for AI success.

Equally important, we’re investing in a world-class, AI-ready workforce. Today, we are announcing a landmark proposal to provide free cloud and AI skills training for up to 100,000 UK public sector workers. This initiative is designed to accelerate the digital transformation of public services and empower the workforce with the talent to lead in the AI era.

These commitments are backed by our continued investment in critical infrastructure. We are close to reaching a major milestone with our data center in Waltham Cross, which will be fully operational by the end of the year. Delivering on our promise, this facility will provide British businesses with the high-performance, low-latency cloud they need to compete globally.

From AI vision to real-world value

Agentic AI is emerging as a major force in enterprise technology, representing a foundational shift in how work gets done. We’re helping British businesses go beyond experimentation and embrace a new era of intelligent agents. With advanced models like Gemini on our Vertex AI platform, we’re giving businesses the toolkits to drive real results and realise value faster.

At the Summit, we’re showcasing how leading UK companies are already leveraging these capabilities to drive impact:

• Imperial War Museums is the first UK museum to use AI at scale to digitise and transcribe oral histories. Implemented by our partner Capgemini, the Gemini-engineered system will make more than 20,000 histories from 1945 to the early 2000s searchable and accessible.

• LUSH is taking advantage of Vertex AI and Cloud Storage to reduce packaging waste through AI-powered product recognition at checkout in its beauty shops.

• Morrisons’ new in-app Product Finder—built with BigQuery and Gemini—helps customers quickly locate products in its grocery stores. It was used 50,000 times a day during the most recent Easter period, proving its immense practical value.

• Starling Bank launched “Spending Intelligence,” an industry-leading AI tool powered by Gemini models. Customers can now understand their spending habits simply by asking natural language questions and getting instant analysis.

• Toolstation now offers customers highly accurate product search results online and in-store, powered by Vertex AI Search for Commerce, which is boosting customer satisfaction and sales.

Extending Our Impact Through a Trusted Ecosystem

Collaboration is fundamental to progress. The achievements we celebrate today wouldn’t be possible without our incredible network of partners. Our work with Capgemini is helping British-born companies like Additive Catchments, a majority non-profit-owned company, to deploy AI-powered water stewardship solutions. Leveraging Google Cloud, they provide real-time insights to improve water quality and safeguard ecosystems across the UK—a perfect example of technology for good. 

This kind of innovation isn’t happening in isolation. Since 2023, more than 60% of UK generative AI startups have chosen Google Cloud due to our deep commitment to the ecosystem. From our ‘Gemini for UK’ initiative, which offers up to £280,000 in cloud credits and access to expert mentorship and training, to our recent partnership with Tech London Advocates, our aim is to help founders scale faster. 

We’ve also secured spaces for top UK AI Startups on the Europe-wide Google for Startups accelerator. These companies, such as Martee ai, are on a mission to improve the face of grab-and-go food, for the benefit of people and the planet, and Dyad AI, which is helping healthcare providers improve care quality and automate administrative workflows. They will be joining 15 top AI startups from across Europe in receiving expert help from Google so they can take their startups to the next level.

Setting a Secure Foundation for a Bold Future

A big part of our mission is to be the most trusted cloud partner for businesses, exemplified through our leading security, data sovereignty, and responsible AI practices. At the summit, Lloyds Banking Group and Vodafone are sharing how they are using Google Cloud’s security tools to help protect their operations and innovate securely for millions of customers.

This deep commitment to security is why Google Cloud announced plans earlier this year to acquire Wiz. By integrating Wiz’s best-in-class, multi-cloud visibility with Mandiant’s unmatched threat intelligence, we will offer UK businesses proactive, comprehensive protection across their entire cloud estate.

For UK organizations, including the public sector, we’re also building trust in generative AI by ensuring data sovereignty and compliance, so they can use this technology while keeping their data completely secure. Today we are happy to share that Google Cloud is strengthening its UK data residency commitment, empowering organizations with the choice to keep Gemini 2.5 Flash processing entirely within the UK. This expansion opens up even more use cases for businesses operating here.

This combination of world-class security and a commitment to openness is the foundation upon which UK organizations can innovate with confidence. Our leadership in open-source technologies like Kubernetes enables them to build and run applications anywhere, knowing their data is protected by world-class security. Whether it’s a bank securing its services, a retailer reimagining the customer experience, or a startup building the next big thing, our goal is to provide the tools and the trust to build boldly. 

With that mission in mind, UK businesses have a real opportunity to lead on the global stage. Together, with our customers, partners, and the public sector, we’re building much more than a cloud platform; we’re helping UK organisations not only stay ahead but to move boldly into the future and shape the AI era.

Didn’t make it to London? Please browse our catalogue of upcoming events & sign up to join us at another exciting gathering of cloud and AI leaders from across industries and around the world.

Today, we’re making it even easier to achieve breakthrough performance for your AI/ML workloads: Google Cloud Managed Lustre is now GA, and available in four distinct performance tiers that deliver throughput ranging from 125 MB/s, 250 MB/s, 500 MB/s, to 1000 MB/s per TiB of capacity — with the ability to scale up to 8 PB of storage capacity. The Managed Lustre solution is powered by DDN’s EXAScaler, combining DDN’s decades of leadership in high-performance storage with Google Cloud’s expertise in cloud infrastructure.

Managed Lustre provides a POSIX-compliant, parallel file system that delivers consistently high throughput and low latency, essential for:

• High-throughput inference: For applications that require near-real-time inference on large datasets, Lustre provides high parallel throughput and sub-millisecond read latency.

• Large-scale model training: Accelerate the training cycles of deep learning models by providing rapid access to petabytes-sized datasets. Lustre’s parallel architecture ensures GPUs and TPUs are fed with data, minimizing idle time.

• Checkpointing and restarting large models: Save and restore the state of large models during training faster, improving goodput and allowing for more efficient experimentation.

• Data preprocessing and feature engineering: Process raw data, extract features, and prepare datasets for training, reducing the time spent on data pipelines.

• Scientific simulations and research: Beyond AI/ML, Lustre excels in traditional HPC scenarios like computational fluid dynamics, genomic sequencing, and climate modeling, where massive datasets and high-concurrency access are critical.

Lustre is designed for the highly parallel and random I/O that characterizes many AI/ML training and inference tasks. This parallel processing capability across multiple clients ensures your compute resources are never starved for data.

Performance tiers and pricing

Managed Lustre offers flexible pricing and performance tiers designed to meet the diverse needs of your workloads, whether you’re focused on capacity or highest throughput density. 

Please see more details at the Managed Lustre pricing page.

Irrespective of the aggregate throughput, all tiers come with sub-millisecond read latency, high single-stream throughput, and are perfect for parallel access to many small files.

Driving innovation together: partnering with DDN

Google Cloud’s Managed Lustre is powered by DDN’s EXAScaler, bringing together two industry leaders in high-performance computing and elastic cloud infrastructure. This partnership represents a joint commitment to simplifying the deployment and management of large-scale AI and HPC workloads in the cloud, thanks to:

• Trusted leaders: By combining DDN’s decades of expertise in high-performance Lustre with Google Cloud’s global infrastructure and AI ecosystem, we are delivering a foundational capability that removes storage bottlenecks and helps our customers solve their most complex challenges in AI and HPC.

• Fully managed and supported solution: Enjoy the benefits of a fully managed service from Google, with comprehensive support from both Google and DDN, for seamless operations and peace of mind.

• Global availability and ecosystem integration: Managed Lustre is now globally accessible in multiple Google Cloud regions and integrates with the broader Google Cloud ecosystem, including Google Kubernetes Engine (GKE) and TPUs.

These benefits caught the attention of one of our largest partners, NVIDIA, who is looking forward to having it as part of its NVIDIA AI platform. 

“Enterprises today demand AI infrastructure that combines accelerated computing with high-performance storage solutions to deliver uncompromising speed, seamless scalability and cost efficiency at scale. Google and DDN’s collaboration on Google Cloud Managed Lustre creates a better-together solution uniquely suited to meet these needs. By integrating DDN’s enterprise-grade data platforms and Google’s global cloud capabilities, organizations can readily access vast amounts of data and unlock the full potential of AI with the NVIDIA AI platform (or NVIDIA accelerated computing platform) on Google Cloud — reducing time-to-insight, maximizing GPU utilization, and lowering total cost of ownership.” – Dave Salvator, Director of Accelerated Computing Products, NVIDIA

Get started today!

Ready to supercharge your AI/ML and HPC workloads? Getting started with Managed Lustre is simple:

1. Navigate to Managed Lustre in the Google Cloud console.

2. Provision your Managed Lustre instance, choosing the performance tier and size that best fits your needs.

3. Connect your compute instances, GKE clusters to your new high-performance file system.

For detailed instructions and documentation, please visit the Managed Lustre documentation. And if needed, reach out to Google Cloud sales specialists.

Watch the Fireside Chat

Don’t miss the opportunity to learn more about the strategic partnership between Google Cloud and DDN, and the unique capabilities of Managed Lustre. Read the official DDN press release here.

Watch the fireside chat with Sameet Agarwal, VP/GM Storage and Sven Oehme, CTO of DDN, here.

Today, we are thrilled to announce the expansion of the Z3 Storage Optimized VM family with the general availability of nine new Z3 virtual machines that offer local SSD capacity ranging from 3 TiB to 18 TiB per VM, complementing existing Z3 VMs which offer 36TiB of Local SSD per VM. We are also very pleased to launch a Z3 bare metal instance, which includes up to 72 TiB of Local SSDs. Z3 VMs enable customers like Shopify, Tenderly and ScyllaDB to achieve impressive performance improvements for their high performance storage workloads by reducing the IO access latency by up to 35% compared to VM instances using previous-generation local SSDs. 

Z3 VMs are designed to run I/O-intensive workloads that require large local storage capacity and high storage performance, including SQL, NoSQL, and vector databases, data analytics, semantic data search and retrieval, and distributed file systems. The Z3 bare metal instance provides direct access to the physical server CPUs and is ideal for workloads that require low-level system access like private and hybrid cloud platforms, custom hypervisors, container platforms, or applications with specialized performance or licensing needs. 

Both Z3 VMs and the bare metal instance are based on Titanium SSDs, which offload local storage processing from CPU resources to deliver real-time data processing, low-latency, high-throughput storage performance and enhanced storage security. Z3 VMs with Titanium SSD offer up to 36 GiB/s of read throughput and up to 9M IOPS, increasing write storage performance by up to 25% compared to previous generation Local SSDs¹.

Based on the 4th Gen Intel Xeon scalable processor, Z3 VMs come with up to 176 vCPUs, 1,408 GiB of memory, and 36 TiB of local storage in 11 virtual machine shapes. The Z3 bare metal instance offers 192 vCPUs, 1,536 GiB of memory and 72 TiB of local storage. Z3 VMs and the bare metal instance deliver the connectivity and storage performance that enterprise workloads need, with up to 100 Gbps in standard bandwidth and up to 200 Gbps with Tier1 networking for high-traffic applications.

The expanded Z3 virtual machine portfolio lets you rightsize your infrastructure and scale your clusters to meet workloads requirements by providing larger total local SSD capacity and higher local SSD capacity per vCPU. Z3 offers two different VM types: the standardlssd VM types, which include five VM shapes that offer about 200 GiB of local SSD per vCPU. They are optimized for data analytics (OLAP), and SQL databases like MySQL and Postgres workloads. The highlssd VM types include six different VM shapes and the Z3 bare metal instance. They offer about 400 GiB of local SSD per vCPU and are optimized for distributed databases, data streaming, large parallel file systems and data search.

What our customers and partners are saying

“We are thrilled to announce Nutanix Cloud Clusters coming to Google Cloud at the end of CY25 as part of Nutanix’s commitment to delivering flexible, hybrid cloud solutions. Google Cloud’s Z3 instance types represent a perfect foundation for Nutanix to enable performance and resilience for enterprise applications. We’re excited about our partnership with Google Cloud in empowering our joint customers with greater choice and simplicity in their cloud journey.” – Saveen Pakala, Vice President of Product Management, Nutanix

“OP Labs contributes to the Optimism protocol, which enables orders of magnitude of improved performance and scalability for Ethereum. Z3 reduces p99 block insertion tail latencies by 30-50% for our most I/O-demanding blockchain nodes compared to N2. By migrating our solution to Z3, we will be able to scale our blockchain nodes to handle L2 state growth in a more performant and cost-effective way.” – Zach Howard Senior Staff Engineer, OP Labs

The launch of Google Cloud’s Z3 storage optimized instances with smaller VM shapes  represents a leap forward in performance for high-traffic NoSQL environments. In internal tests and customer projects, ScyllaDB has impressively leveraged the advantages of Z3 including extremely low latencies under high read and write loads, high IOPS capacity enabling the processing of massive amounts of data and excellent cost-performance ratio for large-scale production systems. We are very excited to offer Z3 family servers in ScyllaDB Cloud, including Bring Your Own Account (BYOA).” – Avi Kivity, Co-founder and CTO, ScyllaDB

“Shopify has found Z3s to be an excellent platform to build our most performance sensitive storage systems on. We experienced a critical need for both large data volumes while remaining sensitive to latency and throughput on the storage side. While Google has a lot of options, local SSD was really the best fit, and Z3s allowed us to achieve the best price/performance along with enhanced stability appropriate for a source of truth Storage workload. Right now we see these storage optimized VMs as our platform of choice for the future.” – Mattie Toia, VP Infrastructure, Shopify

“Tenderly is built to be your go-to for Web3 production and development, bringing all the necessary infrastructure into one place. This allows teams to operate with speed and confidence, making blockchain technology accessible. We’ve seen impressive results running blockchain workloads on Z3 instances, with a 40% improvement on read latency compared to N2 and N2D instances.” – Ilija Petrovic, SRE Lead, Tenderly

“The VAST AI Operating System gives organizations a unified platform to reason over all of their data – structured, unstructured, and streaming through a global namespace that spans cloud and on-prem environments – enabling intelligent agents and applications to operate with full context and real-time speed. ,For customers running on Google Cloud, Z3 VMs complement this vision by providing the ideal storage infrastructure to accelerate these workloads, ensuring AI pipelines run fast and scale effortlessly in the cloud.” – Renen Hallak, Founder & CEO, VAST Data

Z3 VMs are also the physical foundation of AlloyDB, our flagship PostgreSQL-compatible database service, delivering sophisticated multi-level caching. AlloyDB uses Z3’s expansive local SSDs as an ultra-fast cache, holding datasets up to 25x larger than can be stored in memory. Database queries can access these large, cached datasets at latencies that closely approach in-memory performance, particularly when factoring in overall end-to-end application response times. This is a significant advantage for very large databases, including real-time analytical workloads, as AlloyDB’s high-performance columnar engine operates entirely within this massive cache. AlloyDB on Z3 VMs will soon be available in preview, delivering up to 3x better performance than N-series VMs for transactional workloads, particularly for large datasets.

Enhanced maintenance experience

Z3 instances make it easier for you to plan ahead and schedule maintenance operations at a time of your choosing by providing notice from the system several days in advance of a required maintenance. The new Z3 VMs further enhance the maintenance experience by allowing you to live-migrate an instance during maintenance events for VMs with 18 TiB or less of local SSD storage. For Z3 VMs with 36 TiB of local SSD and for Z3 bare metal instances, you’ll also receive in-place upgrades that preserve your data through the planned maintenance events.

Support for Hyperdisk

Z3 VMs support Hyperdisk, Google Cloud’s workload-optimized block storage that lets you optimize the performance for each workload by independently tuning the storage performance and capacity for each instance.

Z3 VMs are compatible with Hyperdisk Balanced, Hyperdisk Throughput, and Extreme Hyperdisk storage for scalable, high-performance network-attached storage, supporting up to 512 TiB of capacity per instance. For general-purpose workloads, Hyperdisk Balanced, with up to 160K IOPS per instance, offers a mix of performance and cost-efficiency. Hyperdisk Extreme delivers ultra-low latency and supports up to 350K IOPS and 5,000 MiB/s throughput per Z3 VM instance and up 500K IOPS and 10,000 MiB/s throughput for the Z3 bare metal instance — making it well-suited for demanding workloads like databases. Using Hyperdisk for persistent storage and Z3 Local SSD for caching creates an optimal storage architecture for high end databases and mission critical workloads

Get started with Z3 today

Z3 VMs and bare metal instances are available today in most regions worldwide. To start using Z3 instances, select Z3 under the new Storage-Optimized machine family when creating a new VM or GKE node pool in the Google Cloud console. Learn more at the Z3 machine series page. Contact your Google Cloud sales representative for more information on regional availability.

^{1. Results are based on Google Cloud’s internal benchmarking}

Developers are racing to productize agents, but a common limitation is the absence of memory. Without memory, agents treat each interaction as the first, asking repetitive questions and failing to recall user preferences. This lack of contextual awareness makes it difficult for an agent to personalize their assistance–and leaves developers frustrated. 

How we normally mitigate memory problems: So far, a common approach to this problem has been to leverage the LLM’s context window. However, directly inserting entire session dialogues into an LLM’s context window is both expensive and computationally inefficient, leading to higher inference costs and slower response times. Also, as the amount of information fed into an LLM grows, especially with irrelevant or misleading details, the quality of the model’s output significantly declines, leading to issues like “lost in the middle” and “context rot”. 

How we can solve it now: Today, we’re excited to announce the public preview of Memory Bank, the newest managed service of the Vertex AI Agent Engine, to help you build highly personalized conversational agents to facilitate more natural, contextual, and continuous engagements. Memory Bank helps us address memory problems in four ways: 

• Personalize interactions: Go beyond generic scripts. Remember user preferences, key events, and past choices to tailor every response.

• Maintain continuity: Pick up conversations seamlessly where they left off across multiple sessions, even if days or weeks have passed.

• Provide better context: Arm your agent with the necessary background on a user, leading to more relevant, insightful, and helpful responses.

• Improve user experience: Eliminate the frustration of repeating information and create more natural, efficient, and engaging conversations.

Where you can access it: Memory Bank is integrated with the Agent Development Kit (ADK) and Agent Engine Sessions. You can define an agent using ADK, enable Agent Engine Sessions to store and manage conversation history within individual sessions. Now, you can enable Memory Bank to provide long-term memory for agents to store, retrieve, and manage relevant information across multiple sessions. You can also use Memory Bank to manage your memories with other agent frameworks including LangGraph and CrewAI.

Here’s how Memory Bank works

1. It understands and extracts memories from interactions: Using Gemini models, Memory Bank can analyze a user’s conversation history with the agent (stored in Agent Engine Sessions) to extract key facts, preferences, and context to generate new memories. This happens asynchronously in the background, without you needing to build complex extraction pipelines.

2. It stores and updates memories intelligently: Key information—like “My preferred temperature is 71 degrees,” or “I prefer aisle seats on flights” — is stored persistently and organized by your defined scope, such as user ID. When new information arises, Memory Bank (using Gemini) can consolidate it with existing memories, resolving contradictions and keeping the memories up to date.

3. It recalls relevant information: When a user starts a new conversation (session), the agent can retrieve these stored memories. This can be a simple retrieval of all facts or a more advanced similarity search (using embeddings) to find the memories most relevant to the current topic, ensuring the agent is always equipped with the right context.

This entire process is grounded in Google Research’s novel research method (accepted by ACL 2025), which enables an intelligent, topic-based approach to how agents learn and recall information, setting a new standard for agent memory performance.

Let’s take an example. Imagine you’re a retailer in the beauty industry. You have a personal beauty companion equipped with memory that recommends products and skincare routines.

As shown in the illustration, the agent is able to remember the user’s skin type (maintaining context) even after it evolves over time and be able to make personalized recommendations. This is the power of an agent with long-term memory.

Get started today with Memory Bank

You can integrate Memory Bank into your agent in two primary ways:

1. Develop an agent with Google Agent Development Kit (ADK) for an out-of-the-box experience

2. Develop an agent that orchestrates API calls to Memory Bank if you are building your agent with any other framework.

To get started, please refer to the official user guide and the developer blog. For hands-on examples, the Google Cloud Generative AI repository on GitHub offers a variety of sample notebooks, including integration with ADK and deployment to the Agent Engine runtime. For those wishing to try Memory Bank with third-party frameworks, we also provide notebook samples for LangGraph and CrewAI.

• Memory Bank with ADK agent

• Memory Bank with CrewAI agent

• Memory Bank with LangGraph agent

If you’re a developer using Agent Development Kit (ADK) but have never used Google Cloud before, you can still start by using our new express mode registration for Agent Engine Sessions and Memory Bank. Here’s how it works:

1. Sign up with your Gmail account to receive an API key

2. Use the key to access Agent Engine Sessions and Memory Bank

3. Build and test your agent within the free tier usage quotas

4. Seamlessly upgrade to a full Google Cloud project when you are ready for production

If you want to know more about Memory Bank, join the Vertex AI Google Cloud community to share your experiences, ask questions, and collaborate on new projects.

We’re thrilled to announce that Google has been named a Leader in the Gartner® Magic Quadrant™ for Search and Product Discovery.

We believe this recognition affirms Google’s evolving commitment to delivering AI solutions tuned to unique industry challenges, empowering businesses to transform the digital commerce experience and deliver high ROI-generating product discovery, through the use of AI in relevance, ranking, and personalization.

Built upon Google’s deep expertise & knowledge of the way users search, interact with & purchase products across a broad landscape of commerce domains, Vertex AI Search for commerce is a fully-managed, AI-first solution tailored for e-commerce. 

Redefining product discovery with generative AI

As digital channels continue to increase in traffic and adoption, businesses in e-commerce need to navigate the challenges involved in providing great digital experiences. Customers expect relevance, convenience, and personalization.

Vertex AI Search for commerce uses the best of Google AI to drive personalized product discovery at scale, optimized for revenue-per visitor. This enables businesses to not only understand user intent, but proactively surface the right products and content to shoppers through a variety of multimodal capabilities that leverage the latest advancements in generative AI. 

• Search and browse: Powering digital commerce sites and applications with Google-quality capabilities highly tuned for e-commerce and revenue maximization.

• Conversational search: Enabling real-time, back and forth conversation, powered by Gemini models to guide users through their shopping journeys, while optimizing for conversion.

• Recommendations: Deliver highly personalized recommendations at scale.

• Semantic image search: Users can search by image (“find this blouse”) or by a combination of text & image (“find the shoes that would look great with this blouse”).

In particular, our customers have noted our leadership in conversational search through one of our latest offerings, Conversational Commerce, which helps retailers provide assisted shopping on any digital channel, to engage with customers in a more natural and human-like conversation. This includes helping customers find their desired products online or helping a store associate answer questions using data from multiple sources to increase buying confidence for customers worldwide.

Intuitive search at the speed of intent, tuned for ecommerce

Search is changing as consumers are leveraging longer queries, seeking more comprehensive answers to questions.

Over the past decade, we’ve made significant strides in understanding user intent and refining our proprietary algorithms, now enhanced with gen AI. This has become increasingly important as user search behavior and the shopping journey have evolved, transforming the retail landscape. Unlike traditional models that are not AI-native, Vertex AI Search for commerce nearly eliminates the need for manual overriding to re-rank results and compensate for suboptimal search quality. Our solution is  developed to help businesses, from digitally native e-commerce companies to traditional retailers, adapt to this new era of buying and selling by leveraging AI to maximize revenue.

• Access to cutting edge AI/ML models: Vertex AI Search for commerce redefines e-commerce search using powerful AI / ML models like Gemini for unparalleled accuracy and relevance.

• Leverage Google’s proprietary intelligence: Vertex AI Search for commerce utilizes Google Shopping’s vast query & click datasets, along with our most advanced knowledge graphs, to train our industry-leading AI Relevance models.

• Advanced intent detection: Interprets complex queries and nuances in user intent beyond simple keyword matching, focusing on semantic meaning for intuitive results.

• AI-based catalog optimization:  Retail catalog search results are enhanced by combining Google Shopping’s effective web crawlers with Google’s extensive understanding of web content and our unique AI models.

• Deep and scalable personalization: Analyzes individual shopper behavior, preferences, and historical data to deliver tailored product recommendations and search rankings, boosting satisfaction, conversions, and loyalty.

• Managed deployment: Offers a fully managed solution for seamless integration and launch, minimizing engineering overhead.

• Flexible customization and control: Businesses can customize the search experience with specific business rules and optimization objectives, aligning with unique KPI goals and other unique metrics. 

The future of e-commerce is personalized and simplified

We believe being positioned as a Leader in the Gartner® Magic Quadrant™ for Search and Product Discovery underscores Google’s proven ability to deliver real business value. Vertex AI Search for commerce provides a comprehensive, AI-first solution that guides your customers through the buying journey, ensuring they find exactly what they need, every time. 

To download the full 2025 Gartner Magic Quadrant for Search and Product Discovery report, click here, and for more information on Vertex AI Search for commerce, see here or register for our upcoming product roadmap webinar. 

^{Gartner, Magic Quadrant for Search and Product Discovery – Mike Lowndes, Noam Dorros, Sandy Shen, Aditya Vasudevan, June 24, 2025}

^{Disclaimer: Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google.}

^{GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and MAGIC QUADRANT is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved.}

In the high-speed world of global motorsport, operational efficiency and technological innovation are as critical off the track as they are on it. And when it comes to innovating in the field, Formula E, with its focus on the future of mobility and sustainability, regularly takes the checkered flag.

Formula E orchestrates thrilling races with super-electric-power cars in cities worldwide. A central part of this experience is the management and distribution of immense volumes of media content. Over the course of a decade of high-octane racing, Formula E has accumulated a vast media archive, comprising billions of files and petabytes of data. 

This valuable collection of content, previously housed in AWS S3 storage, necessitated a more robust, high-performance, and globally accessible solution to fully harness its potential for remote production and content delivery. These needs became especially acute as AI offers more opportunities to leverage archives and historic data.

At the same time, the organization faced internal operational challenges, including common issues like disconnected systems, which limited collaboration, alongside escalating costs from multiple software subscriptions. Formula E sought a unified solution to foster greater integration and prepare for the future of work. 

This led to the widespread adoption of Google Cloud’s ecosystem, including Google Workspace, to address both its large-scale data storage needs and internal collaboration and productivity workflows.

A decade of data and disconnected operations

Formula E’s media archive represented a significant challenge due to its sheer scale. With 10 years of racing history captured, the archive contained billions of files and petabytes of data, making any migration a formidable task. 

The demands of its global operations, which include taking production to challenging race locations across the globe, require seamless connectivity and high throughput. The organization had previously encountered difficulties connecting to its S3 storage from remote geographical locations. These experiences prompted successful experiments with Google Cloud, underscoring the critical need for a cloud solution that could deliver consistent performance, even in areas with less reliable internet infrastructure.

Internally, Formula E’s use of disparate systems, including Office 365 and multiple communication tools, resulted in disjointed workflows and limited collaborative capabilities. The reliance on multiple software subscriptions for communication and collaboration was also driving up operational costs. The team needed a more integrated and cost-effective environment that could streamline processes and foster a more collaborative culture, setting the stage for future advancements in workplace technology.

A pitstop in the cloud drives better performance

To address these multifaceted challenges, Formula E embarked on a strategic, two-pronged migration. This involved moving its massive media archive to Google Cloud Storage and transitioning its entire staff to Google Workspace.

For the monumental task of migrating its petabyte-scale archive, Formula E collaborated closely with Google Cloud and Mimir, its media management system provider. Following meticulous planning with Mimir and technical support from Google Cloud, the team chose the Google Cloud Storage Transfer Service (STS). 

STS is a fully managed service engineered for rapid data transfer, which crucially required no effort from Formula E’s team during the migration itself. A pivotal element that ensured a smooth transition was Cloud Storage’s comprehensive support for S3 APIs. This compatibility enabled Formula E to switch cloud providers without any interruption to its business, guaranteeing continuity for its critical media operations.

To revolutionize its internal operations, Formula E partnered with Cobry, a Google Workspace migration specialist, for a two-phased migration process. The initial phase focused on migrating Google Drive content, which was soon followed by the full migration to Workspace. Cobry not only helped with the technical migration, the company also provided support for change management and training. 

To ensure a smooth transition and encourage strong user adoption, Google champions from across the business received in-person training at Google’s offices. The project successfully migrated 220 staff members along with 9 million emails and calendar invites.

The need for speed, collaboration, and savings

The migration of the media archive to Google Cloud proved as successful as a clever hairpin maneuver at the track. 

The Storage Transfer Service performed beyond expectations, moving data out of S3 at an impressive rate of 240 gigabits per second — which translates to approximately 100 terabytes per hour. The entire sprawling archive was transferred in less than a day, a level of throughput that impressed even Google Cloud’s internal teams and confirmed that STS could deliver results at major scale. Such efficiency meant that Formula E experienced no business interruption and maintained continuous access to its valuable media assets.

Beyond the rapid migration, Formula E now benefits significantly from Google Cloud’s global network. By leveraging this infrastructure, the organization enjoys lower latency and higher throughput, which are critical for its remote production studios when operating worldwide. 

A core technology behind this enhanced performance is Google Cloud’s “Cold Potato Routing.” This strategy keeps data on Google’s private, internal backbone network for as long as possible, using the public internet for only the shortest final leg of the journey. This approach guarantees improved throughput and latency, effectively resolving the connectivity challenges Formula E previously faced in remote race locations.

Google Cloud’s commitment to an open cloud ecosystem, demonstrated by its full support for S3 APIs, was instrumental in facilitating a smooth transition without vendor lock-in.

The transition to Google Workspace, meanwhile, has transformed Formula E’s internal operations, fostering a more integrated and collaborative work environment. Developed with collaboration at its core, Google Workspace has seamlessly integrated Gemini into the team’s workflow. Initially adopted by office-based teams for automating repetitive tasks, formatting, and summarizing data, Gemini is now accessible across the entire business, empowering staff to work more intelligently. 

“Moving from Office 365 to Google Workspace has been a great step for our team in productivity,” said Hayley Mann, chief people officer at Formula E. “The enhanced collaboration features have been really beneficial, and we’re excited about how Google’s integrated Al capabilities will empower and enable our people to work smarter and even more efficiently every day.”

The migration is also delivering financial benefits, as Formula E was also able to replace their usage of Slack and Zoom with Chat and Meet in Google Workspace. Further savings are anticipated as other existing software contracts expire.

Racing toward an open ecosystem

Looking to the future, Formula E is also positioned to realize significant cost savings by using Cloud Storage’s Autoclass feature, which intelligently manages storage classes based on access patterns to optimize costs. 

With its petabyte-scale media archive now residing on Google Cloud, Formula E is well positioned to continue innovating in media management, ensuring its content is high-performance, cost-effective, and globally accessible for years to come. This includes leveraging new AI capabilities emerging across the Google ecosystem.

And by embracing integrated AI through Gemini and fostering a truly collaborative environment with Google Workspace, Formula E is accelerating its journey towards peak operational efficiency — mirroring the innovation it champions on the racetrack.

In 2024, retail sales for consumer packaged goods were worth $7.5 trillion globally. Their sheer variety — from cosmetics to clothing, frozen vegetables to vitamins  — is hard to fathom. And distribution channels have multiplied: Think big box stores in the brick-and-mortar world and mega ecommerce sites online. Most importantly, jury-rigged digital tools can no longer keep pace with the ever-growing web of regulations designed to protect consumers and the environment. 

SmarterX uses AI to untangle that web. Our Smarter1 AI model aggregates and triangulates publicly available datapoints — hundreds of millions UPCs and SKUs, as well as product composition and safety information — from across the internet. By matching specific products to applicable regulatory information and continuously updating our models for a particular industry or client, SmarterX helps retailers make fully compliant decisions about selling, shipping, and disposing of regulated consumer packaged goods. 

And just like our clients, we needed to accelerate and expand our capabilities to keep pace with that data deluge and build better AI models faster. Migrating to Google Cloud and BigQuery gave us the power, speed, and agility to do so. 

Embracing BigQuery: a flexible, easy-to-use, AI-enabled data platform

Because we deal with data from so many sources, we needed a cloud-based enterprise data platform to handle multiple formats and schemas natively. That’s exactly what BigQuery gives us. Since data is the foundation of our company and products, we began by migrating all our data — including the data housed in Snowflake — to BigQuery. 

With other data platforms, the data has to be massaged before you can work with it: a time-consuming, often manual process. BigQuery is built to quickly ingest and aggregate data in many different formats, and its query engine allows us to work with data right away in whatever format it lands. Coupled with Looker, we can create easy-to-understand visualizations of the complex data in BigQuery without ever leaving Google Cloud. 

In addition, because Gemini Code Assist is integrated with BigQuery,  even our less-technical team members can do computational and analytical work.

An integrated tech stack unleashes productivity and creativity

After 10 years in business, SmarterX was also suffering from system sprawl. 

Just as migrating data between platforms is inefficient, developers become less efficient when they have to bounce around different tools. Even with the increase in AI agents to help with coding and development, the tools struggle, too: When hopping among multiple systems, they pick up noise along the way. And governing identity and access management (IAM) individually for all those systems was time-consuming and left us vulnerable to potential security risks caused by misapplied access privileges.

Google Cloud provides a fully integrated tech stack that consolidates our databases, data ingestion and processing pipelines, models, compute power — even our email, documents, and office apps — in a single, unified ecosystem. And its LLMs are integrated throughout that ecosystem, from the Chrome browser to the SQL variants themselves. This obviates building custom pipelines for most new data sets and allows us to work more efficiently and coherently: 

• We’re now releasing new products 10-times faster than we were prior to migrating to Google Cloud. 

• We onboard new customers in less than a week instead of six months. 

• Our data pipelines handle 100 times the data they did previously, so we can maintain hundreds of customer deployments with a relatively small staff.

Consolidating on Google Cloud also lowered our overhead by 50% because we deprecated several of other SaaS platforms and teams can easily engage with Google’s tools without specialized expertise. Our entire team now lives in Google Cloud: Not an hour goes by that we aren’t using some form of the platform’s technology. 

Eliminating system sprawl also means we no longer need to maintain security protocols for separate platforms. Permissioning and identity and access management are handled centrally, and Google Cloud makes it easy to stay current on compliance requirements like SOC-2. 

A vision for AI in tune with our own: Gemini

The value SmarterX provides our customers relies heavily on our platform’s AI-driven capabilities. Finding the right AI model development platform and AI models was therefore one of the driving forces behind our choice of a new data platform. And when it comes to creating AI models, philosophy matters.  

Google’s philosophy dovetails with our own because they’ve always been at the forefront of understanding how people want to access information. Since the company’s expertise makes web data searchable on an enterprise scale, its Gemini models are tuned beautifully to do what SmarterX needs them to. Before switching to Vertex AI and Gemini, it took us months to release a new model; we can now do the same work in a matter of weeks.

When SmarterX hires new team members, we look for creative thinkers, not speakers of a specific coding language. And we want to give our developers the brainspace to focus on complex problem-solving rather than puzzling over syntax for SQL coding. Gemini Code Assist in BigQuery is easy to learn and can accurately handle the syntax for them. That leaves our developers more time for finding innovative solutions. 

A smooth migration by a team that knows its stuff

We couldn’t have completed our migration without the support of the Google Technical Onboarding Center. They really know their way around their technology and had spelunking tools at the ready for tricky scenarios we encountered along the way. 

In less than a month, we migrated terabytes of data from Snowflake to BigQuery: more than 80 databases and thousands of tables from 21 different sources. We used a two-pronged approach that leveraged external tables for rapid access to data and native tables for optimized query performance. 

Prior to the migration, Google provided foundational training for managing and operating the Google Cloud Platform. They also took the time to understand SmarterX technology. So instead of being constrained by a cookie-cutter migration plan, the Google team helped us to design and schedule a migration — with minimal disruptions or downtime — in the way that made the most sense for SmarterX and our customers. Google’s expertise in best-practices for security and identity and access management further enhanced the security of our new cloud environment.  

Even though we’re not a huge customer pumping petabytes of data through Google Cloud daily, the team treated us as if we were on par with larger organizations. When you’re literally moving the foundation of your entire business, it feels good to know that Google has your back.

Snowflake felt like a traditional enterprise data warehouse grafted onto the cloud, completely uninfluenced by the AI revolution, with a database that forced us to work in a specific, predetermined way. With BigQuery, we have a real information production system: a computing cloud with a built-in SQL-friendly data platform, a wide-ranging toolset, embedded AI and model development, and a single user interface for developing products our own way.

Unlimited imagination, not roadmaps

Many people are surprised when I tell them that SmarterX doesn’t have roadmaps — we make bets. We’re wagering that companies want AI to solve whatever real-world use cases arise. Rather than telling you what to do, AI has the ability to understand things that were previously impossible to understand, and to help people express ideas that were previously impossible to express. 

SmarterX works with over 2,000 brands. Ultimately, what they’re purchasing is the speed at which we can help them solve their business challenges with artificial intelligence. In much the same way, Google Cloud is solving our own technology challenges, sometimes before we even know we have them, so we can deliver top-notch products to our customers. 

Instead of doing battle with a growing sprawl of outdated technology, BigQuery and the rest of the Google Cloud integrated toolset is allowing us to relentlessly reinvent ourselves. Not a week goes by when I don’t hear someone say, “Oh, wow, we can do that with Google Cloud too?” 

^{Company description
SmarterX helps retailers, manufacturers, and logistics companies minimize regulatory risk, maximize sales, and protect consumers and the environment by giving them AI-driven tools to safely and compliantly sell, ship, store, and dispose of their products. Its clients include global brands that are household names all across the world.}

The landscape of business intelligence is evolving rapidly, with users expecting greater self-service and natural language capabilities, powered by AI. Looker’s Conversational Analytics empowers everyone in your organization to access the wealth of information within your data. Select the data you wish to explore, ask questions in natural language, as you would a colleague, and quickly receive insightful answers and visualizations that are grounded in truth, thanks to Looker’s semantic layer. This intuitive approach lowers the technical barriers to data analysis and fosters a more data-driven culture across your teams.

How does this work? At its core, Conversational Analytics understands the intent behind your questions. Enhanced by Gemini models, the process involves interpreting your natural language query, generating the appropriate data retrieval logic, and presenting the results in an easy-to-understand format, often as a visualization. This process benefits from Looker’s semantic model, which simplifies complex data with predefined business metrics, so that Gemini’s AI is grounded in a reliable and consistent understanding of your data.

Prioritizing privacy in Gemini

The rise of powerful generative AI models like Gemini brings incredible opportunities for innovation and efficiency. But you need a responsible and secure approach to data. When users use AI tools, questions about data privacy and security are top of mind. How are prompts and data used? Are they stored? Are they used to train the model?

At Google Cloud, the privacy of your data is a fundamental priority when you use Gemini models, and we designed our data governance practices to give you control and peace of mind. Specifically:

Your prompts and outputs are safe

Google Cloud does not train models on your prompts or your company’s data. 

Conversational Analytics only uses your data to answer your business questions — making data queries, creating charts and summarizations, and providing answers. We store agent metadata, such as special instructions, to improve the quality of the agent’s answers, and so you can use the same agent in multiple chat sessions. We also store chat conversations so you can pick up where you left off. Both are protected via IAM and not shared with anyone outside your organization without permission.

Your data isn’t used to train our models

The data processing workflow in Conversational Analytics involves multiple steps:

1. The agent sees the user’s question, identifies the specific context needed to answer it, and uses tools to retrieve useful context like sample data and column descriptions.

2. Using business and data context and the user question, the agent generates and executes a query to retrieve the data. The data is returned, and the resulting data table is generated. 

3. Previously gathered information can then be used to create visualizations, text explanations, or suggested follow-up questions. Through this process, the system keeps track of the user’s question, the data samples, and the query results, to help formulate a final answer.

4. When the user asks a new question, such as a follow-up, the previous context of the conversation helps the agent understand the user’s new intent.

Enhancing trust and security in Conversational Analytics

To give you the confidence to rely on Conversational Analytics, we follow a comprehensive set of best practices within Google Cloud:

• Leverage Looker’s semantic layer: By grounding Conversational Analytics in Looker’s semantic model, we ensure that the AI model operates on a trusted and consistent understanding of your business metrics. This not only improves the accuracy of insights but also leverages Looker’s established governance framework.

• Secure data connectivity: Conversational Analytics connects to Google Cloud services like BigQuery, which have their own robust security measures and access controls. This helps ensure that your underlying data remains protected.

• Use data encryption: Data transmitted to Gemini for processing is encrypted in-transit, safeguarding it from unauthorized access. Agent metadata and conversation history are also encrypted.

• Continuously monitor and improve: Our teams continuously monitor the performance and security of Conversational Analytics and Gemini in Google Cloud.

Role-based access controls

In addition, Looker provides a robustrole-based access control (RBAC) framework that Conversational Analytics leverages to offer granular control over who can interact with specific data. When a Looker user initiates a chat with data, Looker Conversational Analytics respects their established Looker permissions. This means they can only converse with Looker Explores to which they already have access. For instance, while the user might have permission to view two Looker Explores, an administrator could restrict conversational capabilities to only one. As conversational agents become more prevalent, the user will only be able to use those agents to which they have been granted access. Agent creators also have the ability to configure the capabilities of Conversational Analytics agents, for example limiting the user to viewing charts while restricting advanced functionalities like forecasting.

Innovate with confidence

We designed Gemini to be a powerful partner for your business, helping you create, analyze, and automate with Google’s most capable AI models. We’re committed to providing you this capability without compromising your data’s security or privacy, or training on your prompts or data. By not storing your prompts, data, and model outputs or using them for training, you can leverage the full potential of generative AI with confidence, knowing your data remains under your control.

By following these security principles and leveraging Google Cloud’s robust infrastructure, Conversational Analytics offers a powerful, insightful experience that is also secure and trustworthy. By making data insights accessible to everyone securely, you can unlock new levels of innovation and productivity in your organization. Enable Conversational Analytics in Looker today, and start chatting with your data with confidence.

Written by: Jaysn Rye

Executive Summary

As adversaries grow faster, stealthier, and more destructive, traditional recovery strategies are increasingly insufficient. Mandiant’s M-Trends 2025 report reinforces this shift, highlighting that ransomware operators now routinely target not just production systems but also backups. This evolution demands that organizations re-evaluate their resilience posture. One approach gaining traction is the implementation of an isolated recovery environment (IRE)—a secure, logically separated environment built to enable reliable recovery even when an organization’s primary network has been compromised.

This blog post outlines why IREs matter, how they differ from conventional disaster recovery strategies, and what practical steps enterprises can take to implement them effectively.

The Backup Blind Spot

Most organizations assume that regular backups equal resilience; however, that assumption doesn’t hold up against today’s threat landscape. Ransomware actors and state-sponsored adversaries are increasingly targeting backup infrastructure directly, encrypting, deleting, or corrupting it to prevent recovery and increase leverage.

The M-Trends 2025 report reveals that in nearly half of ransomware intrusions, adversaries used legitimate remote management tools to disable security controls and gain persistence. In these scenarios, the compromise often extends to backup systems, especially those accessible from the main domain.

In short: your backup isn’t safe if it’s reachable from your production network. During an active incident, that makes it irrelevant.

What Is an Isolated Recovery Environment?

An isolated recovery environment (IRE) is a secure enclave designed to store immutable copies of backups and provide a secure space to validate restored workloads and rebuild in parallel while incident responders carry out the forensic investigation. Unlike traditional disaster recovery solutions, which often rely on replication between live environments, an IRE is logically and physically separated from production.

At its core, an IRE is about assuming a breach has occurred and planning for the moment when your primary environment is lost, ensuring you have a clean fallback that hasn’t been touched by the adversary.

Key Characteristics of an IRE

• Separation of infrastructure and access: The IRE must be isolated from the corporate environment. No shared authentication, no shared tooling, no shared infrastructure or services, no persistent network links or direct TCP/IP connections between the production environment and the IRE.
• Restricted administrative workflows: Day-to-day access is disallowed. Only break-glass, documented processes exist for access during validation or recovery.
• Known-good, validated artifacts: Data entering the IRE must be scanned, verified, and stored with cryptographic integrity checks all while maintaining the isolation controls.
• Validation environment and tools: The IRE must also include a secured network environment, which can be used by security teams to validate restored workloads and remove any identified attacker remnants.
• Recovery-ready templates: Rather than restoring single machines, the IRE should support the rapid rebuild of critical systems in isolation with predefined procedures.

Implementation Strategy

Successfully implementing an IRE is not a checkbox exercise. It requires coordination between security, infrastructure, identity management, and business continuity teams. The following breaks down the major building blocks and considerations.

Infrastructure Segmentation and Physical Isolation

The foundational principle behind an IRE is separation. The environment must not share any critical infrastructure, identity, network, hypervisors, storage, or other services with the production environment. In most cases, this means:

• Dedicated platforms (on-premises or cloud based) and tightly controlled virtualization platforms

• No routable paths from production to the IRE network

• Physical air-gaps or highly restricted one-way replication mechanisms

• Independent DNS, DHCP, and identity services

Figure 2 illustrates the permitted flows into and within the IRE.

Identity and Access Control

Identity is the primary attack vector in many intrusions. An IRE must establish its own identity plane:

• No trust relationships to production Active Directory

• No shared local or domain accounts

• All administrative access must require phishing resistant multi-factor authentication (MFA)

• All administrative access should be via hardened Privileged Access Workstations (PAW) from within the IRE

• Where possible, implement just-in-time (JIT) access with full audit logging

Accounts used to manage the IRE should not have any ties to the production environment; this includes being used from a device belonging to the production domain. These accounts must be used from a dedicated PAW.

Secure Administration Flows

Administrative access is often the weak link that attackers exploit. That’s why an IRE must be designed with tight control over how it’s managed, especially during a crisis.

In the following model, all administrative access is performed from a dedicated PAW. This workstation sits inside an isolated management zone and is the only system permitted to access the IRE’s core components.

Here’s how it works:

• No production systems, including IT admin workstations, are allowed to directly reach the IRE. These paths are completely blocked.

• The PAW manages the IRE’s:

• Isolated Data Vault, where validated backups are stored.

• Management Plane, which includes IRE services such as Active Directory, DNS, PAM, backup, and recovery systems.

• Green VLAN, which hosts rebuilt Tier-0 and Tier-1 infrastructure.

This design ensures that even during a compromise of the production environment, attackers can’t pivot into the recovery environment. All privileged actions are audited, isolated, and console-restricted, giving defenders a clean space to rebuild from.

One-Way Replication and Immutable Storage

How data enters the IRE is just as important as how it’s managed. Backups that are copied into the data transfer zone must be treated as potentially hostile until proven otherwise.

To mitigate risk:

• Data must flow in only one direction, from production to IRE, never the other way around.*

• This is typically achieved using data diodes or time-gated software replication that enforces unidirectional movement and session expiry.

• Ingested data lands in a staging zone where it undergoes:

• Hash verification against expected values.

• Malware scanning, using both signature and behavioural analysis.

• Cross-checks against known-good backup baselines (e.g., file structure, size, time delta).

Once validated, data is committed to immutable storage, often in the form of Write Once, Read Many (WORM) volumes or cloud object storage with compliance-mode object locking. Keys for encryption and retention are not shared with production and must be managed via an isolated KMS or HSM.

The goal is to ensure that even if an attacker compromises your primary backup system, they cannot alter or delete what’s been stored in the IRE.

*Depending on overall recovery strategies, it’s possible that restored workloads may need to move from the IRE back to a rebuilt production environment. 

Recovery Workflows and Drills

An IRE is only useful if it enables recovery under pressure. That means planning and testing full restoration of core services. Effective IRE implementations include:

• Templates for rebuilding domain controllers, authentication services, and core applications

• Automated provisioning of VMs or containers within the IRE

• Access to disaster recovery runbooks that can be followed by incident responders

• Scheduled tabletop and full-scale recovery exercises (e.g., quarterly or bi-annually)

Many organizations discover during their first exercise that their documentation is out of date or their backups are incomplete. Recovery drills allow these issues to surface before a real incident forces them into view.

Hash Chaining and Log Integrity

If you’re relying on the IRE for forensic investigation as well as recovery, it’s essential to ensure the integrity of system logs and metadata. This is where hash chaining becomes important.

• Implement hash chaining on logs stored in the IRE to detect tampering.

• Apply digital signatures from trusted, offline keys.

• Regularly verify the chain against trusted checkpoints.

This ensures that during an incident, you can prove not only what happened but also that your evidence hasn’t been modified, either by an attacker or by accident.

Choosing the Right IRE Deployment Model

The right model depends on your environment, compliance obligations, and team maturity.

Common Pitfalls

• Over-engineering for normal operations: The IRE is not a sandbox. Avoid mission creep.

• Using the IRE beyond cyber recovery: The IRE is not for DR testing, HA, or daily operations. Any non-incident use risks breaking its isolation and trust model.

• Assuming cloud equals isolation: Isolation requires deliberate configuration. Cloud tenancy is not enough.

• Neglecting insider threats: The IRE must defend against sabotage from inside the organization, not just ransomware.

Closing Thoughts

As attackers accelerate and the blast radius of intrusions expands, the need for trusted, tamper-proof recovery options becomes clear. An isolated recovery environment is not just a backup strategy, it is a resilience strategy.

It assumes breach. It accepts that visibility may be lost during a crisis. And it gives defenders a place to regroup, investigate, and rebuild.

The Mandiant M-Trends 2025 report makes it clear; the cost of ransomware isn’t just in ransom paid, but in days or weeks of downtime, regulatory penalties, and reputation loss. The cost of building an IRE is less than a breach, and the peace of mind it offers is far greater.

For deeper technical guidance on building secure recovery workflows or assessing your current recovery posture, Mandiant Consulting offers strategic workshops and assessment services.

Acknowledgment

A special thanks to Glenn Staniforth for their contributions.

For organizations with stringent sovereignty and regulatory requirements, Google Distributed Cloud (GDC) air-gapped delivers a fully-managed experience with critical advanced networking capabilities. But operating in a completely isolated environment presents some unique networking challenges. Routine tasks become significantly more complex and manual, demanding more planning and bespoke solutions than on a connected network.

Today, we’re helping to solve these challenges with three major advancements in networking for GDC air-gapped: native IP address management (IPAM), multi-zone load balancing, and workload-level firewall policies — all powerful new capabilities designed to give you more control over your air-gapped environment. 

Let’s take a look at these new capabilities.

Streamlined IP management for GDC 

With GDC IP address management, you can now plan, track, and monitor IP addresses for all your workloads and infrastructure. IPAM for GDC is a valuable tool, since many air-gapped deployments consume IP addresses from your organization’s existing private IP address space, which can be difficult to manage, not very scalable, lacking in security, and finite. IPAM for GDC provides the following capabilities:

• Automated and streamlined IP management: Minimize manual errors and speed up deployments with capabilities that include Per-Org BYO-External IP and Internal-only VPC subnets.
• Scalable IP management: Expand your network for Day-2 IP growth, free from duplicate IP address conflicts, and with support for non-contiguous subnets.
• Enhanced security and compliance: Strengthen your posture and meet strict compliance requirements with robust IPAM controls, including subnet delegation and private IPs for zonal infrastructure.
• Optimized IP resource utilization: Reduce IP sprawl and maximize your finite IP resources.

IPAM for GDC provides the intelligent automation and centralized oversight essential for managing your complete IP lifecycle in secure, air-gapped environments, helping to ensure both operational excellence and adherence to critical regulations.

High availability with multi-zone load balancers

For critical applications, downtime is not an option. Now, you can help your workloads remain resilient and accessible, even in the event of a zone failure.

Our new multi-zone load balancing capability allows you to distribute traffic across multiple availability zones within your GDC environment. Both internal and external load balancers now support this multi-zone functionality, simplifying operations while maximizing uptime. This provides:

• Continuous availability: Applications remain accessible even during a complete zone failure.

• Operational simplification: There’s a single Anycast IP address for the application (regardless of where backends are located).

• Optimized performance: Traffic is routed to the nearest available instance based on network topology and routing metrics.

The load balancing system operates by creating load balancer (LB) objects, which are then handled by new LB API controllers. These controllers manage object conditions, including cross-references and virtual IP address (VIP) auto-reservations, and create Kubernetes services across all clusters.

Workload-level network firewall policies

To secure an environment, you need to control traffic not just at the edge, but between every component inside. That’s why we’re launching workload-level firewall policies as part of the GDC air-gapped product. This feature provides fine-grained control over communication between individual workloads, such as VMs and pods, within a project. This feature helps:

• Strengthen your security posture: Isolate workloads and limit communication between them.

• Easily apply policies: Define and apply policies to specific workloads or groups of workloads.

• Meet regulatory standards: Help adhere to regulatory requirements and internal standards.

GDC air-gapped implements default base network policies to create a secure architecture. In order to allow intra-project or cross-project traffic at the workload level, you can update these default policies as you wish. Policies are multi-zone by default. This means they affect all zones where your labeled workloads are present. You can enforce policies at the workload level using labels and workload selectors.

A new era of network control

These new capabilities — GDC IPAM, multi-zone load balancing, and workload-level firewall policies — represent a significant step forward in providing a robust, resilient, and secure networking experience for the air-gapped cloud. They work together to simplify your operations, strengthen your security posture, and empower you to run your most sensitive applications with confidence.

To learn more about these features, please refer to our documentation or contact your Google Cloud account team.

Editor’s Note: Today, we’re sharing insights from IDC Research Director, Devin Pratt, as he offers his analysis of recent research on Cloud SQL. In this post, you’ll see how Cloud SQL’s highly flexible, fully managed database service for MySQL, PostgreSQL, and SQL Server workloads can boost performance and cut costs, ultimately freeing your team to focus on core tasks. If you’re interested in exploring Google Cloud’s full range of database services, you can find more at our Google Cloud Databases homepage.

In today’s data-driven landscape, effectively managing databases requires solutions that tackle performance, scalability, and integration challenges. With years of experience analyzing database management systems (DBMS), I have witnessed the industry’s evolution in response to increasing demands for efficiency and innovation. This transformation is notably highlighted in IDC’s recent comprehensive Business Value White Paper, The Business Value of Cloud SQL: Google Cloud’s Relational Database Service for MySQL, PostgreSQL, and SQL Server.

The study examines the experiences of organizations that transitioned from self-managed database servers in their data centers or cloud environments to Cloud SQL. Through my analysis of the database market, I have observed how these transitions can significantly enhance an organization’s operational efficiency and reshape its cost structure. The findings align with my observations, revealing benefits such as reduced operational costs and access to advanced automation and expertise.

These results underscore the evolving nature of the database market and present valuable opportunities for businesses to optimize their operations through the strategic adoption of cloud solutions.

The challenges of modern database management

As a professional in database management, I’ve observed several key challenges facing organizations today:

• Performance demands: Applications require faster read/write speeds to maintain responsiveness under heavy workloads.

• Downtime issues: Maintenance tasks often disrupt operations, leading to costly interruptions.

• Scaling limitations: Technical constraints can hinder database growth and adaptability.

• Complex disaster recovery: Executing reliable recovery processes remains difficult and risky.

• AI integration complexity: Incorporating AI typically requires external tools, adding layers of intricacy.

• Resource-intensive management: A DBMS requires expertise and significant investment in maintenance, upgrades, and system resources, often straining IT budgets.

Addressing these issues is crucial for innovation and cost-efficiency in our increasingly data-driven world.

IDC’s Business Value White Paper found that organizations using Cloud SQL have achieved an impressive average three-year ROI of 246%, with a rapid 11-month payback period. Study participants attributed this high return to several factors, including:

• Increased operational efficiency: Database administrators and infrastructure teams can focus on strategic and innovative tasks rather than routine maintenance.

• Cost reduction: Organizations benefit from lower overall database operational costs, including reduced infrastructure and database expenses.

• Enhanced agility: Faster deployment and scaling of database resources enable businesses to better support development activities and adapt to changing needs.

• Business growth: Organizations are winning more business by delivering faster, higher-quality products and services, improving application performance, and user experiences.

Further advancements in database management

Since the publication of the IDC study, Google Cloud has enhanced Cloud SQL in two key areas: price performance and generative AI capabilities.

First, Enterprise Plus edition now provides businesses with an available and reliable database solution in addition to the core service. This includes increased read throughput and improved write latency, enhanced scalability with tenfold expanded table support, greater efficiency through near-zero planned downtime with rolling updates for both scaling up and down, and improved disaster recovery capabilities via enhanced failover processes and testing.

Second, Cloud SQL provides a comprehensive set of generative AI tools and capabilities. This includes pgvector support in PostgreSQL and native vector support in MySQL for efficient vector similarity search, alongside streamlined connectivity to Vertex AI, LangChain, and various foundation models through extensions. This enables direct AI application development within the database.

Conclusion

The IDC Business Value White Paper on Cloud SQL provides data that aligns with many of my observations regarding cloud-based database solutions, highlighting several key areas of improvement:

• 44% increase in DBA efficiency

• 28% lower three-year cost of operations

• 96% faster creation and deployment of new databases

• An average annual revenue increase of $21.75 million per organization

These results suggest that managed database services like Cloud SQL may offer significant benefits in operational efficiency, cost reduction, and potential revenue growth.

For those interested in a more comprehensive analysis of these findings and their potential implications, I recommend reviewing the full IDC Business Value White Paper, “The Business Value of Cloud SQL: Google Cloud’s Relational Database Service for MySQL, PostgreSQL, and SQL Server,” sponsored by Google Cloud.