Cloud

The age of AI is here, and organizations everywhere are racing to deploy powerful models to drive innovation, enhance products, and create entirely new user experiences. But moving from a trained model in a lab to a scalable, cost-effective, and production-grade inference service is a significant engineering challenge. It requires deep expertise in infrastructure, networking, security, and all of the Ops (MLOps, LLMOps, DevOps, etc.).

Today, we’re making it dramatically simpler. We’re excited to announce the GKE inference reference architecture: a comprehensive, production-ready blueprint for deploying your inference workloads on Google Kubernetes Engine (GKE).

This isn’t just another guide; it’s an actionable, automated, and opinionated framework designed to give you the best of GKE for inference, right out of the box.

Start with a strong foundation: The GKE base platform

Before you can run, you need a solid place to stand. This reference architecture is built on the GKE base platform. Think of this as the core, foundational layer that provides a streamlined and secure setup for any accelerated workload on GKE.

Built on infrastructure-as-code (IaC) principles using Terraform, the base platform establishes a robust foundation with the following:

• Automated, repeatable deployments: Define your entire infrastructure as code for consistency and version control.

• Built-in scalability and high availability: Get a configuration that inherently supports autoscaling and is resilient to failures.

• Security best practices: Implement critical security measures like private clusters, Shielded GKE Nodes, and secure artifact management from the start.

• Integrated observability: Seamlessly connect to Google Cloud Observability for deep visibility into your infrastructure and applications.

Starting with this standardized base ensures you’re building on a secure, scalable, and manageable footing, accelerating your path to production.

Why the inference-optimized platform?

The base platform provides the foundation, and the GKE inference reference architecture is the specialized, high-performance engine that’s built on top of it. It’s an extension that’s tailored specifically to solve the unique challenges of serving machine learning models.

Here’s why you should start with our accelerated platform for your AI inference workloads:

1. Optimized for performance and cost

Inference is a balancing act between latency, throughput, and cost. This architecture is fine-tuned to master that balance.

• Intelligent accelerator use: It streamlines the use of GPUs and TPUs, so you can use custom compute classes to ensure that your pods land on the exact hardware they need. With node auto-provisioning (NAP), the cluster automatically provisions the right resources, when you need them.

• Smarter scaling: Go beyond basic CPU and memory scaling. We integrate a custom metrics adapter that allows the Horizontal Pod Autoscaler (HPA) to scale your models. Scaling is based on real-world inference metrics like queries per second (QPS) or latency, ensuring you only pay for what you use.

• Faster model loading: Large models mean large container images. We leverage the Container File System API and Image streaming in GKE along with Cloud Storage FUSE to dramatically reduce pod startup times. Your containers can start while the model data streams in the background, minimizing cold-start latency.

2. Built to scale any inference pattern

Whether you’re doing real-time fraud detection, batch processing analytics, or serving a massive frontier model, this architecture is designed to handle it. It provides a framework for the following:

• Real-time (online) inference: Prioritizes low-latency responses for interactive applications.

• Batch (offline) inference: Efficiently processes large volumes of data for non-time-sensitive tasks.

• Streaming inference: Continuously processes data as it arrives from sources like Pub/Sub.

The architecture leverages GKE features like the cluster autoscaler and the Gateway API for advanced, flexible, and powerful traffic management that can handle massive request volumes gracefully.

3. Simplified operations for complex models

We’ve baked in features to abstract away the complexity of serving modern AI models, especially LLMs. The architecture includes guidance and integrations for advanced model optimization techniques such as quantization (INT8/INT4), tensor and pipeline parallelism, and KV Cache optimizations like Paged and Flash Attention.

Furthermore, with GKE in Autopilot mode, you can offload node management entirely to Google, so you can focus on your models, not your infrastructure.

Get started today!

Ready to build your inference platform on GKE? The GKE inference reference architecture is available today in the Google Cloud Accelerated Platforms GitHub repository. The repository contains everything that you need to get started, including the Terraform code, documentation, and example use cases.

We’ve included examples for deploying popular workloads like ComfyUI and a general-purpose online inference with GPUs and TPUs to help you get started quickly.

By combining the rock-solid foundation of the GKE base platform with the performance and operational enhancements of the inference reference architecture, you can deploy your AI workloads with confidence, speed, and efficiency. Stop reinventing the wheel and start building the future on GKE.

The future of AI on GKE

The GKE inference reference architecture is more than just a collection of tools, it’s a reflection of Google’s commitment to making GKE the best platform for running your inference workloads. By providing a clear, opinionated, and extensible architecture, we are empowering you to accelerate your AI journey and bring your innovative ideas to life.

We’re excited to see what you’ll build with the GKE inference reference architecture. Your feedback is welcome! Please share your thoughts in the GitHub repository.

Full view of data

Short query optimizations process the entire query as a single stage, giving the runtime complete visibility into all tables involved. This allows the runtime to read both sides of the join and gather precise metadata, including column cardinalities. For instance, join columns in the query below have a low cardinality, so it is efficiently stored using dictionary and run-length encodings (RLE). Consequently, the runtime devises a much simpler query execution plan by leveraging encoding during execution.

The following query calculates top-ranked products and their performance metrics for an e-commerce scenario. It’s based on a type of query observed in Google’s internal data pipelines that benefited from short query optimizations. The following query uses this BigQuery public dataset, allowing you to replicate the results. Metrics throughout this blog were captured during internal Google testing.

By skipping the shuffle layer, overall query execution requires less CPU, memory, and network bandwidth. In addition to that, short query optimizations take full advantage of enhanced vectorization described in the Understanding BigQuery enhanced vectorization blog post.

Queries with joins and aggregations

In data analytics, it’s common to join data from several tables and then calculate aggregate results. Typically, a query performing this distributed operation will go through many stages. Each stage can involve shuffling data around, which adds overhead and slows things down. BigQuery’s short query optimizations can dramatically improve this process. When enabled, BigQuery intelligently recognizes if the amount of data being queried is small enough to be handled by a much simpler plan. This optimization leads to substantial improvements: for the query described in Figure 3, during internal Google testing we observed 2x to 8x faster execution times and an average of 9x reduction in slot-seconds.

We can see how the execution graph changes when short query optimizations are applied to the query in Figure 3.

Optimal impact

BigQuery’s short query optimizations feature is designed primarily for BI queries intended for human-readable output. Even though BigQuery utilizes a dynamic algorithm to determine eligible queries, it works with other performance-enhancing features like history-based optimizations, BI Engine, optional job creation, etc. Nevertheless, some workload patterns will benefit from short query optimizations more than others. 

This optimization may not significantly improve queries that read or produce a lot of data. To optimize for short query performance, it is crucial to keep the query working set and result size small through pre-aggregation and filtering. Implementing partitioning and clustering strategies appropriate to your workload can also significantly reduce the amount of data processed, and utilizing the optional job creation mode is beneficial for short-lived queries that can be easily retried.

Short query optimizations in action

Let’s see how these optimizations actually impact our test  query by looking closer at the query in Figure 1 and its query plan in Figure 2. The query shape is based on actual workloads observed in production and made to work against a BigQuery public dataset, so you can test it for yourself.

Despite the query scanning only 6.5 MB, running this query without advanced runtime takes over 1 second and consumes about 20 slot-seconds (execution time may vary depending on available resources in the project).

With BigQuery’s enhanced vectorization in the advanced runtime, during internal Google testing this query finishes in 0.5 seconds while consuming 50x less resources.

The magnitude of improvement here is less common, showing an example of real workload improvement from Google internal pipelines. We have also seen classic BI queries with several aggregations, filters, group by and sort, or snowflake joins achieve faster performance and better slot utilization. 

Try it for yourself

Short query optimizations boost query price/performance, allowing for higher throughput and lower latencies for common BI small queries. It achieves this by combining cutting-edge algorithms with Google’s latest innovations across storage, compute, and networking. This is just one of many performance improvements that we’re continually delivering to BigQuery customers like enhanced vectorization, history based optimizations, optional-job-creation mode, column metadata index (CMETA) and others.

Now that both key pillars of advanced runtime are in public preview, all you have to do to test it with your workloads is to enable it using the single ALTER PROJECT command as documented here. This enables both enhanced vectorization and short query optimizations. If you already did that earlier for enhanced vectorization, your project is automatically also enabled for short query optimizations.

Try it now with your own workload following steps in BigQuery advanced runtime documentation here, and share your feedback and experience with us at bqarfeedback@google.com.

In a world where data is your most valuable asset, protecting it isn’t just a nice-to-have — it’s a necessity. That’s why we are thrilled to announce a significant leap forward in protecting the data in your Cloud SQL instances, with Enhanced Backups for Cloud SQL.

This powerful new capability integrates Google Cloud Backup and DR Service directly into Cloud SQL, providing a robust, centralized, and secure solution to help ensure business continuity for your database workloads. The Backup and DR Service already protects Compute Engine VMs, Persistent Disks, and Hyperdisk, extending its ability to protect all of your workloads.   

Modern defense for modern threats

Enhanced Backups for Cloud SQL provides advanced protection by storing database backups in logically air-gapped and immutable backup vaults. Managed by Google and completely separate from your source project, these vaults provide a critical defense against threats that could compromise your entire environment.

For customers like JFrog, Cloud SQL Enhanced Backup with Google Cloud Backup and DR is proving to be a superior and robust alternative:

“Using this integration will help us significantly bolster our security posture by offering logically air-gapped and immutable backup vaults, creating a vital defense layer against diverse data-loss scenarios.” – Shiran Melamed, DevOps Group Leader, JFrog

Control, compliance, and peace of mind

We designed Enhanced Backups to be both powerful and easy to use, giving you fine-grained control over your data protection strategy. These capabilities are now available in Preview for both Cloud SQL Enterprise and Enterprise Plus editions, and offer key features to help ensure your data is always secure and recoverable:

• Immutable, air-gapped vaults: Protect your data with immutable backups stored in a secure, logically air-gapped vault. Setting minimum enforced retention and retention locks ensure backups cannot be deleted or changed for a predefined period, while a zero-trust access policy provides granular control.

• Business continuity: Your data is safeguarded against both source-instance and source-project deletion, so you can recover your data even if the source project itself becomes unavailable.

• Flexible policies that fit your needs: Your business isn’t one-size-fits-all, and your backup strategy shouldn’t be either. We offer highly customizable backup schedules, including hourly, daily, weekly, monthly, and yearly options. You can store backups for periods ranging from days to decades.

• Centralized command and control: Manage everything from a single, unified dashboard in the Google Cloud console. Monitor job status, identify unprotected resources, and generate reports, all in one place.

But you don’t have to take our word for it. See how customers like SQUARE ENIX and Rotoplas are already benefiting from Enhanced Backups for Cloud SQL:

“At SQUARE ENIX, protecting our users’ data is paramount. Google Cloud SQL’s Enhanced Backup integrated with the Backup and DR service is essential to our resiliency strategy. Its robust protection against instance- and even project-level deletion, combined with a secure, isolated vault and long-term retention, provides a critical safeguard for our most valuable asset. This capability will give us confidence in our data’s integrity and recoverability, allowing our teams to focus on creating the unforgettable experiences our users expect.” – Kazutaka Iga, SRE,SQUARE ENIX 

“Google Cloud SQL’s Enhanced Backup feature along with Google Professional Services support is a value add to our backup strategy at Rotoplas. The ability to centralize management, flexibly schedule backups, and store them independent of the source project gives us unprecedented control. This streamlined approach simplifies our operations and enhances security, ensuring our data is always protected and easily recoverable.” – Agustín Chávez Cabrera, Devops manager, Rotoplas

Get started with Enhanced Backups

Getting started with Enhanced Backups is simple. Here’s how you can enable this enhanced protection for your Cloud SQL instances:

1. Create or select a backup vault: In the Backup and DR service, either create a new backup vault or use an existing one.

For years, organizations have struggled with the workload conflict between online transaction processing (OLTP) and analytical query processing. OLTP systems such as Spanner are optimized for high-volume, low-latency transactions, and use row-oriented storage that’s efficient for individual record access. Analytical workloads, conversely, require rapid aggregations and scans across large datasets. These tasks are traditionally handled by separate data warehouses that employ columnar storage and incoming data pipelines from transaction systems. Separating OLTP and analytical workflows requires periodic data transfers, which often leads to stale data, complex ETL pipelines, and operational overhead.

Today, we’re thrilled to announce Spanner columnar engine, which brings new analytical capabilities directly to Spanner databases. Just as AlloyDB’s columnar engine enhanced PostgreSQL analytics, Spanner’s new columnar engine lets you analyze vast amounts of operational data in real-time, all while maintaining Spanner’s global consistency, high availability, and strong transactional guarantees — and without impacting transactional workloads.

The power of Spanner columnar engine helps organizations, such as Verisoul.ai, eliminate the problem of data silos typically found when combining high volume transaction systems with fast analytics. “Detecting fraud in real time is only half the story—showing customers the ‘why’ helps them act faster and turn trust into measurable ROI,” says Raine Scott and Niel Ketkar, founders of Verisoul.ai, a machine-learning platform that stops fake users and fraud. “Spanner’s new columnar engine allows high-velocity transactional writes and rich analytics in one place, eliminating data copies and replication lag so customers get instant answers.” 

Columnar storage meets vectorized execution

The heart of the Spanner columnar engine is its innovative architecture, which combines columnar storage with vectorized query execution.

Columnar Storage in Spanner: Hybrid Architecture

Unlike traditional row-oriented storage, where an entire row is stored contiguously, columnar storage stores data column by column. This offers several advantages for analytical workloads:

• Reduced I/O: Analytical queries often access only a few columns at a time. With columnar storage, only the relevant columns need to be read from disk, significantly reducing I/O operations.

• Improved compression: Data within a single column is typically of the same data type and often exhibits similar storage patterns, leading to much higher compression ratios. This means more data can fit in memory and fewer bytes need to be read.

• Efficient scans: When scanning a column, consecutive values can be processed together, for more efficient data processing.

Spanner columnar engine integrates a columnar format alongside its existing row-oriented storage. This unified transactional and analytical processing design allows Spanner to maintain its OLTP performance while accelerating analytical queries up to 200X on your live operational data.

Vectorized execution: turbocharging your queries

To complement columnar storage, the columnar engine makes use of Spanner’s vectorized execution capabilities. While traditional query engines process data tuple-by-tuple (row by row), a vectorized engine processes data in batches (vectors) of rows. This approach dramatically improves CPU utilization, with:

• Reduced function call overhead: Instead of calling a function for each individual row, vectorized engines call functions once for an entire batch, significantly reducing overhead.

• Optimized memory access: Vectorized processing often results in more cache-friendly memory access patterns, further boosting performance.

The combination of columnar storage and vectorized execution means that analytical queries on Spanner can run orders of magnitude faster, allowing for real-time insights on your global-scale data.

Better with BigQuery: Accelerating federated queries

The Spanner columnar engine takes its integration with Google’s Data Cloud ecosystem a step further, specifically enhancing integrations between Spanner and BigQuery. For enterprises that leverage BigQuery for data warehousing and analytics, federating queries directly to Spanner has always been a valuable capability. Now, with the Spanner columnar engine, this integration becomes even more potent, by delivering faster insights on operational data.

Data Boost, Spanner’s fully managed, elastically scalable compute service for analytical workloads, is at the forefront of this acceleration. When BigQuery issues a federated query to Spanner, and that query can benefit from columnar scans and vectorized execution, Data Boost automatically leverages the Spanner columnar engine. This provides:

• Faster analytical insights: Complex analytical queries initiated from BigQuery that target your Spanner data execute significantly faster, bringing near-real-time operational data into your broader analytical landscape.

• Reduced impact on OLTP: Data Boost helps ensure that analytical workloads are offloaded from your primary Spanner compute resources, preventing impact on transactional operations.

• Simplified data architecture: You can get the best of both worlds – Spanner’s transactional consistency and BigQuery’s analytical prowess – without the need for complex ETL pipelines to duplicate data.

This integration empowers data analysts and scientists to combine Spanner’s live operational data with other datasets in BigQuery for richer, more timely insights and decision-making.

Columnar engine in action: Accelerating your analytical queries

Let’s look at some sample queries that should see significant acceleration with the Spanner columnar engine. These types of queries, common in analytical and graph workloads, benefit from columnar scans and vectorized processing.

Scenario: Imagine a large e-commerce database; for demonstration purposes, we’ll use the same schema as the TPC-H benchmark.

Query 1: Revenue from discounted shipments in a given Year

SQL

Acceleration: This query heavily benefits from scanning only the l_shipdate, l_extendedprice, l_discount, and l_quantity columns from the lineitem table. Vectorized execution rapidly applies the date, discount, and quantity filters to identify qualifying rows.

Query 2: Total quantity of non-discounted items

SQL

Acceleration: This query heavily benefits from scanning only the l_discount and l_quantity columns from the lineitem table. Vectorized execution rapidly applies the equality filter (l_discount = 0) to identify matching rows.

Query 3: Item count and discount range for specific tax brackets..

SQL

Acceleration: This query benefits heavily from scanning only the l_tax and l_discount columns from the lineitem table. Vectorized execution rapidly applies the IN filter on the l_tax column to identify all matching rows.

Query 4: Scan friend relationships to find the N most connected people in the graph using Spanner Graph

GQL

Acceleration:This query benefits heavily from scanning subgraphs and by loading only relevant columns from the graph.

Query 5: Perform a K-nearest neighbor vector similarity search to retrieve the top 10 most semantically similar embeddings with perfect recall

GQL

Acceleration:This query benefits heavily from scanning contiguously stored vector embeddings and by loading only relevant columns from the table.

Get started with Spanner columnar engine today! 

The Spanner columnar engine is designed for businesses looking to unlock faster, deeper, real-time insights from their operational data without compromising Spanner’s foundational strengths. We are incredibly excited about the possibilities this opens up for developers and data analysts alike. We invite you to be among the first to try the Spanner columnar engine. Request access to the Preview of Spanner columnar engine today by signing up at bit.ly/spannercolumnar. We look forward to seeing what you build!

Once you are satisfied with the notebook you’ve co-developed with AI, you can then schedule it for automated runs, or use it in a multi-step DAG with BigQuery Pipelines.

Multi-cell code generation for anything you want to do with data

AI-first Colab Enterprise notebooks also support code generation for a wide range of tasks and follow the same interaction pattern as the Data Science Agent mentioned above. For example, using the chat interface you can prompt to:

• Generate code for arbitrary Python-based data transformation, visualization, analytics (e.g., run a causal analysis).

• Manage Colab environment (e.g., install new libraries).

• Generate code for interacting with other Google Cloud services (e.g., manage a function deployment to Cloud Run).   

The human-in-the-loop interaction design allows for approval, changes and editing of the generated code.

You can also transform your existing code. Simply describe a change in natural language (e.g., “add error handling to this data loading function” or “refactor this monolithic function into smaller, more modular parts”) and the agent will identify and modify the relevant code for you. 

Easy visualizations

The Python visualization ecosystem is rich with many choices such as Matplotlib, Seaborn, Plotly etc. While these already work well in Colab notebooks, using these libraries requires writing boilerplate code and high familiarity with the library to get a chart with good fit and finish.

AI-First Colab Notebooks excel in generating Python code for such visualizations. Simply start with a prompt like “Generate a chart displaying…” referencing your data source which can be a BigQuery table, a local Dataframe in Colab or even an uploaded file. Next just approve and run the code to have the visualization generated for you. To modify the visualization, for example, change axis to log axis or change color of a chart, simply prompt for the incremental changes and the agent will adjust the code to your needs

Explaining and fixing errors

Colab has an built-in error explanation and fixing flow. If your AI generated or user authored code cell runs into an error, you can click the ‘Explain Error’ shortcut which opens the notebook chat, which explains the error and generates the remediation code in diff view for approval.

Fast and intelligent code completion

Code completion in Colab Enterprise offers implicit suggestions as you type, accelerating your  workflow by reducing keystrokes. Accept suggestions with a tab or modify them.

Get started today

The AI-first Colab Enterprise with its Data Science Agent is transforming how data professionals work. Across BigQuery and Vertex AI, the Colab Enterprise experience is seamless and the notebooks created are interoperable, regardless of where they are created. 

To access Colab Enterprise:

• BigQuery: Navigate to Google Cloud Console > BigQuery > Notebook 

• Vertex AI: Navigate to Google Cloud Console > Vertex AI > Colab Enterprise. 

The AI-first notebook experience with Data Science Agent is currently available in US and Asia regions in Preview and will be rolled out to other Google Cloud regions in the coming days.

If you have a feature request, a question on availability in your region or feedback, reach us at vertex-notebooks-previews-external@google.com or fill out this form.