Cloud

At Google Cloud Next, we introduced H4D VMs, our latest machine type for high performance computing (HPC). Building upon existing HPC offerings, H4D VMs are designed to address the evolving needs of demanding workloads in industries such as manufacturing, weather forecasting, EDA, and healthcare and life sciences.

H4D VMs are powered by the 5th Generation AMD EPYC^TM Processors, offering improved whole-node VM performance of more than 12,000 gflops and improved memory bandwidth of more than 950 GB/s. H4D provides low-latency and 200 Gbps network bandwidth using Cloud Remote Direct Memory Access (RDMA) on Titanium, the first of our CPU-based VMs to do so. This powerful combination enables you to efficiently scale your HPC workloads and achieve insights faster.

For open-source High-Performance Linpack (OSS-HPL), a widely-used benchmark for measuring the floating-point computing power of supercomputers, H4D offers 1.8x higher performance per VM and 1.6x higher performance per core compared to C3D. Additionally, H4D offers 5.8x higher performance per VM and 1.7x higher performance per core compared to C2D.

For STREAM Triad, a benchmark to measure memory bandwidth, H4D offers 1.3x higher performance per VM and 1.4x higher performance per core compared to C3D. Additionally, H4D offers 3x higher performance per VM and 1.4x higher performance per core compared to C2D.

Improved HPC application performance

H4D VMs deliver strong compute performance and memory bandwidth, significantly outperforming previous generations of AMD-based VMs like C2D and C3D, allowing for faster simulations and analysis, and delivering significant performance gains (relative to a prior generation AMD-based HPC VM, C2D) across various HPC applications and benchmarks, as illustrated below:

• Manufacturing
  • CFD apps like Siemens^TM Simcenter STAR-CCM+^TM/HIMach show up to 3.6x improvement.
  • CFD apps like Ansys Fluent/f1_racecar_140 show up to 3.6x improvement.
  • FEA Explicit apps like Altair Radioss/T10m show up to 3.6x improvement.
  • CFD apps like OpenFoam/Motorbike_20m show up to 2.9x improvement. 
  • FEA Implicit apps like Ansys Mechanical/gearbox shows up to 2.7x improvement.
• Healthcare and life sciences:
  • Molecular Dynamics (GROMACS) shows up to 5x improvement.
• Weather forecasting
  • Industry standard benchmark WRFv4 shows up to 3.6x improvement.

“Our deep collaboration with Google Cloud powers the next generation of cloud-based HPC with the announcement of the new H4D VMs. Google Cloud has leveraged the architectural advances of our 5^th Gen AMD EPYC CPUs to create an offering that delivers impressive performance uplift compared to previous generations across a variety of HPC benchmarks. This will empower customers to achieve fast insights and accelerate their most demanding HPC workloads.” – Ram Peddibhotla, corporate vice president, Cloud Business, AMD

Faster HPC with Cloud RDMA on Titanium

H4D’s performance is made possible with Cloud RDMA, a new Titanium offload that’s available for the first time on these VMs. Cloud RDMA is specifically engineered to support HPC workloads that rely heavily on inter-node communication, such as computational fluid dynamics, weather modeling, molecular dynamics, and more. By offloading network processing, Cloud RDMA provides predictable, low-latency, high-bandwidth communication between compute nodes, thus minimizing host CPU bottlenecks. 

Under the hood, Cloud RDMA uses Google’s innovative Falcon hardware transport for reliable, low-latency communication over our Ethernet-based data center networks, effectively resolving the traditional challenges of RDMA over Ethernet while helping to ensure predictable, high performance at scale. 

Cloud RDMA over Falcon speeds up simulations by efficiently utilizing more computational resources. For example, for smaller CFD problems like OpenFoam/motorbike_20m and Simcenter Star-CCM+/HIMach10, which have limited inherent parallelism and are typically challenging to accelerate, H4D results in 3.4x and 1.9x speedup, respectively, on four VMs compared to TCP.

For larger models, Falcon also helps maintain strong scaling. Using 32 VMs, Falcon achieved a 2.8x speedup over TCP for GROMACS/Lignocellulose and a 1.3x speedup for WRFv4/Conus 2.5km.

Cluster management and scheduling capabilities

H4D VMs will support both Dynamic Workload Scheduler (DWS) and Cluster Director (formerly known as Hypercompute Cluster).

DWS helps schedule HPC workloads for optimal performance and cost-effectiveness, providing resource availability for time-sensitive simulations and flexible HPC jobs.

Cluster Director, which lets you deploy and scale a large, physically-colocated accelerator cluster as a single unit, is now extending its capabilities to HPC environments. Cluster Director simplifies deploying and managing complex HPC clusters on H4D VMs by allowing researchers to easily set up and run large-scale simulations.

VM sizes and regional availability

We offer H4D VMs in both standard and high-memory configurations to cater to diverse workload requirements. We also provide options with local SSD for workloads that demand high-speed storage, such as CPU-based seismic processing and structural mechanics applications (e.g., Abaqus, NASTRAN, Altair OptiStruct and Ansys Mechanical).

H4D VMs are currently available in us-central1-a (Iowa), and europe-west4-b (Netherlands), with additional regions in progress.

What our customers and partners are saying

Want to try it out?

We’re excited to see how H4D VMs will empower you to achieve faster results with your HPC workloads! Sign up for the preview by filling out this form.

For decades, businesses have wrestled with unlocking the true potential of their data for real-time operations. Bigtable, Google Cloud’s pioneering NoSQL database, has been the engine behind massive-scale, low-latency applications that operate at a global scale. It was purpose-built for the challenges faced in real-time applications, and remains a key piece of Google infrastructure, including YouTube and Ads. 

This week at Google Cloud Next, we announced continuous materialized views, an expansion of Bigtable’ SQL capabilities. Bigtable SQL and continuous materialized views enable users to build fully-managed, real-time application backends using familiar SQL syntax, including specialized features that preserve Bigtable’s flexible schema — a vital aspect of real-time applications. 

Whether you’re building streaming applications, real-time aggregations, or global AI analysis on a continuous data stream, Bigtable just got a whole lot easier — and much more powerful. 

Bigtable’s SQL interface, now generally available   

Bigtable recently transformed the developer experience by adding SQL support, now generally available. SQL support makes it easier for development teams to work with Bigtable’s flexibility and speed.

The Bigtable SQL interface enhances accessibility and streamlines application development by facilitating rapid troubleshooting and data analysis. This unlocks new use cases, like real-time dashboards utilizing distributed counting for instant metric retrieval and improved product search through K nearest neighbors (KNN) similarity search. A wide range of customers, spanning innovative AI startups to traditional financial institutions, are enthusiastic about Bigtable SQL’s potential to broaden developer access to Bigtable’s capabilities.

“Imagine coding with AI that understands your entire codebase. That’s Augment Code, an AI coding platform that gives you context in every feature. Bigtable’s robustness and scaling enable us to work with large code repositories. Its ease of use allowed us to build security features that safeguard our customers’ valuable intellectual property. As our engineering team grows, Bigtable SQL will make it easier to onboard new engineers who can immediately start to work with Bigtable’s fast access to structured, semi-structured, or unstructured data while using a familiar SQL interface” said Igor Ostrovsky, cofounder and CTO, Augment.

“Equifax leverages Bigtable within our proprietary data fabric for the high-performance storage of financial journals.  Our data pipeline team evaluated Bigtable’s SQL interface and found it to be a valuable tool for directly accessing our enterprise data assets and improved Bigtable’s ease of use for SQL-experienced teams. This means more of our team can work efficiently with Bigtable and we anticipate boosted productivity and better integration capabilities,” said Varadarajan Elangadu Raghunathan and Lakshmi Narayanan Veena Subramaniyam, vice-presidents, Data Fabric Decision Science.

Bigtable SQL has also been praised for offering a smooth migration path from databases with distributed key-value architectures and SQL-based query languages, including Cassandra (CQL) and HBase with Apache Phoenix.

“At Pega, we are building real-time decisioning applications that require very low latency query responses to make sure our clients get real-time data to drive their business. The new SQL interface in Bigtable is a compelling option for us as we look for alternatives to our existing database,” said Arjen van der Broek, principal product manager, Data and Integrations, Pega.

This week, Bigtable is also adding new preview functionalities to its SQL language including GROUP BYs and aggregations, an UNPACK transform for working with timestamped data, and structured row keys for working with data that is stored in a multi-part row key.

Continuous materialized views, now in preview

Bigtable SQL integrates with Bigtable’s recently introduced continuous materialized views (preview), offering a solution to traditional materialized view limitations like data staleness and maintenance complexity. This allows for real-time aggregation and analysis of data streams across applications such as media streaming, e-commerce, advertising, social media, and industrial monitoring.

Bigtable materialized views are fully managed and make updates incrementally without impacting your user queries from applications. Bigtable materialized views also support a rich SQL language including functions and aggregations. 

“With Bigtable’s new Materialized Views, we’ve unleashed the full potential of low-latency use cases for clients of our Customer Data Platform. By defining SQL-based aggregations/transformations at ingestion, we’ve eliminated the complexities and delays of ETL in our time series use cases. Moreover, using data transformations during ingestion, we’ve unlocked the ability for our AI applications to receive perfectly prepared data with minimal latencies,” said Sathish KS, Chief Technology Officer, Zeotap.

Ecosystem integrations 

To get useful real-time analytics, you often need to pull data from many sources and do so with very low latency. As Bigtable expands its SQL interface, it is also expanding its ecosystem compatibility making it easier to build end to end applications using simple connectors and SQL. 

Open-source Apache Kafka Bigtable Sink 
Customers often rely on Google Cloud Managed Service for Apache Kafka to build pipelines that stream data into Bigtable and other analytics systems. To help customers build high-performance data pipelines, the Bigtable team has open-sourced a new Bigtable Sink for Apache Kafka so you can send data from Kafka to Bigtable in milliseconds. 

Open-source Apache Flink Connector for Bigtable
Apache Flink is a stream-processing framework that lets you manipulate data in real time. With the recently launched Apache Flink to Bigtable Connector, you can construct a pipeline that lets you transform streaming data and write the outputs into Bigtable using both the high-level Apache Flink Table API and the more granular Datastream API. 

BigQuery Continuous Queries, now generally available  
BigQuery continuous queries are SQL statements that run continuously and can export the output rows produced into a Bigtable table. This feature is now generally available and can help you build a real-time analytics database using the combination of Bigtable and BigQuery. 

Continuous Queries are also available from BigQuery’s Python frameworks as a bigrames.streaming API, so Python developers can create fully-managed jobs that synchronize online datasets in Bigtable with offline datasets in BigQuery. 

“BigQuery continuous queries enables our application to use real-time stream processing and ML predictions by simply writing a SQL statement. It’s a great service that allows us to launch products quickly and easily,” said Shuntaro Kasai and Ryo Ueda, MLOps Engineers, DMM.com.

Bigtable CQL Client: Bigtable is now in preview and Cassandra-compatible

The Cassandra Query Language (CQL) is the query language of Apache Cassandra. With the launch of Bigtable CQL Client, developers can now migrate their applications to Bigtable with minimal to no code change, and enjoy the familiarity of CQL on enterprise-grade, high-performance Bigtable. Bigtable also supports common tools in the Cassandra ecosystem like the CQL shell (CQLsh), as well as Cassandra’s own data migration utilities which enable seamless migrations from Cassandra, with no downtime significantly reducing operational overhead.

Get started using the Bigtable CQL Client and migration utilities here.

Convergence: NoSQL’s embrace of SQL power

In this blog, we discussed a significant advancement that empowers developers to use SQL with Bigtable. You can easily get started with the flexible SQL language from any existing Bigtable cluster using Bigtable Studio and start to create materialized views on streams of data coming from Kafka and Flink. 

To learn more, check out the GoogleSQL for Bigtable overview.

As an object storage service, Google Cloud Storage is popular for its simplicity and scale, a big part of which is due to the stateless REST protocols that you can use to read and write data. But with the rise of AI and as more customers look to run data-intensive workloads, two major obstacles to using object storage are its higher latency and lack of file-oriented semantics. With the launch of Rapid Storage on Google Cloud, we’ve added a stateful gRPC-based streaming protocol that provides sub-millisecond read/write latency and the ability to easily append data to an object, while maintaining the high aggregate throughput and scale of object storage. In this post, we’ll share an architectural perspective into how and why we went with this approach, and the new types of workloads it unlocks.

It all comes back to Colossus, Google’s internal zonal cluster-level file system that underpins most (if not all) of our products. As we discussed in a recent blog post, Colossus supports our most demanding performance-focused products with sophisticated SSD placement techniques that deliver low latency and massive scale. 

Another key ingredient in Colossus’s performance is its stateful protocol — and with Rapid Storage, we’re bringing the power of the Colossus stateful protocol directly to Google Cloud customers.

When a Colossus client creates or reads a file, the client first opens the file and gets a handle, a collection of state that includes all the information about how that file is stored, including which disks the file’s data is stored on. Clients can use this handle when reading or writing to talk directly to the disks via an optimized RDMA-like network protocol, as we previously outlined in our Snap networking system paper.

Handles can also be used to support ultra-low latency durable appends, which is extremely useful for demanding database and streaming analytics applications. For example, Spanner and Bigtable both write transactions to a log file that requires durable storage and that is on the critical path for database mutations. Similarly, BigQuery supports streaming to a table while massively parallel batch jobs perform computations over recently ingested data. These applications open Colossus files in append mode, and the Colossus client running in the application uses the handle to write their database mutations and table data directly to disks over the network. To ensure the data is stored durably, Colossus replicates its data across several disks, performing writes in parallel and using a quorum technique to avoid waiting on stragglers.

The above image shows the steps that are taken to append data to a file.

1. The application opens the file in append mode. The Colossus Curator constructs a handle and sends it to the Colossus Client running in-process, which caches the handle.

2. The application issues a write call for an arbitrary-sized log entry to the Colossus Client.

3. The Colossus Client, using the disk addresses in the handle, writes the log entry in parallel to all the disks.

Rapid Storage builds on Colossus’s stateful protocol, leveraging gRPC-based streaming for the underlying transport. When performing low-latency reads and writes to Rapid Storage objects, the Cloud Storage client establishes a stream, providing the same request parameters used in Cloud Storage’s REST protocols, such as the bucket and object name. Further, all the time-consuming Cloud Storage operations such as user authorization and metadata accesses are front-loaded and performed at stream creation time, so subsequent read and write operations go directly to Colossus without any additional overhead, allowing for appendable writes and repeated ranged reads with sub-millisecond latency.

This Colossus architecture enables Rapid Storage to support 20 million requests per second in a single bucket — a scale that is extremely useful in a variety of AI/ML applications. For example, when pre-training a model, pre-processed, tokenized training data is fed into GPUs or TPUs, typically in large files that each contain thousands of tokens. But the data is rarely read sequentially, for example, because different random samples are read in different orders as the training progresses. With Rapid Storage’s stateful protocol, a stream can be established at the start of the training run before executing massively parallel ranged-reads at sub-millisecond speeds. This helps to ensure that accelerators aren’t blocked on storage latency.

Likewise, with appends, Rapid Storage takes advantage of Colossus’s stateful protocol to provide durable writes with sub-millisecond latency, and supports unlimited appends to a single object up to the object size limit.  A major challenge with stateful append protocols is how to handle cases where the client or server hangs or crashes. With Rapid Storage, the client receives a handle from Cloud Storage when creating the stream. If the stream gets interrupted but the client wants to continue reading or appending to the object, the client can re-establish a new stream using this handle, which streamlines this flow and minimizes any latency hiccups. It gets trickier when there is a problem on the client, and the application wants to continue appending to an object from a new client. To simplify this, Rapid Storage guarantees that only one gRPC stream can write to an object at a time; each new stream takes over ownership of the object, transactionally locking out any prior stream. Finally, each append operation includes the offset that’s being written to, ensuring that data correctness is always preserved even in the face of network partitions and replays.

In the above image, a new client takes over ownership of an object, locking out the previous owner. 

1. Initially, client 1 appends data to an object stored on three disks.

2. The application decides to fail over to client 2, which opens this object in append mode. The Colossus Curator transactionally locks out client 1 by increasing a version number on each object data replica.

3. Client 1 attempts to append more data to the object, but cannot because its ownership was tied to the old version number.

To make it as easy as possible to integrate Rapid Storage into your applications, we are also updating our SDKs to support gRPC streaming-based appends and expose a simple application-oriented API. Writing data using handles is a familiar concept in the filesystems world, so we’ve integrated Rapid Storage into Cloud Storage FUSE, which provides clients with file-like access to Cloud Storage buckets, for low-latency file-oriented workloads. Rapid Storage also natively enables Hierarchical Namespace as part of its zonal bucket type, providing enhanced performance, consistency, and folder-oriented APIs.

In short, Rapid Storage combines the sub-millisecond latency of block-like storage, the throughput of a parallel filesystem, and the scalability and ease of use of object storage, and it does all this in large part due to Colossus. Here are some interesting workloads we’ve seen our customers explore during the preview:

• AI/ML data preparation, training, and checkpointing

• Distributed database architecture optimization

• Batch and streaming analytics processing

• Video live-streaming and transcoding

• Logging and monitoring

Interested in trying Rapid Storage? Indicate your interest here or reach out through your Google Cloud representative. 

Visit us at Google Cloud Next and attend the breakout sessions “What’s new with Google Cloud’s Storage” (BRK2-025), “AI Hypercomputer: Mastering your Storage Infrastructure” (BRK2-020), and “Under the Iceberg: Simple, unified Cloud Storage for analytics data lakes” (BRK2-026) to learn more.

As organizations continue to prioritize cloud-first strategies to accelerate innovation and gain competitive advantage, legacy databases  remain a bottleneck by hindering modernization and stifling growth with unfriendly licensing, complex agreements, and rigid infrastructure. 

That’s why this week at Google Cloud Next, we’re announcing that Database Migration Service (DMS) is extending its comprehensive database modernization offering to support SQL Server to PostgreSQL migrations, enabling you to unlock the potential of open-source databases in the cloud and build modern, scalable, and cost-effective applications.

While holding great benefits, migrating from SQL Server to a modern, managed PostgreSQL offering like AlloyDB or Cloud SQL can be a highly complex task. Even though SQL Server and PostgreSQL both adhere to SQL standards, they still have fundamental differences in their architectures, data types, and procedural languages which require deep expertise in both technologies for a successful migration.

For example, SQL Server’s T-SQL syntax and built-in functions often require manual translation to PostgreSQL’s PL/pgSQL. Data type mappings can be intricate, as SQL Server’s DATETIME precision and NVARCHAR handling differ from PostgreSQL’s equivalents. 

Furthermore, features like SQL Server’s stored procedures, triggers, and functions often necessitate significant refactoring to align with PostgreSQL’s implementation. This requires deep knowledge in both database systems, as well as specific migration expertise that developers typically don’t possess, and it requires hours of painstaking work, even with the benefit of an automated conversion tool.

Simplifying database modernization with Database Migration Service 

DMS is a fully-managed, serverless cloud service that offers a complete set of capabilities to simplify database “lift and shift” migrations and database modernization journeys.

For modernization efforts, DMS offers an interactive experience that includes data migration, as well as schema and resident code conversion, all in the same powerful user interface. For data migration, it offers high-throughput database initial loads followed by low-latency change data capture to reduce downtime and minimize the impact on business critical applications.

Announcing SQL Server to PostgreSQL migration

The new SQL Server to PostgreSQL.migration experience supports the migration of both self-managed and cloud-managed SQL Server offerings to Cloud SQL for PostgreSQL and AlloyDB to accelerate your database modernization journey. Similar to the existing database modernization offerings, this new experience features a high-throughput initial load of the database followed by seamless change data capture (CDC) replication to synchronize the SQL Server source and PostgreSQL destination, all while the production application is up and running to ensure minimal business interruption.

For SQL Server schema and code conversion, DMS offers a fast, customizable algorithmic code conversion engine that automates the conversion of most of the database schema and code to the appropriate PostgreSQL dialect, leaving minimal manual conversion work for the user to complete.  

The algorithmic conversion engine maps the source database data types and SQL commands to the most suitable PostgreSQL ones, and even refactors complex source features which have no direct PostgreSQL equivalents to achieve the same functionality using available PostgreSQL capabilities. Algorithmic engines are extremely accurate, by nature, for the scenarios they are programmed for. However, they’re limited to just those scenarios, and in real-life usage some of the database code will consist of scenarios that can’t be anticipated.

For these situations, we’re pushing the boundaries of automated database modernization with the introduction of the Gemini automatic conversion engine. This new engine automatically augments the output of the algorithmic conversion, further automating the conversion tasks and reducing the amount of remaining manual work. It also provides a comprehensive conversion report, highlighting which parts of the code were enhanced, why they were changed, and how they were converted.

Instead of spending time researching suitable PostgreSQL features and fixing conversion issues, you can simply review the Gemini recommendations in the conversion report and mark the conversion as verified. Reviewing the completed conversions instead of having to research and fix issues can significantly reduce the manual migration effort and speed up the conversion process. 

To further empower SQL Server DBAs, DMS offers a Gemini conversion assist with targeted yet comprehensive SQL Server to PostgreSQL conversion training. Gemini analyzes both the source and the converted code and explains the conversion rationale, highlighting the chosen PostgreSQL features, why they were used, and how they compare to the SQL Server ones. It can then optimize the migrated code for better performance and automatically generate comprehensive comments, for better long-term maintainability. 

At Google Cloud, we’ve been working closely with customers looking to modernize their database estate. One of them is Wayfair LLC, an American online home store for furniture and decor.

“Google Cloud’s Database Migration Service simplifies the process of modernizing databases. Features like Change Data Capture to reduce downtime and AI-assisted code conversion help evolve our database usage more efficiently. This makes the migration process less manual and time-consuming, allowing teams to spend more time on development and less on infrastructure,” said Shashank Srivastava, software engineering manager, Data Foundations, Wayfair.

How to get started

To start your Gemini-powered SQL Server migration, navigate to the Database Migration page in the Google Cloud console, and follow these simple steps:

1. Create your source and destination connection profiles, which contain information about the source and destination databases. These connection profiles can later be used for additional migrations.

2. Create a conversion workspace that automatically converts your source schema and the code to a PostgreSQL schema and compatible SQL. Make sure you choose to enable the new Gemini-powered conversion workspace capabilities.

3. Review the converted schema objects and SQL code, and apply them to your destination Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL instance.

4. Create a migration job and choose the conversion workspace and connection profiles previously created. 

5. Test your migration job and get started whenever you’re ready.

To learn more about how Database Migration Service can help you modernize your SQL Server databases, please review our DMS documentation and start your migration journey today.

Supporting customers where they want to be is a core value at Google Cloud, and a big part of the reason that we have partnered with Oracle — so that you can innovate faster with the best of Google and the best of Oracle. 

This week at Google Cloud Next, we announced significant expansions to our Oracle Database offerings, including the preview of Oracle Base Database Service for a flexible and controllable way to run Oracle databases in the cloud; general availability of Oracle Exadata X11M, bringing the latest generation of the Oracle Exadata platform to Google Cloud; and additional enterprise-ready capabilities including customer managed encryption keys (CMEK).

We are continuing to invest in global infrastructure for Oracle, with a total of 20 locations available in the coming months, adding Oracle Database@Google Cloud presence in Australia, Brazil, Canada, India, Italy, and Japan. 

These announcements follow our developments with Oracle since last July, when we launched Oracle Database@Google Cloud. This partnership enables customers to migrate and modernize their Oracle workloads and start taking advantage of Google’s industry-leading data and AI capabilities such as BigQuery, Vertex AI platform, and Gemini foundation models. 

Additional features provide customers with even more options in their modernization journey, such as the fully managed Oracle Autonomous Database Serverless. They can also benefit from increased reliability and resiliency features, such as cross-region disaster recovery and Oracle Maximum Availability Gold certification.

“Banco Actinver is committed to providing innovative financial solutions to our clients. By combining the security and performance of Oracle Database with Google Cloud’s data analytics and AI tools, we’re gaining deeper insights into market trends, enhancing our services, and delivering personalized experiences to our customers,” said Jorge Fernandez, CIO, Banco Actinver.

Innovative new capabilities

We’re expanding our offerings to empower customers with the flexibility to manage a diverse set of database workloads cost effectively.

• Oracle Base Database Service: The new Base Database Service delivers a highly controllable and customizable foundational database platform, built on Oracle Cloud Infrastructure (OCI) virtual machines and general-purpose infrastructure. It can empower businesses with the flexibility to manage a diverse range of database workloads directly.

• Enhanced Oracle Database Services: In addition to the availability of Exadata Cloud Service, Autonomous Database Service, Oracle Linux, and Oracle on Google Compute Engine (GCE) and Google Kubernetes Engine (GKE), we are pleased to share general availability of Oracle Exadata X11M. Oracle Database@Google Cloud now offers the latest generation of Oracle Exadata machines, the X11M, with enhanced performance and scalability for demanding database workloads. These new machines provide significant performance gains and increased capacity, enabling customers to run even the most intensive Oracle applications with ease. X11M will be available in all new regions.

Customers are embracing Oracle Database@Google Cloud, and to support their global needs, we’re expanding our footprint while maintaining the highest standards of application performance and reliability. 

• Expanding to 20 Oracle Database@Google Cloud Locations in the coming months: To further support the growing demand for Oracle workloads on Google Cloud, we are launching in more locations, including U.S. Central 1 (Iowa), North America-Northeast 1 (Montreal), North America-Northeast 2 (Toronto), Asia-Northeast 1 (Tokyo), Asia-Northeast 2 (Osaka), Asia-South 1 (Mumbai), Asia-South 2 (Delhi), South America-East 1 (Sao Paulo), Europe-West (Italy), Australia-Southeast2 (Melbourne), and Australia-Southeast1 (Sydney) — and additional zones in Ashburn, Frankfurt, London, Melbourne, and Italy. The new regions and expanded capacity are in addition to Google Cloud regions across U.S. East (Ashburn), U.S. West (Salt Lake City), U.K. South (London), and Germany Central (Frankfurt) that are available today.

• New Partner Cross-Cloud Interconnect availability:  Partner Cross-Cloud Interconnect for OCI is pleased to expand our global network offerings with new multicloud connectivity between Google and Oracle Cloud Infrastructure in Toronto and Zurich. This further complements our existing 11 regions already served, ensuring the lowest possible latency between both clouds while keeping traffic private and secure.

• Cross Region Disaster Recovery: Cross Region Disaster Recovery support for Oracle workloads on Oracle Autonomous Database ensures high availability and resilience, protecting against potential outages and providing continuous operation for critical applications.

• Enterprise-grade networking upgrades: Advanced networking upgrades enable enterprises to efficiently deploy their Oracle resources along with Google Cloud and share resources.

Industry-leading certifications and user experience

Google Cloud is committed to providing a seamless and efficient experience for Oracle customers, ensuring that managing and utilizing Oracle databases is straightforward and effective. We offer a combination of native Google Cloud tools and Oracle Cloud Infrastructure (OCI) interfaces, along with robust support for various applications and systems.

• Enhanced user experience: Google Cloud is committed to providing an easy-to-use experience for Oracle customers, offering a Google Cloud integrated user experience for application developers and routine database operations, alongside an OCI-native experience for advanced database management. This includes support for Shared VPC, APIs, SDKs, and Terraform.

• Application support: Google Cloud is pleased to announce the support for Oracle applications running on Google Cloud, ensuring compatibility and optimal performance, including Oracle E-Business Suite, Peoplesoft Enterprise, JD Edwards Enterprise One, Hyperion Financial Management, and Retail Merchandising.

• SAP and Oracle Capability: Oracle workloads on Google Compute Engine are now supported by SAP and Oracle, further validating Google Cloud as a trusted platform for running enterprise applications.

• Integration with Google Cloud Monitoring: Provides enterprises a unified monitoring and alerting mechanism across all their Google Cloud database services, now including Oracle Database.

• New support in Google Cloud Backup and DR: Our backup service now provides central, policy-based management for backup of Oracle workloads along with other Google Cloud services using secure backup vaults for data protection — isolating and protecting data from threats like ransomware and accidental deletion.

Google Cloud’s strengths make it the preferred hyperscaler for running mission-critical Oracle workloads. 

Get started right away from your Google Cloud Console or learn more here.

The high-performance storage stack in AI Hypercomputer incorporates learnings from geographic regions, zones, and GPU/TPU architectures, to create an agile, economical, integrated storage architecture. Recently, we’ve made several innovations to improve accelerator utilization with high-performance storage, helping you to optimize costs, resources, and accelerate your AI workloads:

• Rapid Storage: A new Cloud Storage zonal bucket that provides industry-leading <1ms random read and write latency, 20x faster data access, 6 TB/s of throughput, and 5x lower latency for random reads and writes compared to other leading hyperscalers.

• Anywhere Cache: A new, strongly consistent cache that works with existing regional buckets to cache data within a selected zone. Anywhere Cache reduces latency up to 70% and 2.5TB/s, accelerating AI workloads; and maximizes goodput by keeping data close to GPU or TPUs. 

• Google Cloud Managed Lustre: A new high-performance, fully managed parallel file system built on the DDN EXAScaler Lustre file system. This zonal storage solution provides PB scale at under 1ms latency, millions of IOPS, and TB/s of throughput for AI workloads.

• Storage Intelligence, the industry’s first offering for generating storage insights specific to their environment by querying object metadata at scale and using the power of LLMs. Storage Intelligence not only provides insights into vast data estates, it also provides the ability to take actions, e.g., using ‘bucket relocation’ to non-disruptively co-locate data with accelerators.

Rapid Storage enables AI workloads with millisecond-latency

To train, checkpoint, and serve AI models at peak efficiency, you need to keep your GPU or TPUs saturated with data to minimize wasted compute (as measured by goodput). But traditional object storage suffers from a critical limitation: latency. Using Google’s Colossus cluster-level file system, we are delivering a new approach to colocate storage and AI accelerators in a new zonal bucket. By “sitting on” Colossus, Rapid Storage avoids the typical regional storage latency of having accelerators that reside in one zone and data that resides in another.

Unlike regional Cloud Storage buckets, a Rapid Storage zonal bucket concentrates data within the same zone that your GPUs and TPUs run in, helping to achieve sub-millisecond read/write latencies and high throughput. In fact, Rapid Storage delivers 5x lower latency for random reads and writes compared to other leading hyperscalers. Combined with throughput of up to 6 TB/s per bucket and up to 20 million queries per second (QPS), you can now use Rapid Storage to train AI models with new levels of performance. 

And because performance shouldn’t come at the cost of complexity, you can mount a Rapid Storage bucket as a file system leveraging Cloud Storage FUSE. This lets common AI frameworks such as TensorFlow and PyTorch access object storage without having to modify any code.

Anywhere Cache puts data in your preferred zone 

Anywhere Cache is a strongly consistent zonal read cache that works with existing storage buckets (Regional, Multi-regional, or Dual-Region) and intelligently caches data within your selected zone. As a result, Anywhere Cache delivers up to 70% improvement in read-storage latency. By dynamically caching data to the desired zone and close to your GPUs or TPUs, it delivers performance of up to 2.5 TB/s, keeping multiple epoch training times minimized. Should conditions change, e.g., there’s a shift in accelerator availability, Anywhere Cache ensures your data accompanies the AI accelerators. You can enable Anywhere Cache in other regions and other zones with a single click, with no changes to your bucket or application. Moreover, it eliminates egress fees for cached data — among existing Anywhere Cache customers with multi-region buckets, 70% have seen cost benefits.

Google Cloud Managed Lustre accelerates HPC and AI workloads

AI workloads can access small files, random I/O, while needing the sub-millisecond latency of a parallel file system. The new Google Cloud Managed Lustre is a fully managed parallel file system service that provides full POSIX support and persistent zonal storage that scales from terabytes to petabytes. As a persistent parallel file system, Managed Lustre lets you confidently store your training, checkpoint, and serving data, while delivering high throughput, sub-millisecond latency, and millions of IOPS across multiple jobs — all while maximizing goodput. With its full-duplex network utilization, Managed Lustre can fully saturate VMs at 20GB/s and can deliver up to 1TB/s in aggregate throughput, while support for the Cloud Storage bulk import/export API makes it easy to move datasets to and from Cloud Storage. Managed Lustre is built in collaboration with DDN and based on EXAScaler.

Analyze and act on data with Storage Intelligence

Your AI models can only be as good as the data you train them on. Today, we announced Storage Intelligence, a new service that can help you find the right set of data by querying the metadata across all of your buckets to be used for AI training, improving your AI cost-optimization efforts. Storage Intelligence queries object metadata at scale using the power of LLMs, helping to generate storage insights specific to an environment. The first such service from a cloud hyperscaler, Storage Intelligence lets you analyze the millions — or billions — of object metadata in your buckets, and projects across your organization. With the insights from this analysis, you can make informed decisions about eliminating duplicate objects, identifying objects that can be deleted or tiered to a lower storage class through Object Lifecycle Management or Autoclass, or identifying objects that violate your company’s security policies, to name a few.

High performance storage for your AI workloads

We built our Rapid Storage, Anywhere Cache, and Managed Lustre, high-performance storage solutions to deliver availability, high throughput, low latency, and durable architectures. Storage Intelligence adds to that, providing valuable, actionable insights into your storage estate. 

To learn more about these innovations, visit us at Next 25 and attend the breakout sessions, “What’s new with Google Cloud’s Storage” (BRK2-025) and “AI Hypercomputer: Mastering your Storage Infrastructure” (BRK2-020).

Today at Google Cloud Next ‘25, we’re announcing a major step in making Looker the most powerful platform for data analysis and exploration,by enhancing it with powerful AI capabilities and a new reporting experience, all built on our trusted semantic model — the foundation for accurate, reliable insights in the AI era.

Starting today, all platform users can now leverage conversational analytics to analyze their data using natural language and Google’s latest Gemini models. We’re also debuting a brand-new reporting experience within Looker, designed for enhanced data storytelling and streamlined exploration. Both innovations are now available to all Looker-hosted customers.

Modern organizations require more than just accurate insights; they need AI to uncover hidden patterns, predict trends, and drive intelligent action. Gemini in Looker and the introduction of Looker reports makes business intelligence simpler and more accessible for everyone. This empowers users across the organization, reduces the burden on data teams, and frees analysts to focus on higher-impact work.

Looker’s unique foundation is its semantic layer, which ensures everyone works from a single source of truth. Combined with Google’s AI, Looker now delivers intelligent insights and automates analysis, accelerating data-driven decisions across your organization.

Gemini in Looker now available to all platform users

At Google Cloud Next ’24, we introduced Gemini in Looker to bring intelligent AI-powered BI to everyone, featuring a suite of capabilities, or assistants, that let users ask questions of their data in natural language and simplify tasks and workflows like data modeling, and chart and presentation generation. 

Since then, we’ve brought those features to life in preview, and we are now expanding their access to all platform users, given the product’s level of maturity and accuracy. These include Conversational Analytics, for gaining insights into your data through natural language queries; Visualization Assistant for custom visuals initiated by natural language, letting you easily configure charts and visualizations for dashboard creation; Formula Assistant for powerful on-the-fly calculated fields and instant ad-hoc analysis; Automated Slide Generation for impactful presentations with insightful and instant text summaries of your data; and LookML Code Assistant to simplify code creation, including guidance and suggestions to create dimensions, groups, measures and more.

Also available in preview is our cCode iInterpreter for Conversational Analytics, which enables business users to perform complex tasks and derive advanced analytics, including forecasting and anomaly detection using natural language without needing deep Python expertise. You can learn more about this new capability and sign up here.

Conversational Analytics API

To bring the power of conversational analytics beyond the Looker interface, we are introducing the Conversational Analytics API. Developers can now embed natural language query capabilities directly into custom applications, internal tools, or workflows, backed by trusted data access and scalable, reliable data modeling that can adapt to evolving needs.

This API allows you to build custom BI agent experiences, leveraging Looker’s trusted semantic model for accuracy and Google’s advanced AI models (including NL2SQL, RAG, and VizGen). Developers can embed this functionality easily to create intuitive data experiences, enable complex analysis via natural language, and even share insights generated from these conversations within the Looker platform.(Sign up here for preview access to the Conversational Analytics API.)

Introducing Looker reports

Self-service analysis is key to empowering line-of-business users and fostering collaboration. Building on the success and user-friendliness of Looker Studio, we’re bringing its powerful visualization and reporting capabilities directly into the core Looker platform with the introduction of Looker reports.

Looker reports bring enhanced data storytelling, streamlined exploration, and broader data connectivity to users, including reports generated from native Looker content, direct connections to Microsoft Excel and Google Sheets data, first-party connectors and ad-hoc access to various data sources.

Creating compelling, interactive reports is now easier than ever. Looker reports features the intuitive drag-and-drop interface users love, granular design controls, a rich library of visualizations and templates, and real-time collaboration capabilities.

This new reporting environment lives alongside your existing Looker Dashboards and Explores within Looker’s governed framework. Importantly, Looker Reports seamlessly integrates with Gemini in Looker, allowing you to leverage conversational analytics within this new reporting experience.

Faster, more reliable development with continuous integration

With Google Cloud’s acquisition of Spectacles.dev, we are enabling developers to automate testing and validation of SQL, LookML changes, leading to faster, more reliable development cycles. Robust CI/CD practices build data trust by ensuring the accuracy and consistency of your semantic model — crucial for dependable AI-powered BI.

These advancements – the expanded availability of Gemini in Looker, the new Conversational Analytics API, the introduction of Looker reports, and native Continuous Integration capabilities – represent a major leap forward in delivering a complete AI-for-BI platform. We’re making it easier than ever to access trusted insights, leverage powerful AI, and foster a truly data-driven culture. 

Join us at Google Cloud Next and be sure to watch our What’s new in Looker: AI for BI session on demand after the event to experience the future of BI with Looker, and discover how complete AI for BI can transform your data into a strategic advantage.

When it comes to AI, inference is where today’s generative AI models can solve real-world business problems. Google Kubernetes Engine (GKE) is seeing increasing adoption of gen AI inference. For example, customers like HubX run inference of image-based models to serve over 250k images/day to power gen AI experiences, and Snap runs AI inference on GKE for its ad ranking system. 

However, there are challenges when deploying gen AI inference. First, during the evaluation phase of this journey, you have to evaluate all your accelerator options. You need to choose the right one for your use case. While many customers are interested in using Tensor Processing Units (TPU), they are looking for compatibility with popular model servers. Then, once you’re in production, you need to load-balance traffic, manage price-performance with real traffic at scale, monitor performance, and debug any issues that arise. 

To help, this week at Google Cloud Next, we introduced new gen AI inference capabilities for GKE: 

• GKE Inference Quickstart to help you set up inference environments according to best practices enhancements 

• GKE TPU serving stack to help you easily benefit from the price-perf of TPUs 

• GKE Inference Gateway, which introduces gen-AI-aware scaling and load balancing techniques 

Together these capabilities help reduce serving costs by over 30%, tail latency by 60%, and increase throughput by up to 40% compared to other managed and open-source Kubernetes offerings.

GKE Inference Quickstart

GKE Inference Quickstart helps you select and optimize the best accelerator, model server and scaling configuration for your AI/ML inference applications. It includes information about instance types, their model compatibility across GPU and TPUs, and benchmarks for how a given accelerator can help you meet your performance goals. Then, once your accelerators are configured, GKE Inference Quickstart can help you with Kubernetes scaling, as well as new inference-specific metrics. In future releases, GKE Inference Quickstart will be available as a Gemini Cloud Assist experience.

GKE TPU serving stack 

With support for TPUs and vLLM, one of the leading open-source model servers, you get seamless portability across GPUs and TPUs. This means you can use any open model, select the vLLM:TPU container image and just deploy on GKE without any TPU-specific changes. GKE Inference Quickstart also recommends TPU best practices so you can seamlessly run on TPUs without any switching costs. For customers who want to run state-of-the-art models, Pathways, used internally at Google for large models like Gemini, allows you to run multi-host and disaggregated serving.

GKE Inference Gateway

GKE Gateway is an abstraction backed by a load balancer to route incoming requests to your Kubernetes applications, and traditionally, it has been tuned for web serving applications, using load-balancing techniques such as round-robin, whose requests have very predictable patterns. But LLMs have high variability in their request patterns. This can result in high tail latencies and uneven compute utilization, which can negatively impact the end-user experience and unnecessarily increase inference costs. In addition, traditional Gateway does not support routing infrastructure for popular Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA), which can increase GPU efficiency by model reuse during inference. 

For scale-out scenarios, the new GKE Inference Gateway provides gen-AI-aware load balancing, for optimal routing. With GKE Inference Gateway, you can define routing rules for safe rollouts, cross-regional preferences, and performance goals such as priority. Finally, GKE Inference Gateway supports LoRA, which lets you map multiple models to the same underlying service, for better efficiency.

To summarize, the visual below shows the needs of the customers during the different stages of the AI inference journey, and how GKE Inference Quickstart, GKE TPU serving stack and GKE Inference Gateway help simplify the evaluation, onboarding and production phases.

What our customers are saying

“Using TPUs on GKE, especially the newer Trillium for inference, particularly for image generation, has reduced latency by up to 66%, leading to a better user experience and increased conversion rates. Users get responses in under 10 seconds instead of waiting up to 30 seconds. This is crucial for user engagement and retention.” – Cem Ortabas, Co-founder, HubX 

“Optimizing price-performance for generative AI inference is key for our customers. We are excited to see GKE Inference Gateway with its optimized load balancing and extensibility in open-source. The new GKE Inference Gateway capabilities could help us further improve performance for our customers’ inference workloads “ – Chaoyu Yang, CEO & Founder, BentoML

With GKE’s new inference capabilities, you get a powerful set of capabilities to take the next step with AI. To learn more, join our GKE gen AI inference breakout session at Next 25, and hear how Snap re-architected their inference platform.

Data is the fuel for AI, and organizations are racing to leverage enterprise data to build AI agents, intelligent search, and AI-powered analytics for productivity, deeper insights, and a competitive edge. To power their data clouds, tens of thousands of organizations already choose BigQuery and its integrated AI capabilities.

This decade requires  AI-native, multimodal, and agentic data-to-AI platforms, with BigQuery leading the way as the autonomous data-to-AI platform. Finally, we have a platform that infuses AI, makes unstructured data a first class citizen, accelerates open lakehouses and embeds governance.

As an autonomous data-to-AI platform, BigQuery enables a self-managing multimodal data foundation that’s built for processing and activation of all data types, with advanced engines that can be operated on by specialized agents. The platform’s shared catalog and governance layer helps ensure consistent data access, metadata understanding, and security policies across all data and engines, minimizing silos and simplifying management. BigQuery is built on Google’s global infrastructure, leveraging high-bandwidth networks, low-latency storage, and AI-accelerated hardware (TPUs, GPUs), for virtually unlimited scalability. With our commitment to open standards and AI embedded at every layer, this fully integrated architecture accelerates your journey to AI-driven insights at the lowest cost possible.

AI assistance across the entire data lifecycle

Gemini in BigQuery brings a set of AI-powered assistive capabilities to automate data discovery and exploration, data preparation and engineering, analysis and insight generation, covering the entire data journey.

Thousands of organizations are using Gemini in BigQuery. In fact, usage of code assist in BigQuery grew 350% over the last 9 months, with over a 60% code generation acceptance rate across SQL and Python.

Yesterday we announced the general availability of several additional Gemini in BigQuery features and added new capabilities that further enhance and automate your analytics workflows.

• Simplify data preparation: BigQuery Gemini-assisted data preparation (GA) provides intelligent suggestions for data enrichment, easily identifies and rectifies data inconsistencies, provides low-code visual data pipelines, and automates the execution and monitoring of your data pipelines. 
• Faster time to insights with data canvas: BigQuery data canvas allows you to find, transform, query, and visualize data using natural language prompts and a graphic interface. New dataset-level insights (preview) can surface hidden relationships between tables and generate cross-table queries by integrating query usage analysis and metadata.
• Boost productivity with coding assistance for DataFrames: With AI code assistance in BigQuery, you can use natural language prompts to generate or suggest code in SQL or Python, or to explain an existing SQL query. We are now extending this code assist capabilities to BigQuery DataFrames (preview). 
• Improve data and AI governance: New automated metadata generation (preview) uses profile scans and Gemini to create clear and consistent descriptions for columns, tables, and glossary terms, even with large datasets. This metadata improves governance and helps AI agents find the data they need for exploration and analysis.
• Accelerate BigQuery migrations: SQL translation assistance (GA) is an AI-based translator that lets you create Gemini-enhanced rules to customize your SQL translations. You can describe changes to the SQL translation output using natural language prompts or specify SQL patterns to find and replace. This can also help in rapidly increasing familiarity with BigQuery SQL. 

Even better, Gemini in BigQuery features are being introduced into existing BigQuery pricing models across all BigQuery compute pricing options. Express interest to try the new features today.

A multimodal autonomous data foundation

BigQuery helps you develop an autonomous data foundation by unifying analytics capabilities across diverse data types and enabling the seamless, concurrent analysis of both structured and unstructured data within a single platform. In fact, customer data in BigQuery grew nearly 30% last year, adding to the multiple exabytes already stored. Furthermore, its native, first-party integration with Vertex AI allows you to apply powerful AI models directly to your data, eliminating the requirement for complex data movement or replication.

“BigQuery and Vertex AI bring all our data and AI together into a single platform. This has transformed how we take action on customer feedback from a lengthy manual process, to a simple natural language query in seconds, allowing us to get to customer insights in minutes instead of months.” – TJ Allard, Lead Data Scientist, Mattel

Yesterday we announced several innovations to enhance our unstructured data support and AI processing:

• BigQuery tables for Apache Iceberg (preview:) Connect your Apache Iceberg data to SQL, Spark, AI and third-party engines in an open and interoperable manner so you can get the flexibility of an open data lakehouse alongside the performance and integrated tooling of BigQuery. This offering provides adaptive and autonomous table management, delivers high-performance streaming, auto-AI-generated insights, near-infinite serverless scale and advanced governance.

• Native multimodal support for BigQuery tables: Built on object tables, the new ObjectRef data type (preview) enables storage and querying of unstructured and structured data using Python and SQL functions.

• Multimodal capabilities for Python users: The BigQuery DataFrames library now has  multimodal capabilities for unified structured and unstructured analytics, AI operators for semantic insights, and Gemini code assistance. 
• Easy capture of Unstructured Data Processing: BigQuery ML new capabilities in preview include AI.GENERATE_TABLE for capturing the output of LLM inference within SQL clauses. Additionally, we’ve expanded model choice to include Anthropic’s Claude, Llama, and Mistral models, and open-source models hosted on Vertex AI.
• Scalable, faster and cost-efficient vector search: BigQuery vector search allows you to generate, manage, and search embeddings within a serverless, fully integrated environment for powerful analytics. We are introducing a new index type (GA) based on Google’s ScaNN model coupled with a CPU-optimized distance computation algorithm, enabling scalable, faster and more cost-efficient processing.
• Easier time-series forecasting in BigQuery ML: BigQuery ML simplifies time-series forecasting with the new TimesFM model (preview). This pretrained model, developed by Google Research, is user-friendly, accurate, fast, and scalable.

• Pinpoint the key factors driving changes in your metrics: Organizations constantly need to answer questions like “Why did our sales drop last month?”. ” Answering these “why” questions accurately is vital, but often involves complex manual analysis. BigQuery’s contribution analysis feature (GA) helps you pinpoint the key factors (or combinations of factors) responsible for the most significant changes in a metric. 

Simplified and unified governance in BigQuery

BigQuery offers built-in governance capabilities that simplify how you discover, manage, monitor, govern, and use your data and AI assets. BigQuery universal catalog brings together a data catalog (formerly known as the Dataplex Catalog) and a fully managed, serverless metastore. Yesterday, we announced the following new capabilities for BigQuery governance:

• Enable engine interoperability across BigQuery, Apache Spark, and Apache Flink engines with BigQuery metastore (GA). With support for the Iceberg Catalog it simplifies data discovery and querying across engines, mirroring the open-source experience. 

• Empower your organization with a business glossary (GA), which provides a shared understanding of data. Customers can define and administer company terms in a business glossary, identify data stewards for these terms, and attach them to data asset fields, to improve context, collaboration, and search.

• Perform bulk extract of catalog entries into Cloud Storage with Catalog metadata export (GA). This enables a wide range of use cases including metadata analytics by making the export output queryable from BigQuery, programmatic workloads requiring access to a large scope of metadata, and metadata integration.

• Automatic at-scale cataloging of BigLake and object tables (GA): BigQuery harvests up-to-date metadata for structured and unstructured data from Cloud Storage and automatically creates query-ready BigLake tables at scale.

Enhanced enterprise capabilities 

BigQuery offers easy managed disaster recovery (GA) for compute and storage. It features automatic failover coordination, continuous near-real-time data replication to a secondary region, and fast, transparent recovery during outages. This provides business continuity with industry-leading recovery point objectives (RPO) and recovery time objectives (RTO).

We are also introducing new workload management capabilities (preview) for isolation, resource control, and observability. Users gain granular controls with flexible, securable reservations that allow users to assign to different jobs in the same project to different reservations. Features include reservation level fair sharing of slots, predictability in performance of reservations, and enhanced observability through reservation attribution in billing for better cost tracking.

Improved query performance

To further simplify analytics, we introduced several new innovations to help you get the most out of SQL and make your queries work better for you automatically.  Query performance optimizations (GA) improve query performance and automatically identify and accelerate relevant workloads with no changes required to the schema or queries. These include:

• Low latency API for short queries enables short-query-optimized mode to improve overall latency of short queries that are common in workloads such as data exploration or building dashboards by executing the query and returns the results inline for SELECT statements. 

• History-based optimizations use information from already-completed executions of similar queries to apply additional optimizations and further improve query performance such as query latency and slot-time consumed.

•  Column metadata index (CMETA) provides (almost) infinitely scalable and highly performant metadata management for BigQuery, where you can go from 10GB tables to 100PB and still get great price/performance, without having to worry about redesign or replatforming.

New analytics capabilities

• SQL-based continuous queries (GA): Simplify real-time data processing by enabling users to express complex transformations in SQL. You can run continuously processing SQL statements to help analyze, transform, and reverse ETL data the moment new events arrive in BigQuery. This feature now supports slot autoscaling, greater monitoring through Cloud Monitoring, and exports to other clouds. 

• Simplify SQL with BigQuery pipe syntax (GA): This unique feature extends standard SQL to make it simpler, more concise, and flexible. Pipe syntax lets you apply operators in any order and as often as you need, streamlining SQL queries for tasks like data exploration, dashboard creation, and log analysis. Pipe syntax enhances clarity, efficiency, and maintainability, and its compatibility with most standard SQL operators ensures broad usability.

• Geospatial analytics (preview): We’re integrating rich, analysis-ready geospatial datasets from Earth Engine and Google Maps Platform directly into BigQuery data clean rooms. And with the ST_RegionStats function, BigQuery users can now use Earth Engine to efficiently extract statistics from raster data. For the first time, data analysts and decision-makers can access geospatial insights from Google Maps Platform and Earth Engine that lead to more informed and faster business and sustainability outcomes. Key decisions such as optimal site selection for a new business location, how to optimize operations and maintenance of your infrastructure assets, how to enable sustainable sourcing, and more are now enabled directly in BigQuery.

Continued innovation with the ISV ecosystem

Finally, BigQuery’s capabilities are being significantly extended by its vibrant partner ecosystem, through new and enhanced AI integrations and solutions. Anthropic’s Claude models are now accessible via BigQuery ML, facilitating functions like text generation and summarization. GrowthLoop introduced its Compound Marketing Engine built on BigQuery with Growth Agents powered by Gemini, so marketing can build personalized audiences and journeys that drive rapidly compounding growth. Furthermore, Informatica is expanding their services on Google Cloud to enable sophisticated analytical and AI governance use cases.

Significant advancements have also occurred in data management and observability. Fivetran introduced its Managed Data Lake Service for Cloud Storage with native integration with BigQuery metastore and automatic data conversion to open table formats like Apache Iceberg and Delta Lake, improving data lake management and discoverability. DBT is now integrated with BigQuery DataFrames and DBT Cloud is now on Google Cloud. Finally, Datadog has introduced expanded monitoring capabilities for BigQuery, providing granular visibility into query performance, usage attribution, and data quality metrics. 

These partner innovations provide customers with expanded functionality, improved operational control, and streamlined access to sophisticated capabilities within the BigQuery ecosystem.

A data-to-AI platform for the autonomous era

BigQuery is evolving beyond a data warehouse and becoming the autonomous data-to-AI platform for all your data teams. The Gemini-powered agents, unified architecture, and commitment to open standards are lowering the barriers to entry for AI-powered analytics and enabling you to focus on what you do best: building innovative models and driving data-driven decisions.

As we bring together more capabilities within a unified platform we are making it easy for you to consume and use the platform with unified commercials with our new BigQuery spend commit. This provides commitments across our BigQuery unified platform, giving you the flexibility to move spend across data processing engines, streaming, governance and more.

Learn more about BigQuery and start exploring how these new features can transform your organization. 

^{Special thanks to Geeta Banda, Head of Outbound Product Management, for her contributions to this blog post.}

Data is the critical foundation for AI, yet a vast amount of data’s potential remains untapped. Why? Data quality remains a top barrier. To use enterprise data to drive analytics-driven decisions and build differentiated AI, businesses need to be able to find, understand, and trust their data assets. This requires effective data governance encompassing discovery, cataloging, metadata management, quality assurance, sharing, and access control.

The stakes are high. According to Gartner, “through 2026, those organizations that don’t enable and support their AI use cases through an AI-ready data practice will see over 60% of AI projects fail to deliver on business SLAs and be abandoned.”

At Google Cloud Next 25, we’re announcing BigQuery unified governance, powerful data governance capabilities that help enterprises keep pace with governance complexities. Data silos, fragmented metadata, and ambiguous ownership create significant risks and impede innovation. BigQuery unified governance provides services and tools organizations need to simplify data management and unlock actionable insights.

BigQuery’s built-in, intelligent governance simplifies data and AI management, helping organizations discover, understand, and leverage their assets, transforming governance from a burden into a powerful tool for data activation. Central to BigQuery governance is BigQuery universal catalog, a unified, AI-powered data catalog that natively integrates Dataplex, BigQuery sharing, security and metastore capabilities, bringing together business, technical, and runtime metadata.

BigQuery’s unified governance capabilities are:

1. Unified: BigQuery brings governance directly into the heart of your data-to-AI lifecycle, enabling discovery, understanding, governance, and utilization of your data assets and AI models. This gives data administrators, stewards, and custodians robust tools for metadata management and policy enforcement, providing end-to-end data-to-AI lineage, data profiling, insights, and secure sharing. And with the new universal semantic search, finding the right data is as simple as asking a question in natural language.

2. Intelligent: New governance capabilities powered by gen AI stand to revolutionize data management. By harnessing the power of large language models (LLMs), BigQuery universal catalog can help you uncover hidden relationships between BigQuery data assets, enable automated metadata curation and intelligent query recommendations at scale, automate governance, and democratize data-driven insights across the organization.

3. Open: BigQuery universal catalog insulates you from change with support for open storage standards such as Apache Iceberg, and a unified runtime metastore across SQL, open-source engines, and AI/ML. The BigQuery metastore, which is included in the BigQuery universal catalog, is Iceberg-compliant, enabling a multi-engine, multi-vendor architecture for governance and use of fully managed Iceberg data.

ANZ Bank, a multinational banking and financial services provider, uses the BigQuery universal catalog for comprehensive data governance, discovery, and observability.

“With BigQuery universal catalog, ANZ has significantly improved the reliability and trustworthiness of our data. The centralized data quality monitoring and automated validation features are increasing confidence and efficiency in critical business outputs and decisions based on accurate and consistent information. BigQuery governance has become a cornerstone of our data governance strategy, ensuring our data is not just available, but dependable.” Artur Kaluza, Head of Data Strategy and Transformation, Risk, ANZ

Noteworthy features

The new unified governance experience in BigQuery provides a centralized interface within the BigQuery UI for managing, securing, and sharing data and AI assets. In addition, we are introducing a wide range of key new features and capabilities across governance, sharing, and security.

Governance

1. Full-catalog search with semantic understanding (preview): Users can now discover data and AI resources across projects and data silos within BigQuery using full-catalog semantic search. This feature introduces natural-language search capabilities, making it easier for both technical and non-technical users to search the catalog.

2. Automated metadata curation (preview): BigQuery universal catalog can now automatically generate metadata for BigQuery tables, including table and column descriptions, improving data discovery and support gen AI applications.

3. AI-powered knowledge engine (preview): Users can efficiently discover hidden relationships within a dataset with automated entity-relationship visualization. By leveraging inferred relationships, BigQuery universal catalog generates suggestions for cross-table queries and natural language questions, getting new data teams up to speed fast on unfamiliar data assets.

4. Data products (preview): BigQuery data products allow data owners to create, share, and govern collections of data assets by use case, packaging and sharing them within and across organizations in a way that’s consistent, governed, and that follows security best practices.

5. Business glossary (GA): The BigQuery business glossary provides organizations with a shared understanding of their data. Customers can define and administer company terms, identify data stewards for these terms, and attach them to data asset fields, improving context, collaboration, and search.

6. Automatic at-scale cataloging of BigLake and object tables (GA): BigQuery universal catalog harvests up-to-date metadata for structured and unstructured data from Cloud Storage, and uses it to automatically create query-ready BigLake tables at scale.

7. Automated anomaly detection (preview): BigQuery universal catalog automates data anomaly detection to help you identify data errors, inconsistencies, and outliers in your data, reducing the time you spend identifying and resolving data issues.

Sharing

8. BigQuery sharing integration with Google Cloud Marketplace (preview): Data owners can monetize datasets in BigQuery sharing (formerly Analytics Hub) through Google Cloud Marketplace.

9. Stream sharing in BigQuery (GA): Curate and share valuable real-time streams with Pub/Sub topics in BigQuery sharing.

10. Stored procedure sharing in BigQuery (preview): Share SQL stored procedures and enable execution in the subscriber’s project without revealing the actual code. 

11. Query template sharing in BigQuery (preview): Customize, reuse, and restrict SQL queries in a clean room through publisher-defined query templates.

Security

12. Data policies on columns (preview): Create raw access and data-masking policies associated directly to a column and that can be reused across columns and tables.

13. Subquery support with row-level security (GA): BigQuery universal catalog now supports SQL subqueries in security access policy definitions, enabling row filtering without changing existing data models.

These built-in governance advancements within the BigQuery platform help organizations unlock the full potential of their data and AI initiatives. 

In addition to the innovation in BigQuery, we continue to partner with third-party catalog providers to complement their governance capabilities. For example, Collibra’s enterprise-wide governance for data and AI extends BigQuery universal catalog capabilities to provide end-to-end visibility, quality and stewardship across hybrid and multicloud environments. This partnership helps ensure more teams can discover and trust the data they need to do AI, no matter where it lives, accelerating and strengthening every use case.

By embedding governance into BigQuery and automating metadata management, BigQuery universal catalog is helping businesses move beyond the challenges of data silos and operational inefficiency, ultimately driving innovation and accelerating business impact. Ready to learn more? You can join several sessions covering the latest in BigQuery governance, sharing, and security featuring customer speakers:

1. What’s new in data and AI governance with Levi’s and Verizon | Apr 9, 4pm

2. Redefine data and AI governance in the unified BigQuery platform with Walmart and Box | Apr 11, 12:30pm

3. From chaos to confidence: Master governance in the age of AI with EDMC, ANZ Bank, Intesa Sanpaolo Group, and Google | Apr 9, 11am

4. Unlock the power of secure data sharing with BigQuery with Liveramp and Levi’s | Apr 9, 4pm

5. Data fabric on BigQuery: an architect’s perspective of why and how to do it with Virgin Media O2 | Apr 11, 8:30am

From unraveling the mysteries of our planet and the universe, to accelerating medical research and industrial innovation, scientific discovery impacts nearly every facet of human life. Today, scientific progress depends on the interplay of theory, experimentation, and computation, and increasingly, the most important and challenging problems require high-performance computing (HPC) and other advanced computing technologies and techniques. 

In recent years, artificial intelligence (AI) has emerged as a powerful tool for information assessment and generation, while also becoming a powerful tool for scientific discovery, business innovation, and productivity. More recently, advances in quantum computing are increasing our confidence in shortening the timelines to solving problems beyond the reach of classical computers. Quantum computers under development now will lead to larger production systems that will catalyze the creation of new drugs and materials, reduce costs and risks in complex financial and logistics scenarios, and enable the development of more capable AI models. 

At Google, our vision is to be the most comprehensive, capable, and accessible platform for science. Since 2008, Google Cloud has powered scientific discoveries, providing computational and data storage capabilities — including HPC clusters — to scientists, engineers, and developers worldwide. And this week, to enable continued revolutionary new science, we are bringing the best of Google DeepMind and Google Research together with new infrastructure and AI capabilities in Google Cloud, providing researchers with highly capable, cloud-scale tools for scientific computing. These new capabilities include:

• Supercomputing-class infrastructure for scientific computing: Researchers can now deploy and use supercomputing clusters powered by the latest H4D VMs powered by AMD CPUs, and A4/A4X VMs powered by the latest NVIDIA GPUs. These VMs have new low-latency networking that provides supercomputer-like scaling and performance. We’re also announcing Google Cloud Managed Lustre for high performance storage I/O. These resources will enable scientists to tackle large-scale, complex science problems.

• Advanced scientific applications powered by AI models for weather forecasting and biology: We’re now offering our first AI-powered science applications for the broader science community: AlphaFold 3 for predicting the structure and interactions of biomolecules, and WeatherNext models for weather forecasting. 

• AI agents for quicker ideas and faster discovery: Two new AI agents in Google Agentspace – Deep Research and Idea Generation – can help prepare comprehensive research reports and rapidly generate new scientific hypotheses. 

Let’s take a look at these new capabilities in more detail.

Supercomputing-class infrastructure and tools for science

Supercomputers are designed to achieve maximum performance on very large problems, as well as to train large AI models. With ongoing advances in science and AI, quick and easy access to supercomputing resources is critical. 

Researchers can now deploy and use supercomputering-class HPC clusters in Google Cloud based on new H4D VMs (virtual machines), our most powerful CPU-based VMs that use 5th Generation AMD EPYCTM Processors. H4D clusters are connected with Remote Direct Memory Access (RDMA) networking utilizing Google’s Falcon and Titanium offload technologies, providing low-latency communications for HPC applications. By using standard message-passing libraries over RDMA, H4D VMs can efficiently scale applications up to tens of thousands of cores, resulting in faster time-to-solution. You can register for the H4D VM preview here. 

Harvard University is using Google Cloud to advance heart disease research by simulating large-scale systems of red blood cells and other structures, including magnetically controlled artificial bacterial flagella (ABF), with the goal of developing therapies to attack and dissolve blood clots and circulating tumor cells in human vasculatures.

“With the power of Google’s new H4D-based clusters, we are poised to simulate systems approaching a trillion particles, unlocking unprecedented insights into circulatory functions and diseases. This leap in computational capability will dramatically accelerate our pursuit of breakthrough therapeutics, bringing us closer to effective precision therapies for blood vessel damage in heart disease.” – Petros Koumoutsakos, Harvard University

HPC clusters based on our recently announced A4 and A4X VMs are also a critical component of our scientific discovery portfolio. A4 VMs, built on NVIDIA’s latest HGX B200 GPUs, are a versatile and powerful tool for multiple scientific computing applications, offering excellent performance for direct numerical simulation, and for AI training. A4X VMs, accelerated by NVIDIA GB200 NVL72 GPUs, are purpose-built for training and serving the most demanding, extra-large-scale AI workloads. 

Clusters using these GPU-powered VMs can also unlock supercomputing-class performance for the next frontier of innovation: quantum computing. In the future, quantum computing systems will allow scientists to solve problems that are intractable even with the most powerful  traditional supercomputers. In the meantime, HPC clusters based on A-series VMs can be used to design tomorrow’s quantum computers and optimize quantum algorithms, by simulating large quantum circuits using the quantum simulation solution blueprint.

For example, Google Research’s Quantum AI team leverages Google Cloud to simulate the intricate device physics of quantum hardware, develop sophisticated hybrid quantum-classical algorithms, and explore and test novel quantum algorithms. This robust simulation environment facilitates scientific breakthroughs by delivering the performance and scalability essential for demanding quantum research workflows.

“We observed excellent scalability simulating a 43-qubit circuit with a depth of 30 on Google Cloud’s new GPU-based supercomputers. These results underscore the potential for researchers to develop and test larger and deeper quantum circuits, which is important for understanding the performance of quantum algorithms and accelerating progress toward applications for today’s quantum computers.” – Sergio Boixo, Director, Computer Science, Google Quantum AI

HPC clusters demand high I/O performance to keep computational performance from stalling. Our new Google Cloud Managed Lustre storage service, developed in collaboration with DataDirect Networks and based on EXAScaler technology, provides the I/O performance needed for supercomputing-scale applications. Google Cloud Managed Lustre delivers a high-performance, fully-managed parallel file system optimized for HPC and AI applications. With petabyte-scale capacity and up to 1 TB/s throughput, Managed Lustre ensures researchers have the I/O performance they need to power their scientific discoveries. Request access to the Managed Lustre preview by contacting your account representative. 

Advanced scientific applications powered by AI models

We recently announced our first AI-powered science applications for researchers and enterprises on Google Cloud: the groundbreaking AlphaFold 3 molecular structure and interaction prediction model, and the WeatherNext weather forecasting models.

AlphaFold 3, developed by Google DeepMind and Isomorphic Labs, is revolutionizing biology through its ability to predict the structure and interactions of all of life’s molecules with unprecedented accuracy. Understanding molecular structures and their interactions helps researchers better grasp complex interactions in human health and disease. AlphaFold 3 is now available for non-commercial use on Google Cloud.

“Having access to the scientific capabilities of AlphaFold on Google Cloud can help our research rapidly predict and explore the structure and interactions of all biomolecule classes. This change in capability will accelerate our understanding of diseases and enable the generation of therapeutic hypotheses.” – Sumaiya Iqbal, Senior group lead of the Ladders to Cures Accelerator, Broad Institute

To further support users, we’re simplifying access to AlphaFold 3 through a new high-throughput solution deployable via Cluster Toolkit. This turnkey solution enables efficient batch processing of hundreds to tens of thousands of sequences while minimizing costs by  autoscaling infrastructure.

In the domain of weather, Google DeepMind and Google Research WeatherNext models use AI for fast and accurate weather forecasting, and we recently released live WeatherNext AI forecasts on BigQuery and Earth Engine. Today, we’re introducing access to WeatherNext AI models via Google Cloud’s Vertex AI Model Garden, enabling practitioners to customize and deploy these advanced models for energy prediction, logistics, agriculture, risk management, and more.

With easier and more affordable access to faster and more accurate weather forecasting models, researchers can study far more scenarios, and organizations can better prepare for weather events — such as heat waves, floods, and hurricanes — to reduce their impact on infrastructure, personnel, supply chains, and communities.

For instance, Carrier plans to leverage Google Cloud’s WeatherNext AI models as part of its Home Energy Management System (HEMS) to help enhance grid flexibility and enable smarter energy management. Once deployed, WeatherNext AI models are expected to help HEMS intelligently manage energy flows in real time — charging, discharging, and redirecting energy based on grid conditions, energy demands, and weather forecasts — contributing to a more balanced and sustainable energy grid. 

Using AI as the ultimate research partner

Google’s robust ecosystem of information, productivity, and advanced AI tools has long helped drive scientific research, providing researchers with information and insight. Google Scholar is an indispensable resource for navigating the vast landscape of scientific literature and for discovering and tracking relevant publications. Then there’s Gemini, which can synthesize, summarize and explain information from highly scientific and technical content. And NotebookLM, an AI-powered research assistant, intelligently processes and summarizes selected research papers and datasets, dramatically accelerating literature reviews and extracting crucial information. 

We’re excited to announce two new AI agents in Agentspace that have the potential to further accelerate scientific research and to revolutionize hypothesis generation. Deep Research condenses hours of research by synthesizing information across internal and external sources to generate in-depth research reports. Idea Generation helps rapidly develop novel ideas through AI agents that create ideas, then test them against each other to find the best hypotheses. 

Scientists can also leverage AI Studio and Vertex AI on Google Cloud to develop customized AI applications and advanced machine learning workflows. We also recently announced Gemma 3, a collection of lightweight, state-of-the-art open models built from the same research and technology that powers our Gemini 2.0 models. These are our most advanced, portable and responsibly developed open models yet, and can be used to create scientific applications on local devices. Finally, Google Research’s Geospatial Reasoning framework, leveraging Vertex AI Agent Engine, will allow scientists and analysts to unlock powerful insights about the world through new geospatial foundation models and generative AI. 

Enabling transformational science today and tomorrow

Together, these new advanced infrastructure, AI applications, and AI productivity technologies provide new cloud-scale scientific capabilities for all kinds of computational science research. Combined with our discovery, collaboration, and productivity tools, we are providing scientists and researchers with a comprehensive array of cloud-powered scientific capabilities. 

Argonne National Laboratory, a leading laboratory for open science computational research, is working with Google Cloud to explore how advanced computing technologies and AI tools can empower scientists and engineers to make groundbreaking discoveries faster than ever. Through the collaboration, ANL will use and evaluate Google Cloud solutions for computational research, providing feedback and guidance to further advance the design, performance, and usefulness of Google Cloud for supercomputing-scale science. 

“Having access to powerful computational capabilities is critical for making new scientific discoveries and accelerating innovations that power business and society. We are eager to work with Google Cloud to leverage their comprehensive, global-scale AI and HPC infrastructure, software technologies and AI-powered applications such as AlphaFold 3. Argonne National Laboratory’s collaboration with Google Cloud will effectively drive innovation and enable discoveries that change the world — and bring these capabilities to researchers everywhere.” – Rick Stevens, Associate Laboratory Director for Computing, Environment and Life Sciences, Argonne National Laboratory

Scientific discoveries are more important than ever for solving the world’s greatest challenges. At Google, we’re building powerful advanced computing technologies to enable scientific discoveries and innovations, and we are excited to bring all these capabilities together in Google Cloud.

Practitioners can get started today with credits, training, and more with Google Cloud for Researchers. To stay informed and learn more about Google Cloud can help advance scientific research and discovery, join the Google Cloud Advanced Computing Community.

The transformative power of AI and intelligent agents is driving profound changes, where software can understand natural language questions and commands — and even autonomously act on our behalf. At the heart of this revolution is the “AI-ready” enterprise database, an active, intelligent engine that understands the semantics of structured and unstructured data, and uses the power of foundation models to create a platform where you can unlock unprecedented opportunities from enterprise data.

This week at Google Cloud Next, we’re announcing several new capabilities in AlloyDB AI to accelerate intelligent agent and application development. These include advanced semantic search with high-performance filtered vector search, automatic vector index maintenance, and a major increase in the quality of searches using the newly launched AlloyDB AI query engine and the Vertex AI Ranking API. The AI query engine brings AI-powered operators to SQL queries for filtering, as well.

We’re also launching natural language capabilities to provide users and agents with deep insights from natural language questions. Taken together, these innovations position AlloyDB as the foundation for agentic AI, evolving the database beyond data storage and conventional SQL querying to a future where intelligent agents can converse with the data and autonomously explore it on our behalf.

High-performance, high-quality, and easy semantic search

Modern apps require smart data retrieval that combines structured data with unstructured, multimodal data such as text and images. Previously, AlloyDB AI enabled semantic queries over unstructured data, deeply integrating vector search with PostgreSQL so search results are always up to date. Our next set of AlloyDB AI capabilities addresses customer requests for higher performance, better search result quality, and low-cost automated maintenance.

1. Adaptive filtering: This innovative technique, now in preview, can help ensure that filters, joins, and vector indexes deliver optimal performance when used together. Adaptive filtering optimizes the query plan once it learns the actual filter selectivity as it access data, and then can appropriately switch between filtered vector search methods.

2. Vector index auto-maintenance, also in preview, reduces how often you need to rebuild your vector indexes, while ensuring that vector indexes remain accurate and performant even as data changes. You can enable vector index auto-maintenance during index creation or when altering the index. 

3. Reranking: The newly-released AlloyDB AI query engine can enhance semantic search by combining vector search with high-accuracy AI reranking, through the new Vertex AI cross-attention Ranking API. Our reranking capability uses vector search to efficiently generate initial candidates (such as Top N) and then apply the high quality cross-attention Ranking API to accurately determine the final best results (such as Top 10) from those candidates. To give you as much flexibility as possible, AlloyDB AI can connect with any third-party ranking API, including custom ones.

4. Recall evaluator: Now generally available, this capability provides the transparency you need for managing and tuning the quality of vector search results. With a simple stored procedure, you can evaluate end-to-end recall for any query, including complex ones with filters, joins, and reranking.

5. Parallel index build: Now generally available, index build parallelization allows developers to build indexes of up to 1 billion rows in just hours, down from several times that number. To support this capability, AlloyDB AI spins up parallel processes to distribute the workload and create indexes faster.

These improvements are made possible by the deep integration of AlloyDB AI’s Scalable Nearest Neighbors (ScaNN) vector index with the PostgreSQL query planner, and they lead to notably faster performance:

• 10x faster filtered vector search when compared to the hierarchical navigable small world (HNSW) index in standard PostgreSQL.

• 10x faster index creation when compared to the HNSW index in standard PostgreSQL.

• 4x faster vector search when compared to the HNSW index in standard PostgreSQL.

AlloyDB AI natural language

Natural language interfaces on databases showed great progress in 2024, backed by AI technology that turns questions, posed either by end users or by agents, into SQL queries that provide answers. 

To further improve accuracy, a quantum leap was needed. Building on the natural language support announced last year, we’re introducing new capabilities to help you build interactive natural language user interfaces that decipher user intent accurately, and can build highly-accurate mappings of user questions to SQL queries that answer them.

• Disambiguation: Natural language is inherently ambiguous. The AlloyDB AI natural language interface will ask follow-up questions when it needs more information about user intent. Since ambiguity is often rooted deep in the data, the database is the best at solving it.

  For example, a question may refer to “John Smith,” but there may be two John Smiths in the database, or perhaps there’s a “Jon Smith,” whose first name was spelled differently, or even misspelled. AlloyDB concept types and the AlloyDB values index enable finding the relevant entities and their concepts when they’re not obvious from the question.

• High accuracy and intent explanation: AlloyDB AI natural language uses plain templates, which correspond to parameterized SQL queries, and faceted templates for providing highly-accurate, virtually-certified answers to predictable and important classes of questions.

  For example, a retailer’s product search page could theoretically include dozens of product properties — far too daunting for a screen-based faceted search interface. In contrast, a faceted search template, even with one simple search field, can answer any question that directly or indirectly poses any combination of property requirements. AlloyDB can automatically produce templates from query logs, and you can provide additional templates to boost query coverage. To ensure confidence in results, AlloyDB offers a transparent explanation of its interpretation of user inquiries.

• High accuracy and flexibility: For cases where questions are not predictable but question answering must provide flexibility, AlloyDB enables the user to raise accuracy by automatically enriching the context that is used in the mapping of the question to SQL with the rich data found in the schema, the data (such as sample data that can greatly enhance accuracy), and the query logs.

• Parameterized secure views: AlloyDB offers parameterized secure views, a new kind of database view that locks down access to end-user data at the database level, to help protect against prompt injection attacks.

• Beyond AlloyDB with Agentspace: AlloyDB AI natural language is available in Google Agentspace for building your own agents that, for example, may answer questions by combining AlloyDB data with data from other sources, such as the web or another database.

AlloyDB AI query engine

To empower you to build intuitive and powerful AI applications, AlloyDB AI query engine can unlock deep semantic insights from enterprise data through AI-powered SQL operators. AI query engine leverages Model Endpoint Management, a mechanism for calling any AI model on any platform.

Let’s review AlloyDB AI query engine and other capabilities newly available in AlloyDB AI via new AI models:

• AI query engine: AlloyDB SQL now features simple but powerful AI operators — AI.IF() for filters and joins, and AI.RANK() for ordering. These operators use natural language in SQL queries to express the filtering conditions and the ranking criteria. They can use foundation models to bring reasoning and real-world knowledge to SQL queries, and they can use cross-attention models, which also draw their power from foundation models and their real-world knowledge. In particular, AI.RANK() can use the Vertex AI Ranking API to find the most relevant results.

• Multimodal embedding generation: Previously, AlloyDB AI enabled a SQL developer to easily generate embeddings from text in SQL statements. We’ve expanded this capability to generate embeddings for any modality (text, images, and videos) so you can search using any modality.

• Updated text embedding generation: AlloyDB AI query engine provides out-of-the-box integration with the text-embedding generation model from Google DeepMind.

Getting started

We believe today’s AlloyDB AI announcements — enhanced filtered vector search, next-generation natural language support, and the AI query engine — are the foundation for the future of databases. They provide proactive insights for agents that anticipate and act decisively, powered by AI-ready data. AlloyDB AI is building a database revolution, empowering you to step boldly into this intelligent future and unlock your data’s boundless potential.

Start with a simple vector search on AlloyDB with ScaNN today. You can also get started with our newly-launched AI capabilities from the AlloyDB AI signup form.

Migrating data workloads to BigQuery, our unified Data to AI platform, just got significantly easier. You no longer have to choose between unlocking value from your data assets by migrating to a modern data platform, or mitigating risk by staying put. You can achieve both with BigQuery Migration Services, a collection of free-to-use, cloud-native services that enable large-scale transformations for data warehouses and data lakes by breaking down migrations into templated, iterative and manageable steps. They move data, code, and business logic from on-premises and cloud platforms to BigQuery, utilizing a “next-best action” approach that minimizes time-to-migrate and maximizes ROI for your business transformation.

At Google Cloud Next 25, we announced several new innovations in BigQuery Migration Services, including coverage for data science and expanding support for data engineering and data analytics workloads. New capabilities span across four stages of a data platform migration: 1) automated assessment and planning, 2) automatic code translation, 3) data migration, and 4) validation.

Let’s look at the new innovations in BigQuery Migration Services.

1. Automated discovery and assessment with estimated total cost of ownership

Your data platform migration journey begins with automated discovery and assessment of the source environment. BigQuery Migration Services’ automated assessments provide details of the existing environment, create an insights-filled view of the workloads’ projected landed state on BigQuery (including performance and estimated total cost of ownership), and guide you on how to get to BigQuery (migration planning). You can run an assessment with the push of a button on the Google Cloud console, which delivers a detailed Looker-studio report and BigQuery datasets as output. Assessments are available for Teradata, Snowflake, and Redshift, and today, we also announced that assessments for Oracle/Exadata and Cloudera/Hive are available immediately, and that a Databricks assessment is coming soon.

To help with a structured and successful migration, we also announced a source lineage service in preview. This service automatically identifies and groups dependencies between workloads, creating an explicit ordering in which to move them, helping to minimize risk and disruption, and improving time-to-value.

2. Automated code translations

Some of our heaviest investments in BigQuery Migration Services over the past years have been in our code translation services, which migrate code from 15+ sources. Today, we announced advancements in Gemini-enhanced code translation, which was previously only available in interactive mode, letting you translate code, like you would with, say, Google Translate. 

Now, Gemini-enhanced code translations are also available in batch and API modes, helping you migrate at scale. Coupled with a new unified Translation API that backs all three modes, you can first translate bulk code using batch or API modes, and then fine-tune and debug it using interactive mode. 

Now, you can also preprocess your code with Gemini, so you can migrate code that’s not just SQL, but also other kinds of code, e.g., an ETL job with SQL embedded inside XML. This means you don’t need to submit perfectly clean SQL, and can translate code from sources that don’t have full compiler coverage yet.

Finally, there’s an enhanced user-experience in the console to guide you at each step of the translation process, suggesting the next-best action to get you to the finish line.

These advancements dramatically reduce code conversion times while continuing to deliver over 95% accuracy, helping you tackle large migration jobs with greater efficiency. 

3. Data, metadata and permissions migration

Historically, BigQuery Migration Services have supported large-scale data migrations from Teradata and Amazon Redshift. Today, BigQuery Migration Services support incremental updates from Teradata, batch and incremental file and permission migrations from Cloudera, and batch and incremental data migration from Snowflake, all in preview. All migrated data is automatically validated as part of the migration process.

4.Intelligent end-to-end validation 

Each step of the migration process will soon include an intelligent validation mechanism that can incorporate schema and data-type updates, vs. static data checksum comparisons that exist today. You can combine validation with source lineage, making it easy to quickly identify discrepancies between source and target environments. This comprehensive code, data, and dependency validation helps ensure your business applications stay intact as you incrementally move them. 

Together, these investments  in each of the four stages of a data platform migration help automate your  journey while containing risk, providing deterministic outcomes, and faster ROI.

Customer successes

Customers trust BigQuery Migration Services for migrating their mission-critical workloads. BigQuery Migration Services usage has grown 3x year over year, with thousands of customers using the services to migrate workloads to BigQuery.

“By migrating from Databricks to BigQuery and combining our own models with the models provided by Google Cloud, we’ve improved the performance and efficiency of our machine learning processes and better positioned ourselves for ongoing growth.”  – Hamdi Amroun, Head of AI, Yassir

“BigQuery has unlocked unprecedented scalability and flexibility for VMO2, improving data platform availability and uptime, which ultimately enhances customer experience. By moving all key functions to Google Cloud, VMO2 has reduced its TCO for equivalent on-premises platforms by approximately 30%.” – Vinay Pai, Head of Data Architecture, Virgin Media O2

Take the next steps

Ready to start migrating your data platform to BigQuery? We’re ready to help!

• Learn more about BigQuery Migration Services and try it for free. 

• Connect with Google Cloud Consulting and select migration partners standardized on BigQuery Migration Services to achieve predictable and cost-effective outcomes.

• Sign up today for the BigQuery migration incentives program for additional benefits such as Google Cloud credits, implementation services and cloud egress credits.