gcp

Partner Summit at Google Cloud Next ’25 is your opportunity to hear from Google Cloud leaders on what’s to come in 2025 for our partners. Breakout Sessions and Lightning Talks are your ticket to unlocking growth, mastering AI, and conquering the cloud marketplace. Sign-up today to secure your seat in one of the 40+ exclusive sessions at Partner Summit. 

Need some help deciding where to prioritize your time? Our Google Cloud partner leaders have some ideas!

Jim Anderson,  VP, NA Partner Ecosystem and Channels, Google Cloud

“AI is transforming how businesses discover, utilize, and manage data. Many customers have shared that adopting AI technology will require them to reimagine their business strategies. As we guide our customers through this transformation, it’s crucial to have a partner ecosystem that can offer the necessary industry and process expertise. This session shares valuable insights into how Google’s Go-To-Market teams are approaching this challenge, and how we can partner to deliver better customer outcomes.”

Jim recommends:

• PAR101: North America’s business leaders on the AI revolution and verticalization: Discover the latest trends shaping the region, and gain actionable insights to embrace unprecedented opportunities and exceed customer expectations in this transformative age.

Troy Bertram, Managing Director, Public Sector Partner Ecosystem, Google Public Sector

“We are seeing incredible momentum within the public sector as organizations look to leverage Google AI to drive meaningful impact. Our partners are at the forefront of this evolution, building solutions that empower public servants and improve citizen outcomes. At Google Public Sector, we’re empowering partners to deliver secure, scalable, and innovative solutions that address the unique needs of government agencies, educational institutions, and nonprofits.  Partners benefit from our go-to-market differentiators, including access to Google Public Sector’s Rapid Innovation Team (RIT), Delivery Expertise Badges, specialized SecOps resources, and subscription agreements. Public sector organizations are tackling some of society’s most pressing challenges, and our ecosystem’s expertise is vital in helping them achieve their missions.”

Troy recommends:

• PAR106: Public Sector AI innovation and partner go-to-market: Learn new GTM strategies, understand co-sell, and how to utilize US Public Sector Deal Registration Discount, Partner Development Sprints, and the Rapid Innovation Team, all designed to accelerate partner success.

Javier Carrique, Director, Partner Ecosystem & Channels, LATAM, Google Cloud

“We are building the foundation for future market leadership by strategically evolving our go-to-market engine to diversify our customer base and dramatically enhance market reach. Our approach includes a strong emphasis on portfolio and value differentiation, harnessing the innovation of generative AI to create distinct market advantages, empowering our Partners for mutual growth and expanded reach, and establishing a clear and influential voice in the industry to attract a wider range of clients.”

Javier recommends:

• PAR103: Leveraging AI for partner success in LATAM: Capitalize on opportunities in LATAM & cutting edge AI technology to boost your organization’s revenue.

• PARLT103: Generative AI training for partners:  As AI becomes increasingly integrated into various industries, customers will demand partners who understand and can leverage this technology. Exclusive training ensures the partner ecosystem is prepared to meet these evolving needs.

Aimee Catalano, Senior Director, Global Partner Marketing, Google Cloud

“These sessions should be on the agenda of every marketer interested in capitalizing on Google Cloud’s benefits and resources. We offer critical overviews into our Partner Marketing Studio & various incentive offerings, sharing best practices and how to find the right programs for your business. We also showcase how we’ve integrated AI tools seamlessly into your existing workflows in order to optimize your campaigns and increase ROI. ”

Aimee recommends:

• PARLT107: Excelling with Marketing Funds: program overview and tips for success: Walk through eligibility, funding options, and best practices to help you maximize impact. 
• PARLT108: Maximize your marketing with Google Cloud: Leverage key tools and techniques, like Partner Marketing Studio, to enhance campaigns and optimize resources.

Bruno Heese, Managing Director, Partners and Channels, EMEA, Google Cloud

“The EMEA region presents a dynamic and significant opportunity for growth, and our partners are central to realizing this potential. At Partner Summit, we will be sharing Google Cloud’s strategic vision and investment priorities for EMEA, emphasizing the key industry verticals and emerging technology trends that are ripe for innovation. In a panel with partners Devoteam and Deloitte, we will also delve into the crucial aspect of driving AI adoption across the region, exploring how partners can position AI as a transformative force for their customers and develop AI-powered solutions leveraging Google Cloud’s comprehensive suite of AI tools. For EMEA partners looking to maximize their potential and achieve significant growth with Google Cloud, this is a key session to attend.”

Bruno recommends:

• PAR102: Driving AI innovation through joint opportunities in EMEA: Discover Google Cloud’s vision & investment priorities in EMEA for 2025. We’ll discuss how partners are capitalizing on opportunities to build presence in multiple markets and address generative AI implementation challenges. 

Colleen Kapase, VP, Channels & Partner Programs, Google Cloud

“To thrive in today’s rapidly shifting landscape of customer expectations and emerging technologies, our partner ecosystem and programs are adapting to meet customers where they’re at. These three sessions will highlight how partners can drive revenue through joint co-selling, make the most of our incentives, and accelerate customer AI projects. Together, your organization and Google Cloud will show up for customers as one team. ” 

Colleen recommends:

• PAR107: Partner programs: Growing together in the age of AI: Learn how to stand out as a valued partner as we share our plans for co-Sell and services opportunities, new initiatives & key metrics designed to recognize and amplify partner contributions, and exciting investment opportunities.

• PAR108: Maximizing profits with incentives & benefits: Unlock proven practices to optimize partner benefits and incentives, including programs that reward your success in driving customer cloud adoption

• PAR109: Unlocking new revenue streams with Google AI: Open up new revenue streams by building profitable AI offerings that engage customers, accelerate deals, and showcase real-world ROI.

Anthony McMahon, Managing Director, Partners and Corporate Business, APAC

“Google Cloud is experiencing rapid growth in the APAC market, powered by our partner ecosystem. This session will outline our 2025 strategy for continued growth in the region and highlight how partners can contribute and differentiate themselves. We will also discuss how, together, we can capitalize on the AI opportunity in APAC.” 

Anthony recommends:

• PAR104: Winning together: The AI opportunity in APAC: Unlock the secrets to dominating the APAC market. Get the inside scoop on our 2025 strategy and learn how to position yourself for explosive growth. If APAC is your play, this session is your win.

Victor Morales, VP,  Global System Integrators Partnerships, Google Cloud

“Excelling with Google Cloud  industry solutions means you can apply or build upon our horizontals to address niche or industry-specific business challenges. Learn how our GSIs tailor different offerings to ensure partner and customer success.”

Victor recommends:

• PARLT104: Leading the way in industry: The Google Cloud playbook: Discover Google Cloud’s industry approach & strategy. Hear from a fellow partner on how we jointly worked to bring a new solution to market and the Google Cloud resources and tools they leveraged for success. 

Stephen Orban, VP, Migrations, ISVs, and Marketplace, Google Cloud

“Enterprise software is undergoing a seismic transformation. Cloud marketplaces are changing how enterprises find, use, and manage software, with billions transacted through the Google Cloud Marketplace in 2024. Artificial Intelligence, including agentic AI, is changing how workers in every industry interact and benefit from ISV solutions. These two sessions will help our ISV and Technology Partners understand how building AI into their software solutions and transacting through the Google Cloud Marketplace will accelerate their growth and deliver better customer outcomes.” 

Stephen recommends:

• PARLT119: Accelerate co-sell readiness with Google Cloud Marketplace: Designed for ISV partners, spur your path to co-sell success with Google Cloud. Gain a clear roadmap to co-sell readiness with resources designed specifically for you. 

• PAR111: Transforming technology partnerships: An AI-first GTM playbook for ISVs: Discover how to integrate AI into your go-to-market strategy and stay ahead of the curve. It’s not just about AI; it’s about AI that works for you.

Yumi Ueno, MD, Japan Partner Ecosystem and Channels, Google Cloud

“Japan, a cornerstone of the global economy, is poised for a significant transformation. While facing challenges such as demographic shifts and the need for optimized data infrastructures, these very conditions present compelling opportunities for strategic AI deployment, particularly in the realm of generative AI. Google Cloud, in close collaboration with our valued partners, is dedicated to delivering impactful, scalable solutions that drive tangible business outcomes in the Japanese market.”

Yumi recommends:

• PAR105: Driving business with gen AI in Japan: Discover how generative AI is reshaping the Japanese business landscape, the current market dynamics, and unveil Google Cloud’s strategic AI initiatives.

Dai Vu, Managing Director, Marketplace & ISV GTM Programs, Google Cloud

“We’re seeing tremendous growth in solutions being bought and sold on Google Cloud Marketplace, with partners across the entire ecosystem from AI and data providers to channel partners and systems integrators all taking advantage of cloud marketplace as a route-to-market. I’m excited for my panel discussion with MongoDB, SADA, and Workday, strategic partners who will provide important insights for companies across different stages of the cloud marketplace journey. And I’m looking forward to the lightning talk on monetizing AI agents, which is an exciting next step in our AI evolution.” 

Dai recommends:

• PAR110: Scaling go-to-market success with Google Cloud Marketplace: Learn from the best. Gain insights from MongoDB and Workday on navigating and thriving in the Google Cloud Marketplace.

• PARLT118: Monetizing AI agents on Google Cloud Marketplace: Step into the future of AI monetization. This lightning talk will illuminate the path to capitalizing on the next wave of AI innovation.

Don’t Miss Out. Register Now!

Google Cloud Next 25 is your chance to connect with industry leaders, gain actionable insights, and propel your business forward. Secure your spot today and get ready to transform your partnership with Google Cloud.

When it comes to managing the infrastructure and AI that powers Google’s products and platforms – from Search to YouTube to Google Cloud – every decision we make has an impact. Traditionally, meeting growing demands for machine capacity means deploying new machines and that has an associated embodied carbon impact. That’s why we’re working to reduce the embodied carbon impact at our data centers by optimizing machine placement and promoting the reuse of technical infrastructure hardware.

In this post, we shine a spotlight on our hardware harvesting program, an approach to fleet deployment that prioritizes the reuse of existing hardware.

The hardware harvesting program

The concept is simple: As we deploy new machines or components in our fleet, we repurpose older equipment for alternative and/or additional use cases. The harvesting program prioritizes the reuse of existing hardware, which reduces our carbon emissions compared to exclusively buying brand new machines from the market. This program also helps conserve valuable resources and minimize waste, which contributes to a more circular economy. By scrutinizing the carbon impact of deployment decisions, we’re not just reducing emissions — we’re embedding carbon considerations into the very core of our data center machine operations and business decisions.

Hardware harvesting is not without its challenges. For the program to be successful, we need to ensure the harvested machines meet the specific demands of our workloads and our customers’ requirements, which vary depending on the type of machine and its configuration. However, our heterogeneous fleet, with a wide variety of computational, storage, and accelerator machines, gives us the flexibility to find creative solutions that support both our services and our sustainability goals.

Hardware harvesting in action

Google’s harvesting program has already yielded strong benefits. By prioritizing the reuse of existing hardware, we’ve been able to optimize the use of new equipment, reduce our carbon footprint, minimize waste and lower costs.

For example, in 2024, we needed more specific models and configurations of certain components (PCBs, CPUs, motherboards, and HDDs). We harvested them from existing machines by migrating configuration-agnostic jobs from existing machines to more efficient ones, then reclaimed the components from these specific machines. In 2024, the harvesting program helped us reuse over 293,000 components to fulfill new demand, save carbon emissions, and reduce costs. Scaling this hardware harvesting approach across Google’s data center infrastructure presents an opportunity for cost, resource, and carbon reduction.  

Looking ahead: Leading by example

Harvesting is just one example of how we’re embedding carbon considerations into our data center practices. We believe that these initiatives will play a role in helping us achieve our company-wide net-zero goal and build a more sustainable future for cloud computing and AI. Read our 2024 Environmental Report to learn more about our sustainability practices.

As we continue to refine our strategies, we aim to lead by example and encourage other companies, especially those in the cloud computing industry, to consider similar approaches.

Model endpoint management is available on AlloyDB, AlloyDB Omni and Cloud SQL for PostgreSQL.

Model endpoint management helps developers to build new experiences using SQL and provides a flexible interface to call gen AI models running anywhere — right from the database. You can generate embeddings inside the database, perform quality control on your vector search and analyze sentiment in the database, making it easier to monitor results. This feature is available through the google_ml_integration extension, which enables an integration with Vertex AI for both AlloyDB and Cloud SQL for PostgreSQL. 

Previously, the google_ml_integration extension only allowed users to call models hosted on the Vertex AI platform. With model endpoint management, you can leverage models running on any platform — including your own local environment. We also added ease-of-use support for models running on Open AI, Hugging Face, and Anthropic, as well as Vertex AI’s latest embedding models so you can easily access these models. We have preconfigured the connectivity details and input/output transformation functions for these providers, so that you can easily register the model and simply set up the authentication details.

For Vertex AI models, we have pre-registered embedding and Gemini models so that you can easily start calling them. Plus, newer embedding models have built-in support meaning you are able to access the latest versions of pre-registered models allowing you to start making prediction calls out-of-the-box. 

In this blog, we’ll walk you through three example workflows that leverage model endpoint management to build richer generative AI experiences. 

1. Generating embeddings with Open AI embeddings models

2. Leveraging Gemini to evaluate vector search results

3. Running sentiment analysis to analyze user sentiment

First, register your model.

To use your own model, register your model using the create model function, where you specify model endpoint connectivity details. You can then configure a set of optional parameters that allow you to transform the input and output of the model arguments to a format suitable for your database. Here’s an example of registering Anthropic’s Claude model.

Once you register your model, you can call it with the predict row function for any AI model — or you can use the embedding convenience function to call an embedding model.

#1: Generate embeddings with Open AI embeddings models

Model endpoint management allows you to leverage the embedding convenience function with any embeddings model, even ones that don’t run on Google Cloud. Say you want to generate embeddings with OpenAI’s ada embeddings model. With our ease-of-use support you need only register your authentication credentials, register the model, and start generating embeddings.   You first need to configure the authentication for the endpoint you would like to reach — you can do so either by creating a PostgreSQL function to specify your API key in the header of the API call or by creating a secret with secret manager and then registering the secret with model endpoint management. 

To register your secret, you simply need to specify the secret path and create an ID for the secret. You can find the secret path in the resource manager by clicking on the secret, and then clicking “copy resource name” on the specific version of the secret you want to use.

Once your secret has been registered, you can register your model and point to the secret, open_ai_secret, when you register the openai-ada model. Our ease-of-use support handles the input and output formatting so that you can generate embeddings from data in your database and directly use the output embedding for vector search.

You then need only specify the name of the model you have registered in the first argument and the text in the second argument. For instance, if you want to generate an embedding on the word “I love Google Databases”, you would invoke the embedding function like so:

If you want to generate an embedding in-line while performing a vector search, combine the embedding function with vector search in SQL using the following syntax:

Model endpoint management also has built in integrations with Vertex AI’s latest embedding models, allowing you to access any of Vertex AI’s supported text embedding models. We recommend the embedding() function for in line SQL queries or to generate stored embeddings on datasets smaller than 100k rows.  

#2: Leverage Gemini to evaluate vector search results

In addition to a deep integration with embedding models, model endpoint management provides developers out-of-the-box support for the latest Gemini models. Gemini Pro and Gemini Flash Light are both available as pre-registered models in AlloyDB and Cloud SQL for PostgreSQL. Leveraging Gemini, you can generate content, perform sentiment analysis or analyze the quality of vector search results. Let’s see how you might analyze the quality of your vector search results with Gemini using the predict row function.

Suppose you have a table apparels with an ID, product_description and embedding column. We can use model endpoint management to call Gemini to validate the vector search results by comparing a user’s search query against the product descriptions. This allows us to see discrepancies between the user’s query and the products returned by the vector search.

We are able to pass in the vector search results to Gemini to evaluate how well the user’s query matches the descriptions qualitatively, and note differences in natural language. This allows you to build quality control to your vector search use case so that your vector search application improves over time. For the full end to end use case follow this code lab.

#3: Run sentiment analysis to analyze user sentiment

One of the benefits of calling Gemini in the database is its versatility. Above, we showed how you can use it to check the quality of your vector search. Now, let’s take a look at how you might use it to analyze the sentiment of users.

Say you are an e-commerce company and you want to perform sentiment analysis on user review information stored in the database. You have a table products which stores the name of the product and their descriptions. You have another table of product reviews, product_reviews, storing user reviews of those products joined on the id of the product. You just added headphones to your online offering and want to see how well they are doing in terms of customer sentiment. You can use Gemini through model endpoint management to analyze the sentiment as positive or negative in the database and view the results as a separate column. 

First, create a wrapper function in SQL to send a prompt and the text you want to analyze the sentiment on to Gemini with the predict row function.

Now let’s say you want to analyze the sentiment on a single review — you could do it like so:

You can then generate predictions on only reviews containing the word “headphones” by using a LIKE clause and calling your get sentiment function:

This should output whether the review was “positive, negative or neutral” for user reviews regarding headphones. Allowing you to see what the user sentiment is around this new product. Later, you can use aggregators to see whether the majority of the sentiment is positive or negative. 

Get started

Model endpoint management is now available in AlloyDB, AlloyDB Omni and Cloud SQL for PostgreSQL. To get started with it, follow our documentation on AlloyDB and Cloud SQL for PostgreSQL. 

If you used the internet today, you’ve probably already benefited from generative AI. Whether it helped you get your work done faster, research home repairs, or find the perfect gift, gen AI is transforming how we get things done. These generative AI experiences use searches against vector embeddings — multi-dimensional representations of data’s meaning — to match your intent with the best answer. 

But integrating vector technology into existing applications can be challenging. Many databases have historically not supported vector search, so developers have had to integrate specialized vector databases side-by-side with their existing databases.

Enter MySQL similarity search

Cloud SQL for MySQL now supports vector storage and similarity search, which means you can transform your MySQL databases in place to integrate gen AI capabilities without a specialized vector database. Now generally available, it’s as simple as adding a new column to your existing table and loading in your vector embeddings, which you can generate using your favorite models;  for example, you can use Vertex AI’s pre-trained text embeddings models. Once you’ve imported your dataset, you can perform both k-nearest neighbors (kNN) and approximate nearest neighbors (ANN) searches by adding the right index for your use case; these search indexes were developed using Google’s open-source ScaNN libraries. Our GA offering includes the same ACID support and crash recovery for vectors that you expect from a relational database.

To think about this in action, imagine you’re the developer for a hardware store’s online shopping experience. By integrating ANN similarity search into your catalog, when a shopper asks “what do I need to fix a crack in my dining table?” you can convert this question into a vector embedding and match against all products in your catalog to find items that can be used to fix dining table cracks.

We’ve collaborated closely with companies that rely on MySQL to help them integrate generative AI into their existing applications. For instance, supply chain solution provider Manhattan Associates is exploring similarity search in MySQL to improve search results for customers using its applications.  

“Similarity search in MySQL enables us to easily integrate gen AI capabilities into the fleet of applications we’ve built on Cloud SQL for MySQL. For example, we’re exploring how we can use similarity search against product information to render better search results. This can be expanded to various searches across the application solutions we provide.” – Sanjeev Siotia, Executive Vice President & Chief Technology Officer, Manhattan Associates

Get started building

Ready to build generative AI apps on top of your MySQL databases? We have a few solutions to help you get started:

• Sample app: Lets you customize the datastore for a bot-based app, with Cloud SQL for MySQL as an option. This app uses kNN search as the search type.

• Code lab: Walks you through the basics of deploying a gen AI app with Cloud SQL and LangChain, a popular gen AI app development framework.

We can’t wait to see what you create!

Modern data teams want to use Git to collaborate effectively and adopt software engineering best practices for managing their data pipelines and analytics code. But most tools used by data teams don’t offer integration with Git version control systems, making a Git workflow feel out of reach. This forces users to copy and paste code between UIs, which is not only time-consuming but also error-prone.

To help, we’re introducing repositories in BigQuery in Preview, a new experience in BigQuery Studio that helps data teams collaborate on code stored in Git repositories.

Develop with Git in BigQuery Studio

BigQuery repositories provide a comprehensive set of features to integrate Git workflows directly into your BigQuery environment:

• Set up new repositories in BigQuery Studio where you can develop SQL queries, Notebooks, data preparation, data canvases, or text files with any file extension.

• Connect your repositories to remote git hosts like GitHub, GitLab, and other popular Git platforms.

• Edit the code in your repositories within a dedicated workspace, on your own copy of the code, before publishing changes to branches. 

• Perform most Git operations with a user-friendly interface that lets you inspect differences, commit changes, push updates, and create pull requests — all within BigQuery Studio.

Software engineering best practices for all data practitioners

BigQuery repositories help organizations standardize the way code is developed, version, and deployed. Data teams with members of different levels of technical expertise can all collaborate on the same code base, following the same software engineering best practices.

Data analysts can contribute to code repositories via a simple GUI interface that lets them create workspaces, commit changes, and push code to branches.

Data engineers can develop in BigQuery Studio or with their favourite local IDE on the same codebase.

Data scientists can develop Colab Enterprise notebooks from BigQuery Studio, within their organization’s VPC, but back the code in a remote repository where they can manage versions and ask peers for code reviews. 

Getting started

To begin using BigQuery repositories, navigate to BigQuery Studio in the Google Cloud console or visit the documentation for detailed instructions.

We are excited to announce LlamaIndex integration for Gen AI Toolbox for Databases (Toolbox). We launched Toolbox in beta last month and are thrilled to continue building on that momentum.

Gen AI Toolbox for Databases is an open-source server that streamlines the development and management of sophisticated generative AI tools that can connect to databases. Currently, Toolbox can be used to build tools for a large number of databases: AlloyDB for PostgreSQL (including AlloyDB Omni), Spanner, Cloud SQL for PostgreSQL, Cloud SQL for MySQL, Cloud SQL for SQL Server, and self-managed MySQL and PostgreSQL. Because it’s fully open-source, it includes contributions from third-party databases such as Neo4j and Dgraph. This enables you to develop tools easier, faster, and more securely by handling the complexities such as connection pooling, authentication, and more.

LlamaIndex has emerged as a leading framework for building knowledge-driven and agentic systems. It offers a comprehensive suite of tools and functionality that facilitate the development of sophisticated AI agents. Notably, LlamaIndex provides both pre-built agent architectures that can be readily deployed for common use cases, as well as customizable workflows, which enable developers to tailor the behavior of AI agents to their specific requirements.

In this post, we’ll share how LlamaIndex support for Toolbox works, Toolbox and LlamaIndex use cases, and samples to get started.

Challenges in gen AI tool management

Building AI agents that use different tools, frameworks, and data sources creates challenges, particularly when querying databases. These include:

• Complex database connections that require configuration, connection pooling, and caching for optimal performance.

• Security vulnerabilities when ensuring secure access from gen AI models to sensitive data.

• Scaling tool management due to repetitive code and modifications across multiple locations for each tool.

• Inflexible tool updates that require a complete rebuild and redeployment of the application.

• Limited workflow observability due to lack of built-in support for comprehensive monitoring and troubleshooting.

Gen AI Toolbox for Databases

Toolbox comprises two components: a server specifying the tools for application use, and a client interacting with this server to load these tools onto orchestration frameworks. This centralizes tool deployment and updates, incorporating built-in production best practices to enhance performance, security, and simplify deployments.

How LlamaIndex support works 

LlamaIndex is particularly useful for developers building knowledge assistants over enterprise data. LlamaIndex’s event-based Workflows provide a clean, easy abstraction for building production agents capable of finding information, synthesizing insights, generating reports, and taking action, even with the most complex enterprise data. 

By connecting Large Language Models (LLMs) to virtually any data source to structure data, create indices, and build powerful query engines, LlamaIndex empowers developers to rapidly extract knowledge and build AI agents, accelerating the development and adoption of LLM applications across various industries.

For enterprises, LlamaCloud provides a turn-key solution for data ingestion, parsing, indexing and storage that integrates seamlessly with the rest of the framework to get from prototype to production quickly.

For building agents, the controlled and specified calling of tools, reliable execution, and seamless passing of context back to the LLM are essential. Toolbox handles the execution itself, seamlessly running the tool and returning results. Together, Toolbox and LlamaIndex create a powerful solution for tool calling in agent workflows.

Use cases 

LlamaIndex supports a broad spectrum of different industry use cases, including agentic RAG, report generation, customer support, SQL agents, and productivity assistants. LlamaIndex’s multi-modal functionality extends to applications like retrieval-augmented image captioning, showcasing its versatility in integrating diverse data types. LlamaIndex’s hundreds of data integrations and industry-leading parsing solutions in LlamaParse make it a stand-out choice for building agents that interact with enterprise data sources.

“We’re delighted to work with Google on Gen AI Toolbox, which neatly addresses a number of real pain-points in getting production agentic applications off the ground. We think the simplified security story in particular is going to be really attractive to devs building with these popular databases,” said Laurie Voss, VP of Developer Relations at LlamaIndex.

Get started 

Through our partnership with LlamaIndex, we’re thrilled to offer enhanced value to developers building production-grade agents across diverse knowledge retrieval use cases. Here are some resources to get you started:

• llamaIndex-sdk- repo   

• Documentation  

• Quick Start Guide

All workloads are not the same. This is especially the case for AI, ML, and scientific workloads. In this blog we show how Google Cloud makes the RDMA over converged ethernet version 2 (RoCE v2) protocol available for high performance workloads.

Traditional workloads

Network communication in traditional workloads involves a well-known flow. This includes:

• Movement of data between source and destination. The application initiates requests.

• The OS processes the data, adds TCP headers and passes it to the network interface card (NIC).

• The NIC sends data on the wire based on networking and routing information.

• The Receiving NIC receives data. 

• OS processing on the receiving end strips headers and delivers data based on information.

This process involves both CPU and OS processing, and these networks can recover from latency and packet loss issues and handle data of varying sizes while functioning normally. 

AI workloads 

AI workloads are very sensitive, involve large datasets, may require high bandwidth, low latency and lossless communication for training and inference. Because there is a higher cost for running these types of jobs, it’s important that they are completed as quickly as possible and optimize processing. This can be achieved with accelerators — specialized hardware designed to significantly speed up the training and execution of AI applications. Examples of accelerators include specialized hardware chips like TPUs and GPUs.

RDMA

Remote Direct Memory Access (RDMA) technology allows systems to exchange data directly between one another without involving the OS, networking stack and CPU. This allows faster processing times since the CPU, which can become a bottleneck, is bypassed. 

Let’s take a look at how this works with GPUs.

• An RDMA-capable application initiates an RDMA operation.

• Kernel bypass takes place, avoiding the OS and CPU.

• RDMA-capable network hardware gets involved and accesses source GPU memory to transfer the data to the destination GPU memory. 

• On the receiving end, the application can retrieve the information from the GPU memory, and a notification is sent to the sender as confirmation.

Previously, Google Cloud supported RDMA-like capabilities with its own native networking stack called GPUDirect-TCPX and GPUDirect-TCPXO. Currently the capability has been expanded with RoCEv2, which implements RDMA over ethernet.

RoCE-v2-capable compute

Both the A3 Ultra and A4 Compute Engine machine types leverage RoCE v2 for high-performance networking. Each node supports eight RDMA-capable NICs connected to the isolated RDMA network. Direct GPU-to-GPU communication within a node occurs via NVLink and between nodes via RoCE.

Adopting RoCEv2 networking capabilities offers more benefits including:

• Lower latency

• Increased bandwidth — from 1.6 Tbps to 3.2 Tbps of inter-node GPU to GPU traffic

• Lossless communication due to congestion management capabilities: Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN)

• Use of UDP port 4791

• Support for new VM series like A3 Ultras, A4 and beyond

• Scalability support for large cluster deployments

• Optimized rail-designed network

Overall these features result in faster training and inference, directly improving application speed. It’s achieved through a specialized VPC network, optimized for this purpose. This high-performance connectivity is a key differentiator for demanding applications.

Get started

To enable these capabilities, follow these steps:

1. Create a reservation: Obtain your reservation ID; you may have to work with your support team for capacity requests.

2. Choose a deployment strategy: Specify the deployment region, zone, network profile, reservation ID and method.

3. Create your deployment.

You can see the configuration steps and more in the following documentation:

• Documentation: Hypercompute Cluster

• Blog: Cross-Cloud network support for AI workloads 

• GCT YouTube Channel: AI guide for Cloud Developers 

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

Software development teams are under constant pressure to deliver at an ever-increasing pace. As sponsors of the DORA research, we recently took a look at the adoption and impact of artificial intelligence on the software development lifecycle. The research shows that more than 75% of developers are relying on AI in their daily responsibilities.

With Gemini Code Assist, developers aim to boost their efficiency and code quality. But what’s the process to effectively adopt AI-assisted coding? How do you measure the impact of these tools on your team’s performance? 

In this article, we’ll provide a practical framework for adopting AI-assisted code creation, and for evaluating the effectiveness of AI assistance in your software development workflow. 

This post outlines a four-step framework to adopt AI code assistants like Gemini Code Assist on your software development team: Adoption, trust, acceleration, and impact. 

1. Adoption: Ensure your developers are actively using the tool by tracking measures like daily active use and code suggestions. 

2. Trust: Gauge developers’ confidence in the AI’s output by monitoring code suggestion acceptance rates and lines of code accepted. 

3. Acceleration: Look for improvements in development speed and software quality through existing productivity metrics like DORA measures, story points, or ticket closures.

4. Impact: Connect these improvements to your overall business goals by assessing changes in key performance indicators like revenue, market share, or time to market.

The AI-assistance journey

Committing to code AI-assistance involves change management, a process defined by a transition from one position (development before AI-assistance) to another (development after). Put simply, it’s a journey.

The journey can be understood in four progressive phases. 

1. Adoption: Evaluation and proof of concept activities to identify how and where code AI-assistance may contribute to developer outcomes.

2. Trust: Establishment of confidence in AI-assistance’s output. 

3. Acceleration: Assessment of AI-assistance’s ability to improve team development speed with existing productivity metrics like DORA measures, story points, or ticket closures. 

4. Impact: Confirmation of improvement to key business performance indicators such as revenue, market share, or time to market. 

In addition to what each phase does, it’s important to understand how involved parties contribute. Technology and business leaders frequently initiate evaluations (Adoption) and affirm ongoing success (Impact). Between these two activities, individual developers explore coding AI-assistance, familiarize with its capabilities, and establish regular use (Trust). During this time it is important to allow developers ample time to learn and experiment with ways to utilize AI. As teams of developers explore and develop trust together, the group iterates through feedback to further optimize team productivity (Acceleration).

One common mistake organizations make is believing that using code AI-assistance (Adoption) will yield immediate business results (Impact). Another way to express this is that they believe they can transition from Adoption immediately to Impact, skipping Trust and Acceleration.

If an organization does not meaningfully Adopt AI assistance tools, and has minimal Trust in its suggestions, it’s unrealistic to expect an Acceleration in team productivity, much less substantive business Impact.

Another misconception is that adopting code AI-assistance will have an impact overnight. Every organization is different, but we’ve found that at least 6 to 8 weeks, or four two-week sprints, is needed before impact to your organization’s productivity (Acceleration) may be observed. With each phase progressing into the next, it takes time for the effects of adopting AI assistance to propagate forward. This awareness is particularly important when conducting an evaluation, which we discuss further later in this article.

While the four phases of the AI assistance journey are conceptual, they can and should be measured to affirm progress and impact. Below we describe what measures may be used when and why.

• Adoption. Daily activity (developer use of AI-assistance), code suggestion (AI code recommendations), and chat exposure (AI chat requests) volume are early signals indicating whether developers are taking advantage of AI-assistance. With this phase and using these measures, you want to confirm consistent and growing daily developer engagement. As Adoption grows, you can shift focus to establishing Trust.

• Trust. Are adopting developers accepting AI-assistance? Code suggestions accepted (AI code recommendations accepted), acceptance rate (the percentage of code suggestions accepted divided by the number of code suggestions), and lines of code accepted (number of lines of code accepted) measures can be used to assess trust. Monitoring for low acceptance rates for code suggestions and lines of code should prompt you to investigate why trust may be low. You may also elicit further understanding from developer interviews and surveys (sample survey questions).

• Acceleration. You may already have developer productivity (Acceleration) measures in place, including established DORA software delivery measures. Alternatively, you may want to evaluate it through completed story points or ticket closures per time period, among other measures. Once you’ve established adoption and trust, monitoring acceleration measures for improvement can both confirm the productivity benefits of AI assistance as well as provide line of sight to business impact outcomes and measures.

• Impact. This final phase is expressed in business key performance indicators. Specific impact measures differ among organizations, and should be monitored by organization leaders to evaluate the business yield of AI-assistance. Impact measures may include revenue, market share, reduced time to product improvements, and other business health criteria. An observed improvement in Acceleration would expect to positively contribute to an Impact measure(s) as well.

It is important to be aware that AI-assistance measures, those within Adoption and Trust phases, are not development productivity measures, those found in the Acceleration phase. To illustrate why, consider the following: would a high AI-assistance code suggestion acceptance rate or significant volume of AI-assisted lines of code accepted that negatively impacted DORA measures or average ticket closures still be considered a development productivity improvement? Most would agree it would not and this is why it is important to make the distinction. AI-assistance metrics measure Adoption and Trust of AI-assistance but development productivity metrics express the impact. Impact measures reveal the final effect.

With a journey, phases, and corresponding measures defined, you can collectively use these elements as a guiding framework to progress and affirm the impact of code AI-assistance. 

Measuring impact with Gemini Code Assist

Gemini Code Assist supports Adoption and Trust measurement through Gemini for Cloud Logging logs, in Preview. Through these logs active use, code suggestions, code suggestions accepted, acceptance rate, chat exposures, and lines of code accepted are made visible. This further includes discrete activity on a per user, programming language, IDE client, and activity time basis – deep insights not always available with aggregate AI-assistance measures. These insights can be used to assess both organization journey performance and also answer specific questions like “How many AI-assisted lines of code did we accept last week, by programming language? By developer?”

While Gemini Code Assist logs provide discrete details per activity, we also provide a sample dashboard built on Log Analytics to assist with aggregate Adoption and Trust measures review.

In addition to the above, Cloud Monitoring metrics are provided to monitor active use across Gemini for Cloud, in Preview, including Gemini Code Assist.

Before making a commitment to code AI-assistance, many organizations choose to first conduct an evaluation. Like the AI-assistance adoption journey, an evaluation journey may be a phased process. Here again each phase would feed into the next and involve specific parties. 

1. Success criteria. Before starting an evaluation you need to define and baseline the evaluation success criteria. There are two audiences that need to be considered when defining success criteria, the development team and business decision makers, and both should agree on the definition of success. Success criteria could include improvements to Acceleration measures such as DORA, story point velocity, or tickets resolved, as examples. This phase is often overlooked but it is the most important phase as we have found organizations can overlook, or fail to collectively agree on, a success criteria before initiating their evaluation and then experience difficulty when trying to retroactively assess AI-assistance’s impact. 

2. Participants. While there are multiple approaches to consider, the most common involve either selecting a single team of developers and conducting evaluation with successive project efforts (the first project with AI-assistance and the following without), or comparing two teams performance in an A/B cohort (one team using AI-assistance while the other does not). Whatever you choose, discuss and agree on who will be participating and why upfront. An “apples to apples” comparison should be prioritized. For example, utilizing an A/B cohort where your A team is significantly more experienced and is also using AI-assistance against a B team that is less experienced and is not using AI-assistance may lead to an lopsided evaluation that makes impact evaluation difficult at best if not unreliable. 

3. Measure. The AI-assistance journey can guide progressing and monitoring your evaluation. Quantitative and qualitative measures, alongside success criteria, can be regularly reviewed to ensure an evaluation progresses to a point where the impact of AI-assistance can be assessed.

4. Commit. If you have agreed on your success criteria and have been measuring and facilitating progress along the way, you will be able to confirm or reject an code AI-assistance commitment pending satisfaction of the accepted success criteria.

The intensity of a code AI-assistance evaluation can vary. A Minimal evaluation may involve a handful of developers, qualitative surveys, and the monitoring of Acceleration measures only. A more typical approach in our experience, and one opted for by most organizations, is a Moderate investment looking at both quantitative and qualitative measures and involves either a team performing successive development with and without code AI-assistance, or two teams in an A/B cohort doing the same. An Involved evaluation, listed for completeness, utilizes formal research, lab studies, and analysis. In practice, we have found few organizations for an Involved evaluation.  

Whatever intensity model is pursued, we want to again reiterate the importance of defining success criteria up front and gathering baseline data for comparison.

In defining commitment success criteria it is important to consider targeting what may be sufficient to confirm AI-assistance’s impact. Often, improvements to Acceleration phase productivity measures can provide line of sight to Impact phase measure improvement (business key performance indicator). Conversely, choosing a commitment success criteria that requires significant time or effort to conclude may reduce the opportunity to confirm AI-assistance impact sooner if at all. 

Get started today

Ready to get started with AI assistance for software development? For more information about Gemini Code Assist, visit our product page, documentation, and quickstart. 

Additional resources:

• An AI-assistance commitment decision may find further support in the presence of 2024 DORA report AI-assistance impact insights and Google Research foundations.

• DORA’s software delivery performance metrics are good indicators for the Acceleration and Impact phases of your AI assistance adoption journey. Teams that are not yet tracking these metrics can use the DORA Quick Check to capture their current software delivery performance.

• DORA’s research also shows many capabilities lead to improved software delivery performance. Measurements for these capabilities can inform the Adoption and Trust phases of your adoption journey and can serve as leading indicators for later journey phases.

• Surveys (reference samples) can qualitatively assist with the adoption of AI and target specific areas you’d like to improve with the use of code assistants which may include software delivery performance, documentation quality, code review time, flow, or user centricity.

Two years ago, on a mission to redefine enterprise-grade databases we released AlloyDB for PostgreSQL in production. We saw the immense popularity and flexibility of PostgreSQL — a database developers love for being open-source — and we knew we could build upon that strong foundation to create a next generation fully managed database with a cloud-first architecture that offered strong performance, scalability, autopilot features and analytical capabilities. 

To celebrate this journey marked by significant customer adoption and industry recognition, we are launching a series of blogs to explore the innovations, growth, and customer successes that define AlloyDB, and offer a glimpse into its future. You’ll get an inside look at AlloyDB’s key differentiators, including scalability, performance, analytics and our powerful generative AI capabilities, and learn how customers are leveraging its power.

But first, if you’re new to AlloyDB, here’s a recap of what makes it special, and the milestones we’ve achieved since launching it. 

Built on the best: PostgreSQL and Google Cloud

AlloyDB’s strength lies in its foundation: the power and versatility of PostgreSQL combined with Google Cloud’s infrastructure and innovation. PostgreSQL’s popularity among developers is undeniable — it ranked as the most popular database in the 2024 Stack Overflow Developer Survey. And, according to the 2024 Gartner® Magic Quadrant™, Google Cloud is a Leader in both Strategic Cloud Platform Services (SCPS) and Cloud Database Management Systems (CDBMS). This combination makes AlloyDB uniquely positioned to meet the challenges of modern enterprise workloads.

Architectural innovations

AlloyDB’s exceptional performance and scalability are a direct result of Google Cloud’s differentiated infrastructure and an optimized database kernel. Here’s how:

• Disaggregated compute and storage: AlloyDB separates database compute from an intelligent, PostgreSQL-optimized intelligent storage service. By offloading database log processing to the storage, we reduce compute resource usage allowing those resources to be dedicated to user queries and transactions. The storage service itself further disaggregates compute (log processing) and storage (block storage), enabling independent scaling, dynamic adaptation to workloads, fault tolerance, and cost-effective, horizontally scaling. Tiered caching with automatic data placement further boosts performance. This multi-layered disaggregation is a key driver of AlloyDB’s scalability.

• Optimized database kernel: AlloyDB has a highly optimized PostgreSQL engine. This includes efficient resource management, advanced query processing, highly scalable and improved concurrency control, all focused on minimizing overhead and maximizing transaction throughput. This allows AlloyDB to vertically scale transactional workloads to 128 vCPUs.

• Superior read performance and scalability: Built on Google’s Colossus, a distributed regional storage service designed for exceptional durability and near-unlimited scalability, AlloyDB offers read pools, which enable massive horizontal scaling for read-intensive workloads. Scale effortlessly to thousands of vCPUs and handle immense concurrent read traffic without impacting transactional performance. Regional storage enables  instant provisioning of read pools without data copying, and with improvements in the replication logic allows AlloyDB to achieve over 25x lower replication lag compared to standard PostgreSQL.

• High availability and disaster recovery: AlloyDB provides high availability through instance redundancy, data replication, and automated failover mechanisms. Within a region, when configured in high availability, it maintains a standby in another zone, with automatic promotion if the primary instance fails. For disaster recovery, AlloyDB supports cross-regional replicas, which allow data to be available closer to your application’s region, helping to keep your data available and protected.

• Analytics Accelerator: AlloyDB has a built-in, in-memory analytics accelerator that provides real-time analytics on your operational data. 

• Auto-pilot features: AlloyDB has a number of optimizations to help with automatic memory management, adaptive autovacuum which is built on top of PostgreSQL’s autovacuum, and index advisor to help accelerate query performance.

Staying true to PostgreSQL, while pushing the boundaries

Choosing to build on PostgreSQL gave us a head start with AlloyDB, providing a mature, feature-rich database engine and a vast ecosystem of tools and extensions. This allowed the AlloyDB team to focus on developing unique, cloud-first optimizations, including its groundbreaking generative AI capabilities. AlloyDB includes innovations that directly address enterprises’ core performance, scalability, and analytical challenges, while developers benefit from PostgreSQL’s broad data-type support, advanced features (like full-text search and geospatial functions), and the flexibility of open standards to avoid vendor lock-in.

Two years of groundbreaking achievements

Since the AlloyDB launch, we’ve consistently pushed the boundaries of what’s possible with a PostgreSQL-compatible database. We’ve concentrated on strengthening the core platform, resulting in over 70 enterprise feature releases driven by direct customer feedback. Key highlights include:

1. Powerful generative AI Capabilities: AlloyDB is transforming how you interact with your data. Drawing on Google’s decades of investment in AI, we’ve added powerful gen AI capabilities, making it possible to build sophisticated AI applications without ever leaving your database. Imagine being able to easily generate natural language summaries of complex datasets, or build intelligent chatbots that understand your data. Additionally, AlloyDB’s ScaNN index (Scalable Nearest Neighbors) delivers lightning-fast vector similarity search — crucial for modern AI applications such as recommendation engines, image and video search, and anomaly detection. 

2. Top-tier performance and scalability: AlloyDB is significantly faster than standard PostgreSQL, delivering up to 4x faster transactional processing and up to 100x faster analytical queries. 

3. Run anywhere with AlloyDB Omni: Run the optimized AlloyDB database anywhere with AlloyDB Omni, a downloadable edition that you can run on-premises, in other clouds, or even on your laptop. Through a partnership with Aiven, a leading provider of managed open-source data platforms, AlloyDB Omni is available as a managed service on Google Cloud, AWS, and Azure, reinforcing our commitment to a flexible, multi-cloud strategy.

A growing ecosystem and industry recognition

• Real-world Successes: AlloyDB’s growing customer base spans e-commerce, fintech, and manufacturing. Major enterprises in the retail vertical, financial powerhouses across the globe, and technology innovators all trust AlloyDB. Ocean Network Express (ONE), a major player in global container shipping, consolidated their database fleet on AlloyDB, realizing an estimated 50% reduction in database licensing costs, and SEEBURGER’s Business Integration Suite (BIS) runs on AlloyDB to power its critical business operations. 

• Industry recognition: AlloyDB was recognized as a Leader in the Forrester Wave™: Translytical Data Platforms, Q4 2024, highlighting its strengths in accelerating the convergence of transactional and analytical workloads. The Forrester Wave cited Google Cloud’s unified platform approach, its ability to handle complex data operations, and its strong vision for the future of data management. 

What’s next: Deep dives into AlloyDB’s differentiators

This blog post marks the beginning of an exciting new series where we’ll explore the core features that make AlloyDB stand out. In the coming weeks, we’ll publish dedicated posts on:

• Scalability: How AlloyDB handles massive datasets and growing workloads, helping to ensure your database keeps pace with your business needs.

• Performance: A closer look at the architectural innovations that deliver the speed you need for both transactional and analytical workloads.

• Analytical capabilities: Uncovering how AlloyDB empowers you to gain deeper insights from your data with advanced analytics and reporting.

• Vector capabilities and generative AI: Building AI agents with AlloyDB, leveraging its built-in ScaNN vector index for efficient similarity search. We’ll also explore how to create RAG applications using frameworks like LangChain and LlamaIndex, and how the Gen AI Toolbox for Databases can streamline these integrations.

Join the AlloyDB revolution!

Ready for a database that can handle your modern enterprise needs? Discover AlloyDB with a 30-day free trial. AlloyDB free trial clusters make it easy to get started.

Learn more:

• AlloyDB Product Page

• AlloyDB Omni

• Google is a leader in The Forrester Wave: Translytical Data Platforms, Q4 2024

• Aiven for AlloyDB Omni

• SEEBURGER Runs its Business Integration Suite on AlloyDB

Today, I’m pleased to announce the launch of AI Luminaries programming at the upcoming Google Cloud Next conference. This is a unique forum where some of the top researchers, scientists, and technology leaders in the world will engage with Google experts across a variety of critical AI domains, including research, infrastructure, distributed systems, and AI hardware.

The AI Luminaries content is a curated selection of talks and discussions that offer unparalleled access to Google’s leading AI innovators. Our customers and partners will be able to engage with a diverse range of content, including focused breakout sessions, interactive panels with AMAs, presentations on significant research papers, and networking opportunities within the Google Cloud Showcase.

AI Luminaries sessions 

These speakers represent not only academic thought leadership, but engineering execution and strategic vision, offering a unique glimpse into the next era of technological advancement. 

Building the future of AI: Intelligent infrastructure and sustainable energy: The explosive growth of artificial intelligence demands a paradigm shift in how we build and power its infrastructure. Join Urs Hölzle, Google Fellow, and Parthasarathy Ranganathan, VP, Engineering Fellow at Google Cloud, for a fireside chat exploring the critical intersection of intelligent infrastructure and sustainable energy. They will delve into the challenges and opportunities of scaling AI, from the hardware powering massive models to the energy sources that fuel them. Discover how innovative approaches to infrastructure design and energy efficiency are shaping the future of AI, 25 years of cloud infrastructure design at Google, and what is coming for the next 25 years.

Google Cloud TPUs and specialized AI hardware: Jeff Dean on what’s next: Join an insightful fireside chat with Jeff Dean, a pioneering force behind Google’s AI leadership and Sabastian Mugazambi, Senior Product Manager, Cloud AI Infrastructure. As Google’s Chief Scientist at DeepMind & Research, Jeff will share his vision on AI and specialized AI hardware like Google Cloud TPUs. What exciting things might we expect to see next? What drives Google’s innovation in specialized AI hardware? We’ll discuss how TPUs enable efficient large-scale training and optimal inference workloads including exclusive, never-before-revealed details of the latest Gen TPU, differentiated chip design, data center infrastructure and software stack co-design that makes TPUs the most compelling choice for AI workloads.

Isomorphic Labs: Solving disease with AI Isomorphic Labs has created an AI drug design engine that reimagined traditional drug discovery approaches and allowed the company to go after some of the hardest disease targets. IsoLabs’ frontier AI research has yielded models that can accurately predict properties of proteins and other molecules and allow chemists to rationally design the next generation of medicines in-silico, in record time. Featuring Isomorphic Labs Chief Technology Officer, Sergei Yakneen. 

In my session, Now in focus: The fifth, gen AI epoch computing infrastructure, I will share my thoughts on the future of AI infrastructure, exploring the design principles behind scale-out architectures and the shift towards specialized hardware, high-bandwidth networks, liquid cooling and advanced memory technologies necessitated by Generative AI. 

Join us at Google Cloud Next

By sharing our knowledge and expertise in these sessions, we aim to foster a deeper understanding of the technologies we are building for the future of AI and inspire further innovation across the industry. We invite you to join us at Google Cloud Next to witness the next era of AI development.

Protecting your data in the cloud is more critical than ever. As your Google Cloud deployments grow, managing your data protection strategy for Compute Engine workloads can become complex. That’s why we’re excited to announce a streamlined experience that simplifies data protection with powerful capabilities that are purpose-built for Persistent Disk, Hyperdisk, and Backup and DR Service.

Introducing the Data protection tab

Say hello to the new Data protection tab in the Compute Engine VM creation experience. 

This new tab brings together backup and continuous data replication options for block storage in a single unified interface, making it possible to configure your data protection settings when creating a VM in the Google Cloud console, saving you time and effort, and allowing you to focus on what matters most: building and deploying your applications.

Selecting backup plans and snapshot schedules

Snapshot schedules for Persistent Disk and Hyperdisk provide foundational data protection for Compute Engine workloads, including basic scheduling automation at no charge. Snapshot schedules are a simple, cost-effective way to back up VM disks into the region or multi-region of your choice.

For more comprehensive protection, Backup and DR provides immutable backup vaults, securing data from accidental or malicious deletion, ransomware attacks, and project- or organization-level threats. These vaults help ensure that backups remain tamper-proof and resilient against unauthorized changes.

Beyond immutability, Backup and DR offers advanced backup management features, including centralized monitoring, detailed reporting, complex scheduling rules, and compliance-focused capabilities. Designed for teams with sophisticated data protection needs, Backup and DR supports centralized backup management across multiple cloud services (currently Compute Engine and Google Cloud VMware Engine) and includes a separate management pricing model.

Configure continuous data replication 

In addition to selecting snapshots or backup plans, you can choose to protect your data with continuous replication from the new Data protection tab. All capabilities for backup, cross-zone replication, and cross-region replication are designed to work together for layered protection against different types of risk. Continuous replication helps ensure minimal data loss in case of a disaster, complementing the historical recovery points provided by backups.

While some workloads such as test environments may not need any protection at all, most production workloads need protection from temporary outages, disasters, user errors, and malware. For mission-critical workloads that require zero or near-zero recovery point objectives (RPOs) for high availability and disaster recovery, you can choose continuous data replication to another point of presence. In the Data protection tab, this is as simple as selecting a secondary zone or region.  

How it works

When creating a new VM through the console, simply navigate to the Data protection tab.In a project where Backup and DR has been enabled, a default backup plan will be pre-selected; if not, a snapshot schedule will be applied. 

We sometimes see customers lose important data because they haven’t configured any protection for their workloads. Using backup protection for production VMs is a valuable way to minimize the risk of data loss. For test environments, setting up backup protection can help you verify that your restore processes will work once deployed. You can always customize and fully control your backup protection:

• Choose between snapshot schedules and backup plans. Select the option that best meets your needs and budget. You can select either a pre-defined snapshot schedule or backup plan.

• For workloads that need synchronous cross-zone replication (with RPO=0 between zones), you can choose to set your Persistent Disks to be Regional Persistent Disks, or your Hyperdisk Balanced disks to be Hyperdisk Balanced High Availability disks.

• For workloads that need cross-region replication (with RPO < 1 minute between regions), select Asynchronous Replication and choose your secondary region. Replication will be started automatically. 

• You can opt out of using the default data protection options by simply selecting “No Backups” when configuring the VM or by modifying the Compute Engine Project settings. Learn more. 

• Data protection tab selections only affect the choices for the VM being created in the console. To configure VMs at scale, you can configure your data protection options with the API, gCloud, or Terraform.

Get started today

Protect your VMs with a comprehensive data protection strategy using snapshot schedules or backup plans, along with continuous data replication to secondary zones and regions. Ensure your data is safeguarded and manage your data protection needs in one place.

The new Data protection tab is available now in the Compute Engine VM creation experience. Try it out today and streamline your data protection strategy! Learn more here.

At Google Cloud, we’re thrilled to return to NVIDIA’s GTC AI Conference in San Jose CA this March 17-21 with our largest presence ever. The annual conference brings together thousands of developers, innovators, and business leaders to experience how AI and accelerated computing are helping humanity solve the most complex challenges. Join us to discover how to build and deploy AI with optimized training and inference, apply AI with real-world solutions, and experience AI with our interactive demos.

After being the first hyperscaler to make both NVIDIA’s HGX B200 and GB200 NVL72 available to customers with A4 and A4X VMs. We’re are pleased to announce that A4 VMs are generally available, and that A4X VMs are in preview with general availability coming soon.

• A4X VMs: Accelerated by NVIDIA GB200 NVL72 GPUs, A4X VMs are purpose-built for training and serving the most demanding, extra-large-scale AI workloads — particularly those involving reasoning models, large language models (LLMs) with long context windows, and scenarios that require massive concurrency. This is enabled by unified memory across a large GPU domain and ultra-low-latency GPU-to-GPU connectivity. Each A4X VM contains 4 GPUs, and an entire 72 GPU system is connected via fifth-generation NVLink to deliver 720 petaflops of performance (FP8). A4X has achieved 860,000 tokens/sec of inference performance on a full NVL72 running Llama 2 70b

• A4 VMs: Built on NVIDIA HGX B200 GPUs, the A4 VM provides excellent performance and versatility for diverse AI model architectures and workloads, including training, fine-tuning, and serving. Each A4 VM contains eight GPUs for a total of 72 petaflops of performance (FP8). A4 offers easy portability from prior generations of Cloud GPUs. This enables an easy upgrade to 2.2x increase in training performance over A3 Mega (NVIDIA H100 GPU).

“We’re excited that we were among the first to test A4 VMs, powered by NVIDIA Blackwell GPUs and Google Cloud’s AI Hypercomputer architecture. The sheer compute and memory advancements, combined with the 3.2 Tbps GPU-to-GPU interconnect via NVLink and the Titanium ML network adapter, are critical for us to train our models. Leveraging the Cluster Director simplifies the deployment and management of our large-scale training workloads. This gives our researchers the speed and flexibility to experiment, iterate, and refine trading models more efficiently.” – Gerard Bernabeu Altayo, Compute Lead, Hudson River Trading

The Google Cloud advantage 

A4 and A4X VMs are part of Google Cloud’s AI Hypercomputer, our supercomputing architecture designed for high performance, reliability, and efficiency for AI workloads. AI Hypercomputer brings together Google Cloud’s workload-optimized hardware, open software, and flexible consumption models to help simplify deployments, improve performance, and optimize costs. A4 and A4X VMs benefit from the following AI Hypercomputer capabilities:

• AI-optimized architecture: A4 and A4X VMs are built on servers with our Titanium ML network adapter, which builds on NVIDIA ConnectX-7 network interface cards (NICs) to deliver a secure, high-performance cloud experience for AI workloads. Combined with our datacenter-wide 4-way rail-aligned network, A4 VMs deliver non-blocking 3.2 Tbps of GPU-to-GPU traffic with RDMA over Converged Ethernet (RoCE). You can scale to tens of thousands of NVIDIA Blackwell GPUs with our Jupiter network fabric with 13 Petabits/sec of bi-sectional bandwidth.
• Simplified deployment with pre-built solutions: For large training workloads, Cluster Director offers dense co-location of accelerator resources, to help ensure host machines are allocated physically close to one another, provisioned as blocks of resources, and interconnected with a dynamic ML network fabric that minimizes network hops and optimizes for the lowest latency.
• Scalable infrastructure: With support for up to 65,000 nodes per cluster, Google Kubernetes Engine (GKE) running on AI Hypercomputer is the most scalable Kubernetes service with which to implement a robust, production-ready AI platform. A4 and A4X VMs are natively integrated with GKE. And with integration to other Google Cloud services such as Hyperdisk ML for storage or BigQuery as a data warehouse, GKE facilitates data processing and distributed computing for AI workloads.
• Fully-integrated, open software: In addition to support for CUDA, we work closely with NVIDIA to optimize popular frameworks with XLA such as PyTorch and JAX (including the reference implementation, MaxText), enabling increased performance of GPU infrastructure. Developers can easily incorporate powerful techniques like a latency hiding scheduler to minimize communication overhead (see XLA optimizations).
• Flexible consumption models: In addition to the on-demand, committed use discount, and Spot consumption models, we reimagined cloud consumption for the unique needs of AI workloads with Dynamic Workload Scheduler, which offers two modes for different workloads: Flex Start mode for enhanced obtainability and better economics, and Calendar mode for predictable job start times and durations.  Dynamic Workload Scheduler improves your access to AI accelerator resources, helps you optimize your spend, and can improve the experience of workloads such as training and fine-tuning jobs, by scheduling all the accelerators needed simultaneously.

NVIDIA and Google Cloud: Better together 

We’re continuously working together to provide our joint customers with an optimized experience. One  of our recent collaborations includes bringing together software innovations to accelerate AI-driven drug discovery. Using the NVIDIA BioNeMo framework and blueprints on GKE,  and PyTorch Lightning, we’re providing ready-to-use reference workflows for domain specific tasks. The NVIDIA BioNeMo Framework provides an optimized environment for training and fine-tuning biomolecular AI models. Read more here. 

Meet Google Cloud 

To connect with Google Cloud, please visit us at booth #914 at NVIDIA GTC, join our expert-led sessions listed below, or email us to set up a private meeting. Whether it’s your first time speaking with Google Cloud or the first time connecting with us at NVIDIA GTC, we’re looking forward to meeting with you. 

Deep dive into AI at expert-led sessions

Join our expert-led sessions to gain in-depth knowledge and develop practical skills in AI development on Google.

Tuesday, March 18

Optimizing the Future of Ads with MoE Models
Time: 2:00 PM – 2:40 PM PDT
Speaker: Tris Warkentin, Director of Product Management, Google DeepMind

Wired for AI: Lessons from Networking 100K+ GPU AI Data Centers and Clouds
Time: 4:00 PM – 5:00 PM PDT
Speakers: Dan Lenoski, VP Networking, Google and more industry leaders

Wednesday, March 19

Accelerate AI: Enhance Performance and Efficiency Using Google Cloud
Time: 10:00 AM – 10:40 AM PDT
Speakers: Roy Kim, Director Google Cloud GPUs, and Scott Dietzen, CEO, Augment Code 

Optimize your Workloads for Rack-Scale Interconnected GPU systems
Time: 3:00 PM – 3:40 PM PDT 
Speakers: Jon Olson, Software Engineer, Google Cloud, and Pramod Ramarao, Product Manager, Google

Thursday, March 20

Unlock the Speed of Light for Data Science Workflows With Gemini Coding Assistant 
Time: 8:00 AM – 8:40 AM PDT
Speaker: Paige Bailey, ENG MGR, Developer Relations, Google

Build Next-Generation AI Factories With DOCA-Accelerated Networking
Time: 9:00 AM – 9:40 AM PDT
Speakers: Valas Valancius, Senior Staff Software Engineer, Google Cloud; Ariel Kit, Director, Product Management, NVIDIA; David Wetherall, Distinguished Engineer, Google Cloud

Physical AI for Humanoids: How Google Robotics Uses Simulation to Accelerate Humanoid Robotics Training
Time: 9:00 AM – 9:40 AM PDT
Speaker: Erik Frey, Lead Researcher, Google

Toward Rational Drug Design With AlphaFold 3
Time: 10:00 AM – 10:40 AM PDT
Speakers: Max Jaderberg, Chief AI Officer, Isomorphic Labs (DeepMind) and Sergei Yakneen, Chief Technology Officer, Isomorphic Labs (DeepMind)

AI in Action: Optimize Your AI Infrastructure
Time: 11:00 AM – 11:40 AM PDT
Speakers: Chelsie Czop, Senior Product Manager, Google Cloud; Kshetrajna Raghavan, Machine Learning Engineer, Shopify; Ashwin Kannan, Principal Machine Learning Engineer, Palo Alto Networks; Jia Li, Chief AI Officer, Livex.AI

Horizontal Scaling of LLM Training with JAX 
Time: 2:00 PM – 2:40 PDT
Speakers: Andi Gavrilescu, Sr. Engineering Manager, Google; Matthew Johnson, Research Scientist, Google
Abhinav Goel, Senior Deep Learning Architect, Google

On-Demand, Virtual Sessions

S74318: Deploy AI and HPC on NVIDIA GPUs With Google
Speakers: Annie Ma-Weaver, HPC  group Product Manager, Google Cloud; Wyatt Gorman, HPC and AI Solutions Manager, Google Cloud; Sam Skillman, HPC Software Engineer, Google Cloud

S74319: Supercharge Large-Scale AI with Google Cloud AI Hypercomputer
Speakers: Rajesh Anantharaman, Product Management Lead, ML Software, Google Cloud  and Deepak Patil, Product Manager, Google Cloud

In addition to our expert-led sessions at NVIDIA GTC, we invite you to join us at the following events onsite (limited space available):

• Executive Roundtable, Wednesday, March 19 at 8 AM

• DGX Cloud on Google Cloud Roundtable, Thursday, March 20 at 8 AM

• Developer Hands On Lab, Thursday, March 20 at 10 AM

The quest to develop new medical treatments has historically been a slow, arduous process, screening billions of molecular compounds across decade-long development cycles. The vast majority of therapeutic candidates do not even make it out of clinical trials. 

Now, AI is poised to dramatically accelerate this timeline. 

As part of our wide-ranging, cross-industry collaboration, NVIDIA and Google Cloud have supported the development of generative AI applications and platforms. NVIDIA BioNeMo is a powerful open-source collection of models specifically tuned to the needs of medical and pharmaceutical researchers.

Medical and biopharma organizations of all sizes are looking closely at predictive modeling and AI foundation models to help disrupt this space. With AI, they’re working on accelerating the identification and optimization of potential drug candidates to significantly shorten development timelines and address unmet medical needs. This has become a significant turning point for analyzing DNA, RNA, and protein sequences, and chemicals, predicting molecular interactions, and designing novel therapeutics at scale. 

With BioNeMo, companies in this space gain a more data-driven approach to developing medicines while reducing reliance on time-consuming experimental methods. But these breakthroughs are not without their own challenges. The shift to generative medicine requires a robust tech stack, including: powerful infrastructure to build, scale, and customize models; efficient resource utilization; agility for faster iteration; fault tolerance; and orchestration of distributed workloads.

Google Kubernetes Engine (GKE) offers a powerful solution for achieving many of these demanding workloads, and when taken together with NVIDIA BioNeMo, GKE can accelerate work on the platform. With BioNeMo running on GKE, organizations can achieve medical breakthroughs and new research with levels of speed and effectiveness that were unheard of before. 

In this blog, we’ll show you how to build and customize models and launch reference blueprints using  NVIDIA BioNeMo platform on GKE

NVIDIA’s BioNeMo platform on GKE 

NVIDIA BioNeMo is a generative AI framework that enables researchers to model and simulate biological sequences and structures. It places major demands for computing with powerful GPUs, scalable infrastructure for handling large datasets and complex models, and robust managed services for storage, networking, and security. 

GKE offers a highly scalable and flexible platform ideal for AI and machine learning — and particularly the demanding workloads found in biopharma research and development. GKE’s autoscaling features ensure efficient resource utilization, while its integration with other Google Cloud services simplifies the AI workflow. 

NVIDIA’s BioNeMo platform offers two potential synergistic components:

1. BioNeMo Framework: Large-Scale Training Platform for Drug Discovery AI

A scalable, open-source, training system for biomolecular AI models like ESM-2 and Evo2. It provides an optimized environment for training and fine-tuning biomolecular AI models. Built on NVIDIA NeMo and PyTorch Lightning, it offers:

• Domain-Specific Optimization: Provides performant biomolecular AI architectures that can be scaled to billions of parameters (eg: BERT, Striped Hyena) along with representative model examples (e.g., ESM-2, Geneformer) built with CUDA-accelerated tooling tailored for drug discovery workflows.
• GPU-accelerated performance: Delivers industry-leading speed through native integration with NVIDIA GPUs at scale, reducing training time for large language models and predictive models.
• Comprehensive open-source resources: Includes programming tools, libraries, prepackaged datasets, and detailed documentation to support researchers and developers in deploying biomolecular AI solutions

Explore the preprint here for details.

2. BioNeMo Blueprints: Production Ready Workflows for Drug Discovery

BioNeMo Blueprints provide ready-to-use reference workflows for tasks such as protein binder design, virtual screening, and molecular docking. These workflows integrate advanced AI models like AlphaFold2, DiffDock 2.0, RFdiffusion, MolMIM, and ProteinMPNN to accelerate drug discovery processes. These blueprints provide solutions to patterns identified across several other industry use cases. Scientific developers can try NVIDIA inference microservices (NIMs) at build.nvidia.com and access them to test via a NVIDIA developer license.

The following graphic shows the components and features of GKE used by the BioNeMo platform. In this blog, we demonstrate how to deploy these components on GKE, combining NVIDIA’s domain-specific AI tools with Google Cloud’s managed Kubernetes infrastructure for:

• Distributed pretraining and finetuning of models across NVIDIA GPU clusters
• Blueprint-driven workflows using NIMs
• Cost-optimized scaling via GKE’s dynamic node pools and preemptible VMs

Solution Architecture of BioNeMo framework

Here, we will walk through setting up the BioNeMo framework on GKE to perform ESM2 pretraining and fine-tuning.

The above diagram shows an architectural overview of deploying the NVIDIA BioNeMo Framework on GKE for AI model pre-training, fine-tuning, and inferencing. Here’s a breakdown from an architectural perspective:

• GKE: The core orchestration platform including the control plane managing the deployment and scaling of the BioNeMo Framework. This is deployed as a regional cluster, and can be optionally configured as a zonal cluster.

• Node Pool: A group of worker nodes within the GKE cluster, specifically configured with NVIDIA GPUs for accelerated AI workloads.

• Nodes: Individual machines within the node pool, equipped with NVIDIA GPUs.

• NVIDIA BioNeMo Framework: The AI software platform running within GKE, enabling pre-training, fine-tuning, and inferencing of AI models.

• Virtual Private Cloud (VPC): A logically isolated network within GCP, ensuring secure communication between resources.

• Load Balancer: Distributes incoming traffic to the BioNeMo services running in the GKE cluster, enhancing availability and scalability.

• Filestore (NFS): Provides high-performance network file storage for datasets and model checkpoints.

• Cloud Storage: Object storage for storing datasets and other large files.

Steps

We have published an example to pre-train, fine-tune, and infer an ESM-2 model using BioNeMo Framework on GKE in Pretraining and Fine-tuning ESM-2 LLM on GKE using BioNeMo Framework 2.0 GitHub repo. Here is an outline of the steps for pretraining:

1. Create a GKE cluster

2. Add node pool with NVIDIA GPUs

3. Mount Google Cloud Filestore across all the nodes

4. Run the pretraining job

5. Visualize results in TensorBoard

Open a web browser pointing to http://localhost:8000/#timeseries to see the loss curves. The details for fine-tuning and inference are laid out in the GitHub repo.

Solution Architecture of BioNeMo Blueprints

The below graphic shows a BioNeMo Blueprint that is deployed on GKE for inferencing. From an infrastructure standpoint, the components used across the Compute, Networking and Storage layer are similar to Figure 2:

• NIMs are packaged as a unit with runtime and model-specific weights. Blueprints deploy one or more NIMs using Helm charts. Alternatively, they can be deployed using gcloud or docker commands and configured using kubectl commands. Each NIM needs a minimum of one NVIDIA GPU accessible through a GKE node pool. 

• Three NIMs—AlphaFold2, DiffDock, and MolMIM—are deployed as individual Kubernetes deployments. Each deployment uses a GPU and a NIM container image, mounting a persistent volume claim for storing model checkpoints and data. Services expose each application on different ports. The number of GPUs can be configured to a higher value for better scalability.

Steps

We have an example of deploying a BioNeMo blueprint for Generative Virtual Screening at Generative Virtual Screening for Drug Discovery on GKE GitHub repo. The setup steps, such as GKE cluster, node pool, and mounting filestore, are similar to BioNeMo training. The below steps give an outline of deploying the BioNeMo blueprint and using it for inference:

1. Deploy the BioNeMo blueprint

2. Use port forwarding to interact with the pod

3. Test MolMIM NIM locally using a curl statement. The output will have the generated molecule.

NVIDIA BioNeMo Blueprints workflows can be adapted to various domain-specific use cases beyond drug discovery. For example, researchers can leverage generative AI models like RFdiffusion and ProteinMPNN in protein engineering to design stable protein binders with high affinity, drastically reducing the experimental iteration cycles. 

By integrating modular NIM microservices with scalable platforms like GKE, industries ranging from biopharma to agriculture can deploy AI-driven solutions tailored to their unique challenges, enabling faster insights and more efficient processes at scale.

Conclusion

As we’ve explored in this blog post, GKE provides a robust and versatile platform for deploying and running both NVIDIA BioNeMo Framework and NVIDIA BioNeMo  Blueprint. By leveraging GKE’s scalability, container orchestration capabilities, and integration with Google Cloud’s ecosystem, you can streamline the development and deployment of AI solutions in the life sciences and other domains. 

Whether you’re accelerating drug discovery with BioNeMo or deploying generative AI models with NIMs, GKE empowers you to harness the power of AI and drive innovation. By leveraging the strengths of both platforms, you can streamline the deployment process, optimize performance, and scale your AI workloads seamlessly. 

Ready to experience the power of NVIDIA BioNeMo on Google Cloud? Get started today by exploring the BioNeMo Framework and NIM catalog, deploying your first generative AI model on GKE, and unlocking new possibilities for your applications.

^{We’d like to thank the NVIDIA team members who helped contribute to this guide, Juan Pablo Guerra, Solutions Architect, and Kushal Shah, Senior Solutions Architect.}

Recently, we announced Gemini Code Assist for individuals, a free version of our AI coding assistant. Technology that was previously available only to the biggest enterprises is now within reach for startups and individual developers. The same applies to the AI/ML related infrastructure like powerful GPUs, specialized TPUs, or extremely efficient storage solutions.

But despite moving towards greater accessibility, enterprises need ways to optimize large AI workloads because these resources can still be quite expensive. It’s always a good idea to look for ways to optimize your workloads to reduce costs. In this article, we’ll share five tips to optimize your workflow on Google Cloud Platform.

Disclaimer: Not every idea fits every use case. These are not official recommendations.

1. Research different platforms to run your training/inference

A few years ago, if you wanted to train, tune, or use any AI model, you had to manually set up a cluster of GPU or TPU powered machines, orchestrate the whole training pipeline, and carefully manage all the resources consumed by this process. You could make it a bit easier by using Kubernetes, but apart from that, there were no widely available services that would make your AI work easier. This generated additional costs, not only did you have to pay for the hardware you used, but also for the time it took to manage all this infrastructure. 

Luckily, this is no longer the case. Google Cloud offers a wide range of solutions that can help you run your jobs, from fully-managed to fully-customizable, just take your pick. Here’s a quick summary:

• Vertex AI,  a fully-managed, unified AI development platform. Whether it’s training, tuning or inferencing, all this can be done through a simple web interface. You can save a lot of man-hours and many headaches, by letting Google manage all the necessary infrastructure your workloads require. Additionally, you will only pay for what you use – no more cost generated by idle GPUs waiting for the next task.
• In Cloud Run, there is now an option to run your containers on GPU equipped machines. This is a great way to set up a scalable, fully managed inferencing service, without the need of learning new platforms.
• Cloud Batch also offers access to GPUs – this is a great option for long-running tasks like training or tuning of your AI models. Cloud Batch will take care of provisioning all required infrastructure, retrying jobs that encountered an error and releasing the used resources once the job is done. The auto-retry feature combined with Spot Instances can result in a significant reduction of cost for your workloads.
• Google Kubernetes Engine (GKE) allows you to control your infrastructure, while handling all the provisioning, setting up and managing. It’s a great solution for organizations that are already using Kubernetes and have the experience required to fully benefit from the control it provides.
• Google Compute Engine (GCE) is on the opposite side of the “hands-off scale” than Vertex AI. With direct access to the GPU or TPU equipped Virtual Machines, you gain full control over every aspect of the workflow.

2. Improve the startup time of your inference containers

When working with GKE or Cloud Run, don’t store your models directly in your containers. Keep your containers lightweight and store your models externally using options like Cloud Storage (with FUSE),  Filestore or share a read-only persistent disks.

Why? Because bulky containers with embedded models take a long time to scale up. Nodes have to download these massive images before they can even start running. Plus, it puts a strain on node storage, which isn’t optimized for high-throughput. By mounting models externally, you separate them from the container, making everything faster and smoother. Your containers start up quicker, scaling is a breeze, and you avoid bottlenecks on your nodes.

Remember, containers are meant to be temporary and nimble, holding just the essentials. Models are big and need long-term storage. So keep them separate and use external storage to build a fast and efficient auto scaling deployment.

In case you are not able to modify the containers you are working with, GKE offers an option to use a secondary boot disk to speed up the startup process for new nodes. Nodes with secondary boot disk, start up with additional disk that already has some of the container images preloaded. Since we’re talking about startup times, I suggest having a look at the Image streaming feature of GKE, which can make every workload start faster, not only AI related.

3. Storage is not as simple as it seems

Machine Learning usually needs a lot of data, hundreds of TBs to PBs, to produce valuable results – especially when it deals with non-text data and multimodal models. Maximizing GPU/TPU utilization (goodput) during training, checkpointing, and serving is critical and often not a trivial task. For smaller AI workload requirements, with a handful of nodes and TBs of data, Filestore is a good ‘simple’ NFS solution. 

For companies that have written their AI workloads to consume object storage, Cloud Storage is a fully managed object storage service suitable for AI and ML workloads of any scale. However, many companies’ AI workloads require a file system interface. Cloud Storage FUSE lets you mount Cloud Storage buckets as a file system. Cloud Storage FUSE is not fully Portable Operating System Interface (POSIX)-compliant; therefore, understanding its limitations and differences from traditional file systems is crucial. With Cloud Storage FUSE, you can access your training data, models, and checkpoints with the scale, affordability, and performance of Cloud Storage. 

For workloads that require lower latency and are working with smaller files, Parallelstore is a fully managed, scratch parallel file system in Google Cloud and is ideal for AI and ML workloads that need to provide low-latency (sub-millisecond) access with high throughput and high input/output operations per second (IOPS). The Parallestore reference architecture is here.

Depending on your serving needs, Hyperdisk ML is a high-performance storage solution and particularly well-suited for serving tasks, providing exceptionally high aggregate throughput (~1TB/s) up to 2500 virtual machines concurrently. 

4. Use DWS and/or future reservations to get your resources

Problems with acquiring hardware to run your big jobs cost money. You line up all the data, ready to start your work and it turns out, there are not enough GPUs available. This surprise might not generate cost in any obvious way, but the fact that you now need to adjust your schedule and “hunt” for required resources is slowing you down and time is money. Using Dynamic Workload Scheduler and Future Reservations is a way to mitigate such problems. 

Future Reservations do exactly what the name suggests. They allow you to put a reservation on Cloud resources you plan to use in the future. Once a reservation is accepted by the system, you can stop worrying about the availability of GPUs you need. When the time comes, the system will provide you with the requested reservations for hardware in your projects. Once it happens, you can use those reservations however you want – the resources are available only to you, as long as the reservation exists. Keep in mind that you pay for those resources, no matter if they are utilized or not.

Dynamic Workload Scheduler (DWS) is a backend platform used by multiple Google Cloud products to make acquisition of popular hardware easier. With its Flex and Calendar mode, you can make sure that you won’t waste time or money trying to grab GPUs one-by-one as they are released by other Cloud customers. You can learn more about the DWS and how it’s utilized by various products from this video.

5. Use custom disk images to make setup faster

Running AI workloads on virtual machines requires a lot of setup. You need to have an up-to-date operating system, installed GPU drivers, installed and configured AI frameworks like JAX, Pytorch or Tensorflow. The full setup of a system like that, if started from a clean operating system image, might take even up to an hour, depending on the freshness of the OS image and your software choices. It would make sense to have to do all this setup only once, right?

This is where the power of custom disk images lies. You can configure your VM once, install all the necessary software, shut it down, create a disk image based on the disk of the configured VM and from now on, it takes only seconds to start new, fully configured workers for your needs. To make your life even easier, you can utilize image families and managed instance groups, to have Google Cloud automatically handle rolling-updates to your setups.

Get started

To keep up to date with everything that’s happening in Google Cloud, consider the following: 

• Subscribe to our YouTube channel (or the less technical-focused channel), 

• Follow our Blog (RSS), 

• Subscribe to our Newsletter

• Join the Google Cloud Innovators program

As AI and machine learning (ML) workloads continue to grow, the infrastructure supporting them must evolve to meet their unique demands. Here on the Google Cloud Storage team, we’re committed to providing AI/ML practitioners with tools to optimize the performance, scalability, and usability of Cloud Storage. In this post, we’ll explore how Cloud Storage’s new hierarchical namespace (HNS) capability can help you maximize the performance and efficiency of your AI/ML workloads. 

Storage’s role in AI/ML workloads

AI/ML data pipelines typically consist of the following steps, which can place heavy demands on the underlying storage system:

1. Data preparation and preprocessing involves data validation, preprocessing, ingesting data into storage and transforming it into the correct format for model training.

2. Model training is a process which uses many GPU/TPU compute instances to iteratively develop and refine an AI/ML model.

This process also involves checkpointing, which periodically saves the state of a model so it can be resumed from the last saved state instead of restarting from scratch, saving valuable time and resources. This provides fault tolerance against failures that are common in large-scale distributed training, and also helps developers experiment with hyperparameters or adjust training objectives without losing prior progress.

3. Model serving typically involves loading the model, weights, and dataset into compute instances with GPUs/TPUs for model inference.

AI/ML workloads can run on large compute clusters that consist of thousands of nodes performing simultaneous I/Os on petabyte-scale datasets. As such, the underlying storage system can often become the bottleneck for AI/ML pipelines, resulting in underutilization of expensive GPU/TPU cycles.

Benefits of using a hierarchical namespace for AI/ML workloads

Cloud Storage’s hierarchical namespace can be enabled when creating a bucket, and it provides several benefits to AI/ML workloads, including:

• A new “folder” resource type and APIs that are optimized for filesystem semantics.

• Atomic and fast folder renames, resulting in faster and more reliable checkpointing.

• An optimized storage layout that handles higher queries per second (QPS) of reads and writes.

Let’s examine these benefits in more detail.

Data organization and access that’s optimized for filesystem semantics

In a hierarchical namespace bucket, folders can contain objects and other folders, which allows organizing (traditionally flat) Cloud Storage data into a tree-like structure that mirrors a traditional filesystem. This lets client libraries like Cloud Storage FUSE map filesystem calls to Cloud Storage APIs that operate directly on folders. While flat namespace buckets often necessitate performing inefficient and costly object-level operations to simulate filesystem operations, using a hierarchical namespace lets you take advantage of filesystem semantics offered natively by the underlying storage system. For example, filesystem libraries typically use resource-intensive ListObject calls to implement inode lookups; these can be replaced with more efficient GetFolderMetadata calls when using a hierarchical namespace. AI/ML workloads benefit greatly as a result, as they often rely on frameworks like TensorFlow and PyTorch that interact with storage via a filesystem interface. 

Customers like AssemblyAI have reported significant improvements using hierarchical namespace with Cloud Storage FUSE to power their AI/ML workloads.

“With HNS and GCSfuse we observed over 10x increase in throughput from GCS, with training speed improving 15x.” – Ahmed Etefy, Staff Software Engineer, AssemblyAI

Up to 20x faster checkpointing

Renaming folders and objects is common when writing checkpoints or managing intermediate outputs. Cloud Storage’s hierarchical namespace buckets introduce a new RenameFolder API that is both fast and atomic. While simulating a folder rename in a flat namespace bucket could involve thousands of individual object rewrites and deletes (depending on how many objects are in the folder), the hierarchical namespace offering provides a folder-level metadata-only operation that accomplishes this in an atomic action that completes in a fraction of the time. Atomicity prevents inconsistencies and complex state management caused by partial failures, which is a common problem with simulated renames in flat buckets.

Looking at folder renames in action, checkpoint benchmarking shows that hierarchical namespace buckets speed up checkpoint writes by up to 20x compared to flat buckets.

Up to 8x higher QPS

AI/ML workloads running on large clusters generate millions of I/O requests on the attached storage system. Checkpoint writes and restores during model training and serving reads for inference are highly bursty workloads where many nodes are synchronized to talk to storage at the same time. High QPS capabilities help avoid storage bottlenecks that could starve expensive GPUs/TPUs.

Hierarchical namespace buckets have an optimized storage layout that provides up to 8x higher initial object read and write requests per second (QPS) compared to flat namespace buckets, while still supporting a doubling of the QPS every 20 minutes per the Cloud Storage ramp-up guidelines. For example, this means a cold hierarchical namespace bucket can achieve 100,000 object write QPS in nearly half the time compared to a flat bucket.

Conclusion

AI/ML workloads require infrastructure tailored to their unique needs: efficient data organization and filesystem semantics for tight integration with frameworks, high-performance checkpointing to maximize GPU/TPU utilization, and high QPS rates to support quick ramp up. Hierarchical namespace buckets provide all these benefits, along with the scalability, reliability, simplicity, and cost-effectiveness that Cloud Storage is known for. We recommend enabling hierarchical namespace on new buckets for AI/ML workloads.

As any data team can tell you, managing Apache Airflow is often a complex undertaking, with many hours spent wrestling with reliability, security, and efficient resource scaling. This valuable time could be better spent deriving insights and driving business value.

Today, we’re announcing the general availability of Cloud Composer 3, the latest version of our fully managed Apache Airflow service.This release represents a significant advancement in data pipeline orchestration, enabling data teams to streamline workflows, reduce operational overhead, and accelerate time-to-value.

Cloud Composer: A leading managed Airflow offering

Cloud Composer 3 empowers data teams to build, run, and manage data pipelines with greater efficiency and control. Its simplified user experience, enhanced performance, and robust security features make it a great platform for accelerating data-driven initiatives. Cloud Composer remains a top choice for many customers for data and analytics workflow orchestration and is widely used to orchestrate AI/ML workloads, supporting MLOps activities that are crucial for organizations in their AI efforts.

What’s new in Cloud Composer 3?

Cloud Composer 3 introduces a host of powerful features and improvements designed to simplify data pipeline management and boost efficiency:  

• Simplified networking: Easily configure network settings with streamlined options, reducing complexity and management overhead.  

• Evergreen versioning: Stay up-to-date with the latest Cloud Composer releases and gain access to new features, security updates, and performance improvements.  

• Hidden infrastructure: Focus on your data pipelines, not infrastructure management. Cloud Composer 3 handles the underlying infrastructure, allowing you to concentrate on building and running your DAGs.  

• Enhanced performance and reliability: Experience improved performance and reliability with optimized infrastructure and enhanced resource management.  

• Per-task CPU and memory control: Fine-tune resource allocation for individual tasks, optimizing performance and cost efficiency.  

• Strengthen your security posture: Cloud Composer 3’s hidden infrastructure lets Google proactively manage the environment’s security posture using industry best practices.

• and many more….

Benefits for data teams

Cloud Composer 3 delivers tangible benefits for data engineers, data scientists, and data architects:

• Increased productivity: Streamlined workflows and simplified environment management free up valuable time for data teams to focus on core tasks.  

• Reduced operational overhead: Automated infrastructure management and evergreen versioning minimize operational burden, allowing teams to focus on innovation.  

• Faster time-to-value: Enhanced performance and scalability accelerate data pipeline execution, enabling faster insights and quicker time-to-market for data-driven initiatives.  

Upgrade to Cloud Composer 3 today

Ready to experience the next generation of data pipeline orchestration? Visit our documentation for detailed instructions and join the many enterprise customers using Cloud Composer today.