gcp

The DORA research program has been investigating the capabilities, practices, and measures of high-performing technology-driven teams and organizations for more than a decade. It has published reports based on data collected from annual surveys of professionals working in technical roles, including software developers, managers, and senior executives.

Today, we’re pleased to announce the publication of the 2024 Accelerate State of DevOps Report, marking a decade of DORA’s investigation into high-performing technology teams and organizations. DORA’s four key metrics, introduced in 2013, have become the industry standard for measuring software delivery performance. 

Each year, we seek to gain a comprehensive understanding of standard DORA performance metrics, and how they intersect with individual, workflow, team, and product performance. We now include how AI adoption affects software development across multiple levels, too.

We also establish reference points each year to help teams understand how they are performing, relative to their peers, and to inspire teams with the knowledge that elite performance is possible in every industry. DORA’s research over the last decade has been designed to help teams get better at getting better: to strive to improve their improvements year over year. 

For a quick overview of this year’s report, you can read in our executive DORA Report summary the spotlight adoption trends and the impact of AI, the emergence of platform engineering, and the continuing significance of developer experience. 

Organizations across all industries are prioritizing the integration of AI into their applications and services. Developers are increasingly relying on AI to improve their productivity and fulfill their core responsibilities. This year’s research reveals a complex landscape of benefits and tradeoffs for AI adoption.

The report underscores the need to approach platform engineering thoughtfully, and emphasizes the critical role of developer experience in achieving high performance. 

AI: Benefits, challenges, and developing trust

Widespread AI adoption is reshaping software development practices. More than 75 percent of respondents said that they rely on AI for at least one daily professional responsibility. The most prevalent use cases include code writing, information summarization, and code explanation. 

The report confirms that AI is boosting productivity for many developers. More than one-third of respondents experienced”‘moderate” to “extreme” productivity increases due to AI.

A 25% increase in AI adoption is associated with improvements in several key areas:

7.5% increase in documentation quality

3.4% increase in code quality

3.1% increase in code review speed

However, despite AI’s potential benefits, our research revealed a critical finding: AI adoption may negatively impact software delivery performance. As AI adoption increased, it was accompanied by an estimated  decrease in delivery throughput by 1.5%, and an estimated reduction in delivery stability by 7.2%. Our data suggest that improving the development process does not automatically improve software delivery — at least not without proper adherence to the basics of successful software delivery, like small batch sizes and robust testing mechanisms. AI has positive impacts on many important individual and organizational factors which foster the conditions for high software delivery performance. But, AI does not appear to be a panacea.

Our research also shows that despite the productivity gains, 39% of the respondents reported little to no trust in AI-generated code. This unexpected low level of trust indicates to us that there is a need to manage AI integration more thoughtfully. Teams must carefully evaluate AI’s role in their development workflow to mitigate the downsides. 

Based on these findings, we have three core recommendations:

Enable your employees and reduce toil by orienting your AI adoption strategies towards empowering employees and alleviating the burden of undesirable tasks.

Establish clear guidelines for the use of AI and address procedural concerns and foster open communication about its impact.

Encourage continuous exploration of AI tools and provide them dedicated time for experimentation, and promote trust through hands-on experience.

Platform engineering: A paradigm shift

Another emerging discipline our research focused this year is on platform engineering. Its focus is on building and operating internal development platforms to streamline processes and enhance efficiency

Our research identified 4 key findings regarding platform engineering:

Increased developer productivity: Internal development platforms effectively increase productivity for developers.

Prevalence in larger firms: These platforms are more commonly found in larger organizations, suggesting their suitability for managing complex development environments.

Potential performance dip: Implementing a platform engineering initiative might lead to a temporary decrease in performance before improvements manifest as the platform matures.

Need for user-centeredness and developer independence: For optimal results, platform engineering efforts should prioritize user-centered design, developer independence, and a product-oriented approach

A thoughtful approach that prioritizes user needs, empowers developers, and anticipates potential challenges is key to maximizing the benefits of platform engineering initiatives. 

Developer experience: The cornerstone of success

One of the key insights in last year’s report was that a healthy culture can help reduce burnout, increase productivity, and increase job satisfaction. This year was no different. Teams that cultivate a stable and supportive environment that empowers developers to excel drive positive outcomes. 

Move fast and constantly pivot’ mentality negatively impacts developer well-being and consequently, on overall performance.  Instability in priorities, even with strong leadership, comprehensive documentation, and a user-centered approach — all known to be highly beneficial — can significantly hinder progress. 

Creating a work environment where your team feels supported, valued, and empowered to contribute is fundamental to achieving high performance. 

How to use these findings to help your DevOps team

The key takeaway from the decade of research is that software development success hinges not just on technical prowess but also on fostering a supportive culture, prioritizing user needs, and focusing on developer experience. We encourage teams to replicate our findings within your specific context.  

It can be used as a hypothesis for your experiments and continuous improvement initiatives. Please share those with us and the DORA community, so that your efforts can become part of our collaborative learning environment.  

We work on this research in hopes that it serves as a roadmap for teams and organizations seeking to improve their practices and create a thriving environment for innovation, collaboration, and business success. We will continue our platform-agnostic research that focuses on the human aspect of technology for the next decade to come.

To learn more:

Download the full report

Share your experiences, learn from others, and get inspiration by joining the DORA community.

Measure your team’s software delivery performance in less than a minute using DORA’s Quick Check.

At Google Cloud, we’ve taken an open approach in building our Vertex AI platform — to provide the most powerful AI tools available along with unparalleled choice and flexibility. That’s why Vertex AI delivers access to over 160 models — including first-party, open-source, and third-party models — so you can build solutions specifically tailored to your needs.

In June, we announced the addition of Anthropic’s Claude 3.5 Sonnet to Vertex AI Model Garden. Today, we’re announcing the general availability of the upgraded Claude 3.5 Sonnet for US customers, featuring a new “computer use” capability in public beta, on Vertex AI Model Garden. This means you can use the upgraded Claude 3.5 Sonnet to direct the model to generate computer actions — like keystrokes and mouse clicks — allowing it to interact with your user interface (UI). You can access the upgraded Claude 3.5 Sonnet model through our Model-as-a-Service (MaaS) offering. 

Additionally, Claude 3.5 Haiku, Anthropic’s fastest, most cost-effective model, will also be available on Vertex AI in the coming weeks. For more details on the upgraded Claude 3.5 Sonnet and Claude 3.5 Haiku, check out Anthropic’s blog.

Why use the upgraded Claude 3.5 Sonnet model on Vertex AI?

As a unified AI platform, Vertex AI simplifies the process of experimenting with, deploying, and managing models like the upgraded Claude 3.5 Sonnet. 

Combining the intelligence of Claude 3.5 models with the power of Vertex AI, you can:

Experiment with confidence: The upgraded Claude 3.5 Sonnet is delivered as a fully managed Model-as-a-Service on Vertex AI. With MaaS, you can explore Claude 3.5 models through simple API calls and perform comprehensive evaluations in an intuitive environment, without worrying about complex deployment processes. 
Get started and manage without overhead: Simplify how you deploy and scale Claude 3.5 models with fully managed infrastructure designed for AI workloads and the option to select provisioned throughput or pay-as-you-go pricing.
Build sophisticated AI agents: Develop agents with Vertex AI’s tools and the unique capabilities of the upgraded Claude 3.5 Sonnet.
Build with enterprise-grade security and compliance: Leverage Google Cloud’s built-in security, privacy, and compliance measures. Enterprise controls, such as Vertex AI Model Garden’s new organization policy, provide the right access controls to make sure only approved models can be accessed.

The addition of the Claude 3.5 models to Vertex AI expands our curated collection of over 160 enterprise-ready first-party, open-source, and third-party models in Model Garden — enabling you to select the best models for your needs through our open and flexible AI ecosystem.

Customers building with Anthropic’s Claude models on Vertex AI 

AES, a global energy company, leverages Claude on Vertex AI to streamline energy safety audits, significantly reducing time spent on this critical but time-consuming task:

“What used to take ten hours for an auditor now takes three minutes with our agents powered by Claude on Vertex AI. I love the accuracy of Anthropic’s Claude models and the security and advanced AI tools that Google Cloud provides to utilize these models for our auditing process.”
— Sean Otto, Director of Analytics at AES

BrainLogic AI, a startup building gen AI products specifically designed for Latin America, uses Claude on Vertex AI to power its personal AI assistant used by millions, Zapia: 

“Working with Google Cloud and Anthropic’s Claude models helps us serve millions of Latin Americans. With Claude, we get a model that’s tailored to the unique needs of Latin Americans. With Vertex AI, we get best-in-class AI infrastructure and tools to deploy to millions of users.” 
— Nicolas Loeff, CTO at BrainLogic AI

Quora, the global knowledge-sharing platform, is harnessing Claude’s capabilities on Vertex AI to facilitate millions of daily interactions through its AI-powered chat platform, Poe:

“We consistently hear from our users about how much they enjoy the intelligence, adaptability, and natural conversational abilities of Anthropic’s Claude models. They’re relying on these qualities for a wide variety of tasks, from the complex to the creative. By leveraging Claude with Vertex AI’s secure and scalable platform, we’re able to facilitate millions of daily interactions, ensuring both speed and reliability.”
— Spencer Chan, Product Lead at Poe by Quora

We’re excited to partner with Anthropic and continue to provide our customers world-class innovation supported by an open and accessible AI ecosystem. 

How to get started with the upgraded Claude 3.5 Sonnet on Google Cloud

Visit Model Garden and select the Claude 3.5 Sonnet v2 model tile. You can also find and easily procure the upgraded Claude 3.5 Sonnet on Google Cloud Marketplace and take advantage of the ability to draw down on your Google Cloud spend commitments.The upgraded Claude 3.5 Sonnet v2 is only available for US customers today. 

Select “Enable” and follow the proceeding instructions.

Use our sample notebook and documentation to start building.

At Google Cloud, we believe that cloud computing will increasingly shift to private, encrypted services where users can be confident that their software and data are not being exposed to unauthorized actors. In support of this goal, our aim is to make confidential computing technology ubiquitous across our Cloud without compromising usability, performance, and scale.

The latest C3 Confidential VMs we’ve developed with Intel have a built-in, on by default accelerator: Intel AMX. Intel AMX improves the performance of deep-learning training and inference on the CPU, and is ideal for workloads including natural-language processing, recommendation systems, and image recognition.

These Confidential VMs leverage confidential computing technology called Intel Trust Domain Extensions (Intel TDX). Confidential VMs with Intel TDX can help safeguard against insider attacks and software vulnerabilities and ensure data and code confidentiality and integrity. Together, Intel’s AMX and TDX technologies can make it easier for customers to secure their AI inference, fine tuning, and small-to-medium sized training jobs.  

“We are excited to partner with Google, delivering Confidential Computing and AI acceleration on C3 instances with Intel TDX and Intel AMX and software acceleration available in the popular AI libraries, to democratize access to secured, accelerated computing,” Andres Rodriguez, Intel Fellow and datacenter AI architect.

Performance testing AI/ML workloads on Confidential VMs

We know that sometimes performance concerns arise when security extensions are enabled, so we conducted ML training and inference experiments to evaluate the performance of Confidential VMs with Intel TDX and Intel AMX. 

For our experiments, we compared three VMs that were the largest machine types by vCPU available:

N2 VM: N2 machine series VMs that do not have Intel AMX or Intel TDX enabled. 
C3 VM: C3 machine series VMs that only have Intel AMX enabled. 
C3+TDX VM: C3 machine series VMs have both Intel TDX and Intel AMX enabled. 

AI Training findings

We compared the training time (in minutes) on the aforementioned N2, C3, and C3+TDX VMs. Compared to the N2 VM, the C3 VM provided a 4.54x speedup on training time whereas the C3+TDX VM provided a 4.14x speedup on training time for one workload, with the same hyperparameters such as global batch size and on-par quality metrics.

We found that workloads running in Confidential VMs with Intel TDX may experience some performance overhead. A greater batch size may speed up training and reduce the Intel TDX overhead. This is due to reduced memory encryption cost involved in I/O operations, given the same amount of ingested data. 

AI Inference findings

We saw similar improvements on inference workloads using Intel AMX and observed some performance overhead from Intel TDX. 

We ran a text generation task on two LLMs, Llama-2-7B and Llama-2-13B. Using the aforementioned N2, C3, and C3+TDX VMs, we evaluated task performance based on two metrics: 

Time Per Output Token (TPOT): Time to generate an output token for a single request. This metric corresponds to how a user would perceive the “speed” of the model. For example, a TPOT of 100 milliseconds per token would be approximately 450 words per minute (WPM), which is faster than an average person’s reading speed.
Throughput (token per sec): The number of output tokens per second an inference server can generate for batch requests.

We compared the N2 VM to C3 and C3+TDX VMs and found that Intel AMX on the C3 VM and C3+TDX VM provided a speedup of approximately a three-fold improvement in latency (TPOT) and approximately a seven-fold increase in throughput (with a batch size of six).

Try Intel AMX on Confidential VMs today 

For a full report on our testing and results, read our whitepaper here. You can collect your own benchmarks for LLMs by following the Intel® Extension for PyTorch instructions. Learn more about Google Cloud’s Confidential Computing products here, and get started today with your own confidential VMs using Intel AMX and TDX.

The Home Depot, the world’s largest home-improvement retailer, has been on a transformation journey on Google Cloud since 2017. We’ve harnessed the power of Google Cloud to enhance our customer experience, optimize critical SAP applications, and build an efficient and integrated supply chain.

Our supply chain, fueled by a data platform on Google Cloud, is a well-oiled machine. Inventory movement across our vast network of stores, online platforms, and replenishment systems is synced in near real-time, thanks to the streaming capabilities of our data platform. This aggregated inventory data is then integrated with our SAP ERP application for financial accounting.

Traditionally, we relied on middleware platforms and third-party add-ons to integrate SAP applications with our data platform. While effective, this approach added complexity and cost to our operations. We needed a solution that would streamline the integration process, reduce costs, and accelerate development. The ABAP SDK for Google Cloud was the answer.

As seasoned ABAP developers at The Home Depot, we’ve been managing our SAP applications for over two decades. The ABAP SDK empowers us to leverage the capabilities of Google Cloud directly within our preferred programming language, ABAP. By handling the complexities of connectivity, authentication, data serialization, and error handling, the SDK allows us to focus on crafting business logic. This has enabled us to rapidly build and deploy a range of impactful use cases, from concept to go-live in just a few weeks. Let’s explore a few use cases:

#1. Boosting financial accounting accuracy with near real-time inventory data

Historically, inventory movement in our physical and online stores was reflected in our ERP system only as aggregated financial transactions. This limited our ability to delve into the details. Now, with the ABAP SDK, we can pull thousands of inventory transactions directly from Google Cloud Pub/Sub into SAP. This provides near real-time inventory visibility, granular transaction details, and improved accuracy in financial postings.

#2. Safeguarding sensitive financial data in integrations

We use third-party systems to manage financial transactions in stores. Data from these systems are ingested into SAP daily. To protect this sensitive data, we create asymmetric keys in Google Cloud Key Management (KMS) and share the public key with these systems. The third-party systems encrypt files using the public key before sending them to SAP, where we decrypt them using Google Cloud KMS with ABAP SDK. This architecture significantly enhances the security of our financial integrations.

#3. Cutting costs and improving performance in data sharing

We have file-based integrations between SAP and various non-SAP systems, including our customer portals. These integrations handle both structured and unstructured data of varying sizes and volumes. Using the ABAP SDK, we leverage Google Cloud Storage as a landing zone for these files, providing a scalable, reliable, and cost-effective file-sharing solution.

At The Home Depot, we’ve taken the ABAP SDK even further by adding monitoring and error handling using the SAP Application Interface Framework. This allows us to proactively monitor the health of our interfaces, respond to issues promptly, and ensure the overall efficiency of our integrations.

Getting started with the ABAP SDK for Google Cloud is easy. You can install the On-premise or any cloud edition in your SAP application for in-place integrations or the BTP edition in your BTP ABAP environment for side-by-side integrations. Google Cloud also offers code samples and sample solutions to help you hit the ground running.

With the ABAP SDK supporting over 75 Google APIs, we’ve only scratched the surface of what’s possible. We’re constantly exploring new use cases to drive efficiencies across our integration landscape. We’d love to hear how you’re using the ABAP SDK for Google Cloud to innovate and transform your business.

In today’s data-driven world, businesses need database solutions that can handle massive data volumes, deliver lightning-fast performance, and maintain near-perfect uptime. This is especially true for companies with critical workloads operating at global scale, where downtime translates directly into lost revenue and frustrated customers. In this blog post, we examine how Palo Alto Networks, a global cybersecurity leader, overcame their performance and scalability challenges by migrating their Advanced WildFire product from Apache Cassandra to Bigtable, Google Cloud’s enterprise-grade, low-latency NoSQL database service, achieving 5x lower latency and cutting their total cost of ownership by half. Read on to learn how they went about this migration.  

At Google, Bigtable has been serving internal systems and external customers for over a decade. Google Cloud’s goal with Bigtable is to solve the industry’s most complex use cases and also reach a broader developer audience. Recent Bigtable features are significant steps in that direction:

Bigtable Data Boost: This breakthrough technology delivers high-performance, workload-isolated, on-demand analytical processing of transactional data. It allows you to execute queries, ETL jobs, and train ML models directly and as frequently as needed on your transactional data without disrupting your operational workloads.
Authorized views: This feature allows multiple teams to leverage the same tables securely and share data from your databases, fostering collaboration and efficient data utilization.
Distributed counters: This functionality aggregates data at write time to help you process high-frequency event data like clickstreams directly in your database to deliver real-time operation metrics and machine learning features consistently and at scale.
SQL support: With over 100 SQL functions now available directly in Bigtable, developers can leverage Bigtable’s performance and scalability using their existing skills.

These enhancements along with existing features make Bigtable the database of choice for a variety of business-critical workloads, including Advanced WildFire.

From Cassandra to Bigtable at Palo Alto Networks

Palo Alto Networks’ Advanced WildFire is the industry’s largest cloud-based malware protection engine, analyzing over 1 billion samples monthly to protect organizations worldwide from sophisticated and evasive threats. To do so, it uses over 22 different Google Cloud services across 21 regions. Palo Alto Networks’ Global Verdict Service (GVS), a core component of WildFire, relies on a NoSQL database to process vast amounts of data that needs to be highly available for service uptime. Initially, when developing Wildfire, Apache Cassandra seemed like a suitable choice. However, as data volumes and performance demands grew, several limitations emerged:

Performance bottlenecks: High latency, frequent timeouts, and excessive CPU usage, often triggered by compaction processes, hampered performance and impacted the user experience.
Operational complexity: Managing a large Cassandra cluster demanded specialized expertise and significant overhead, leading to increased management, complexity, and costs.
Replication challenges: Achieving low-latency replication across geographically dispersed regions proved difficult, requiring a complex mesh architecture to mitigate lag.
Scaling difficulties: Scaling Cassandra horizontally was cumbersome and time-consuming, with node upgrades requiring significant effort and downtime.

To address these limitations, Palo Alto Networks decided to migrate GVS to Bigtable. This decision was fueled by Bigtable’s promise of:

High availability: Bigtable delivers a 99.999% availability SLA, ensuring virtually uninterrupted service and maximum uptime.
Scalability: Its horizontally scalable architecture allows for near-infinite scalability, readily accommodating Palo Alto Networks’ ever-growing data demands.
Performance: Bigtable offers single-digit millisecond latency for read and write operations, resulting in a significantly improved user experience and application responsiveness.
Cost efficiency: As a fully managed service, Bigtable reduces operational costs compared to managing a large, complex Cassandra cluster.

The migration to Bigtable yielded remarkable results for Palo Alto Networks:

5x lower latency: The switch to Bigtable delivered a fivefold reduction in latency, leading to a dramatically improved user experience and application responsiveness.
50% lower cost: Palo Alto Networks achieved a substantial 50% cost reduction thanks to Bigtable’s efficient, managed service model.
Increased availability: Availability jumped from 99.95% to an impressive 99.999%, ensuring near-constant uptime and minimizing service disruptions.
Simplified architecture: The complexities of the mesh architecture required for Cassandra replication were eliminated, leading to  simpler and more manageable infrastructure.
Fewer production issues: The migration resulted in a remarkable 95% reduction in production issues, translating into smoother operations and fewer disruptions.
Enhanced scalability: Bigtable provided the capacity to support 20 times the scale compared to their previous Cassandra setup, providing ample room for growth.

And the best part is that migrating from Cassandra to Bigtable can be a straightforward process — read on to see how. 

The Cassandra to Bigtable migration

While transitioning from Cassandra to Bigtable, Palo Alto aimed to preserve the integrity of their data and continue operations. The following is an overview of the steps in the migration process, which spanned several months:

Initial data migration:

Start by creating a Bigtable instance, clusters, and tables to receive the migrated data.

For every table, use the data migration tool to extract data from Cassandra and load it into Bigtable. The row keys should be designed with read requests in mind. The general rule of thumb is that the Bigtable row key should be the same as Cassandra’s primary key for a table. 

Ensure that the data types, columns and column families in Bigtable match those in Cassandra.

During this phase, continue writing new data to the Cassandra cluster.

Data integrity checks:

Compare the data in Cassandra against the data in Bigtable to verify that the migration was successful using data validation tools or custom scripts. Address any discrepancies or data inconsistencies that are identified.

Enable dual writes:

Implement dual writes to Bigtable (along with Cassandra) for all tables.

Use application code to route write requests to both databases.

Live data integrity checks:

Conduct regular data integrity checks on live data to ensure that the data in Cassandra and Bigtable remains consistent using continuous scheduled scripts.

Monitor the results of the data integrity checks and investigate any discrepancies or issues that are identified.

Redirect reads:

Gradually shift read operations from Cassandra to Bigtable by updating the existing application code and/or load balancers.

Monitor the performance and latency of read operations.

Stop dual writes:

Once all read operations have been redirected to Bigtable, cease writing to Cassandra, and ensure that all write requests are directed to Bigtable.

Decommission Cassandra:

Safely shut down the Cassandra cluster after all data has been migrated and read operations have been redirected to Bigtable.

The workflow and approach is summarized in the diagram below:

Tools for migrating existing data

Palo Alto Networks used the following tools during the migration process: 

Use the ‘dsbulk’ tool for data unloading: The ‘dsbulk’ tool can export data from Cassandra into CSV files. These files are loaded to Cloud Storage buckets for further ingestion. 

Reference: Datastax dsbulk Documentation

Create Dataflow pipelines for loading into Bigtable: Dataflow pipelines were deployed to load the CSV files into Bigtable within a test environment. 

Reference: Google Cloud Dataflow CSV Import Guide

At the same time, given the critical nature of data migration, Palo Alto adopted a two-step approach: an initial dry-run migration followed by the final migration. This strategy helped mitigate risks and refine the process.

Reasons for a dry-run migration:

Evaluate impact: Assess the effects of the ‘dsbulk’ tool on the live Cassandra cluster, especially under load, and adjust configurations accordingly.

Identify issues: Detect and resolve potential issues associated with the massive data volume (in the terabytes).

Estimate time: Gauge the time required for migration to plan for live traffic handling during the final migration.

Then, when it was ready, it moved on to the final migration.

Final migration steps:

Deploy pipeline services:

Reader service: Reads data from all MySQL servers and publishes to a Google Cloud Pub/Sub topic.

Writer service: Writes data to Bigtable from a Pub/Sub topic.

Set cut-off time: Define a cut-off time and repeat the data migration process.

Initiate services: Start both the writer and reader services.

Conduct final checks: Perform comprehensive data integrity checks to ensure accuracy and completeness.

This structured approach ensures a smooth transition from Cassandra to Bigtable, maintaining data integrity and minimizing disruption to ongoing operations. With planning, Palo Alto Networks was able to ensure that migration was both efficient and reliable at each phase.

Migration best practices

Migrating from one database system to another is a complex undertaking, requiring careful planning and execution. Here are some best practices for migrating from Cassandra to Bigtable that Palo Alto employed:

Data model mapping: Analyze and map your existing Cassandra data model to a suitable Bigtable schema. Bigtable offers flexibility in schema design, allowing for efficient data representation.
Data migration tools: Use data migration tools like the open-source “Bigtable cbt” tool to streamline the data transfer process and minimize downtime.
Performance tuning: Optimize your Bigtable schema and application code to maximize performance and fully leverage Bigtable’s capabilities.
Application code adaptation: Adapt your application code to interact with Bigtable’s API and leverage its unique features.

That said, here are some potential pitfalls to watch out for:

Schema mismatch: Ensure that the Bigtable schema accurately reflects the data structures and relationships in your Cassandra data model.
Data consistency: Carefully plan and manage the data migration process to ensure data consistency and avoid data loss.

Get ready to migrate to Bigtable

All told, Palo Alto Networks’ migration to Bigtable was transformative. If your organization is grappling with database challenges like performance bottlenecks, scalability limitations, or operational complexity, consider Bigtable. Recent enhancements including Bigtable Data Boost, authorized views, distributed counters, and SQL support, make it even more compelling for diverse workloads.

Ready to experience the benefits of Bigtable for yourself? Now, Google Cloud offers a seamless migration path from Cassandra to Bigtable, utilizing Dataflow as the primary dual-write tool. This Apache Cassandra to Bigtable template simplifies the setup and execution of your data replication pipeline. Start your journey today and unlock the potential of a highly scalable, performant, and cost-effective database solution.

Vulnerability reward programs play a vital role in driving security forward. By incentivizing security research, vulnerabilities can be found and fixed by vendors before they are potentially exploited by malicious actors, protecting users and strengthening security posture. Also known as bug bounties, Google has long been a leader in supporting them, and they are now an integral part of the security landscape.

As part of our commitment to security, we are pleased to announce the launch of the Google Cloud Vulnerability Reward Program (VRP), dedicated to products and services that are part of Google Cloud. The Google Cloud VRP will continue to focus on coordinating new vulnerabilities and compensating security researchers for helping us in our mission, and offers a top award of $101,010. 

Delivering the most secure cloud

While the broader Google VRP has covered Google Cloud until now, the launch of the Google Cloud-specific VRP enables us to invest more deeply to pursue a more secure cloud. With this launch, we are better aligning our rewards with our top cloud products, resulting in over 150 products coming under the top two reward tiers. 

Additionally, vulnerability researchers will now be directly interacting with our Google Cloud security engineers. Their interactions will enable us to more quickly triage, reproduce, and assess the impact of security research reports. While the new Google Cloud VRP offers an improved reward structure focused on Google Cloud, researchers will still receive the same high quality engagement, transparency, and communication that they have come to expect from the Google VRP.

How to submit a vulnerability to Google Cloud

To streamline vulnerability reporting, researchers should continue to use the same reporting portal that they use for the Google, Chrome, Android, and Abuse VRPs. 

To tell us about a vulnerability, please follow these guidelines:

From the portal, start a report for any Google Cloud product or service. 

Under Bug Location, select Cloud VRP.

Follow our guidance to make it easy for us to quickly reproduce the bug. The easier it is for us to reproduce the attack by following your description, the more streamlined communications will be with our team.

Be as detailed as possible regarding the attack scenario. Make sure to outline who would want to exploit a particular vulnerability and what they may gain. As you explain these attack scenarios, you’ll want to think about the starting position of the attacker and any prerequisites for the attack. It’s also best to articulate assumptions about the victim.

Helping us to quickly reproduce a vulnerability, and understand the attack scenario, can make it easier for us to accurately assess the impact of the vulnerability — and fix the issue quickly. While finding coding flaws is fun, we also want to see our bug hunting community become successful, and that means clearly articulating complex real-world attack scenarios.

VRPs have become such an important part of a robust, mature security program that they can even help organizations achieve their digital transformations. The Google Cloud VRP team and our security engineers look forward to partnering with all of our researchers to help collectively secure the cloud.

Since its invention 50 years ago, SQL has become the universal language used across the database industry. Its familiar syntax and well-established community have truly democratized how people access data everywhere. But let’s face it, SQL isn’t perfect. SQL’s syntax has several issues that make it harder to read and write:  

Rigid structure: A query must follow a particular order (SELECT … FROM … WHERE … GROUP BY…), and doing anything else requires subqueries or other complex patterns.  
Awkward inside-out data flow: A query starts in the middle and then logic builds outwards, starting with FROM clauses embedded in subqueries or common table expressions (CTE).
Verbose, repetitive syntax: Tired of listing the same columns over and over in SELECT, GROUP BY, and ORDER BY, and in every subquery? We hear you.

These issues can make SQL more difficult for new users. Even for expert users, reading or writing SQL takes more work than should be necessary. A more convenient syntax would help everyone.

Over the years, several alternative languages and APIs have been proposed, and some have had some success in narrow applications. Many of these, including Apache Beam and Python DataFrames, use piped data flow, which makes it easy to compose queries arbitrarily. Many users find this syntax easier to understand and more usable than SQL.

But SQL is ubiquitous, and unlikely to be replaced any time soon. Learning a new language and migrating workloads to new tools is simply too much for most users.

This got us wondering. Would it be possible to add something to SQL to give it the same power and flexibility, while keeping everything that’s familiar and works well in SQL? The answer is yes! 

Introducing SQL pipe syntax: the best of both worlds 

At Google Cloud, we’re on a mission to make data analysis easier and more intuitive. That’s why we’re thrilled to introduce pipe syntax, a groundbreaking innovation that brings the elegance of piped data flow to SQL in BigQuery and Cloud Logging. You can read our recent research paper on the topic here. 

What is pipe syntax? 
In a nutshell, pipe syntax is an extension to standard SQL syntax that makes SQL simpler, more concise, and more flexible. It supports the same underlying operators as standard SQL, with the same semantics and mostly the same syntax, but allows applying operators in any order, any number of times. 

How it works:

Queries can start with FROM.

Operators are written sequentially, using the |> pipe symbol.

Each operator consumes its input table and produces an output table.

Most pipe operators use the same syntax as standard SQL:

SELECT, WHERE, JOIN, ORDER BY, LIMIT, etc.

Standard and pipe syntax can be mixed arbitrarily, even within the same query.

Example: Suppose you want to know the average number of taxi trips per year in Chicago by payment_type using a BigQuery public dataset. Here’s how you might write that in standard and pipe syntax.

Using the standard syntax, you typically need to write a subquery:

Here’s that same query using pipe syntax — no subquery needed!

Real-world impact at HSBC

The global banking giant HSBC has already embraced pipe syntax, following impressive results experimenting with a preview version in BigQuery. They saw significant improvements in productivity and code readability, especially when dealing with large JSON datasets. 

“We’re thrilled with the addition of pipe syntax to BigQuery! Our team has immediately put it to use to rewrite business reports and make future reports less time consuming, particularly those involving large JSON datasets. Previously, unpacking this data required extensive filtering and extraction, leading to cumbersome and repetitive SQL query writing engagement. The pipe syntax helps to significantly streamline this process by making up to 80% of the query string reusable, so we can focus on the specific use cases. This is going to be a major productivity boost for our engineers in the future, making it far easier to write and debug complex queries. We’re excited to expand our use of pipe syntax to even more use cases and data types in the future.” – Zoltán Pósfai, GCP Cloud Engineering Lead, HSBC 

Advantages of adding pipe syntax inside SQL

We think that the addition of pipe syntax in SQL brings a number of benefits to SQL developers. Here are a few:

Easy to learn
Learning and adopting new languages is challenging, particularly in organizations with many users, and it’s best if all users use the same languages and tools. Pipe syntax is not a new language — it’s a new feature within the existing SQL language. Users who already know SQL find pipe syntax very easy to learn since it has all the same operators, with mostly the same syntax. 

For users new to SQL, it’s easier to learn pipe syntax first. They still need to learn the operators and some semantics (e.g., inner and outer joins), but they can use those operators to express their intended queries directly, bypassing some of the complexities and workarounds required when expressing queries in standard SQL.

Easy to adopt incrementally, without migrations 
We all know that migrating to a new language or system can be expensive, time-consuming and error-prone. Since pipe syntax is a feature inside GoogleSQL, you don’t need to migrate anything before you can start using it. The new syntax can be used selectively, wherever it’s helpful, and all existing queries continue to work. Any new SQL is fully interoperable with existing SQL code. For example, queries with pipe syntax can call standard views that were written in standard syntax, and vice versa. Using pipe syntax in new SQL code doesn’t make any existing SQL obsolete or unusable.

No performance or cost hit
Pipe syntax works in existing platforms like BigQuery without any extra layers (such as translation proxies) that can add latency, cost, or reliability risks, and that can also make debugging or tuning difficult.

There’s also no additional cost. Queries with pipe syntax still have SQL’s declarative semantics, meaning the SQL query optimizer will still rearrange the query to run more efficiently. In other words, queries written in either standard or pipe syntax typically have exactly the same performance.

What can you use pipe syntax for?

Whether you’re exploring data, creating dashboards, building data pipelines or analyzing logs, pipe syntax empowers you to write clearer, more efficient, and more maintainable SQL queries. And because pipe syntax supports most standard SQL operators, you can use it anywhere you write queries. Here are a few applications to get you started:

Ad-hoc analysis and query debugging 
When performing data exploration, you typically start by looking at rows of a table (starting with a FROM clause) to see what exists, and then add filters, aggregations, joins, ordering, etc. Pipe syntax makes this kind of exploration very easy because you can start with a FROM clause and build from there. At every step, you can see the results so far, and then add another pipe operator, and then run the query again to see the new results. 

Pipe syntax also helps when debugging queries. Queries in pipe syntax have the nice property that any query prefix up to a pipe symbol is also a valid query, so it’s possible to highlight that query prefix and run it, showing the intermediate result up to that point.

Data engineering lifecycle
As the volume of data grows, data processing and transformation become more challenging and time-consuming. In data-heavy environments, building, adapting and maintaining a data pipeline usually involves a large engineering effort. With its linear query structure and more intuitive syntax, pipe syntax streamlines data engineering. Say goodbye to those deeply nested queries and CTEs that always seem to show up when using standard SQL. This new addition to GoogleSQL reimagines how to parse, extract and transform data, making it easier to create and maintain data pipelines. Here’s an example use case showing how to accelerate log data workloads with pipe syntax both in BigQuery and Cloud Logging. 

Applying LLMs and natural language with SQL 
Research shows that SQL can be challenging for large language models (LLMs) to understand or generate, for the same reasons SQL can be complex for humans to read and write. In contrast, pipe syntax breaks queries down into independent steps, which correspond closely to the desired logical data flow. It’s easier for the LLM to generate pipe syntax to express a desired data flow, and the generated queries can be much simpler and more human-readable, also making human validation of generated queries much easier.

Pipe syntax also makes it possible to provide better code assistants and auto-completion, since it’s much easier to understand what’s going on and what’s possible, and to make suggestions for local edits to one pipe operator rather than global edits to a whole query. Smarter AI-generated code suggestions and mixing more natural language-based operators into a query are great opportunities for further enhancing user productivity. 

Unlock the power of pipe syntax today

For 50 years, SQL has been the universal language of data because it works so well. SQL gets many things right, and there’s tremendous power in expressing queries as declarative combinations of relational operators.

But that doesn’t mean SQL couldn’t be better. Pipe syntax brings SQL into the future, solving SQL’s biggest usability challenges, and enabling new ways of working with and extending SQL. This isn’t about replacing SQL or inventing a new language. SQL with pipe syntax is still SQL, but it’s a better SQL — one that’s more user-friendly, flexible and expressive.

We believe pipe syntax is the right direction for the future of SQL and hope to see it adopted across the industry. We can’t wait to see how you plan to use it today! You can sign up to use pipe syntax in BigQuery here. 

Resources

Sign-up form to try out pipe syntax in BigQuery 
Pipe syntax documentation and detailed reference guide 
Demo video 
Blog post on using pipe syntax for log data analysis
Research publication (VLDB 2024) with more background and language design details
Open-source GoogleSQL implementation, ZetaSQL

What kind of performance does your application have, and how do you know? More to the point, what kind of performance do your end users think your application has? 

In this era of rapid growth and unpredictable traffic surges, understanding your application’s scalability is not just a technical concern — it’s a strategic imperative for success. Providing end users with optimal performance is of course non-negotiable, and benchmarking it is a key way to meet their expectations. 

Nor do you want to simply benchmark your application’s components — you need to benchmark entire critical user journeys (CUJs) end-to-end, as the user perceives them, to provide a holistic view of the application’s performance under real-world conditions. Benchmarking components in isolation may overlook potential bottlenecks and performance issues that arise from the interplay of various components, network latencies, and external dependencies. By simulating complete user flows, you gain insights into the actual user experience, enabling you to identify and address performance issues that impact user satisfaction and engagement.

In this blog post, we discuss why it is important to incorporate end-user perceived performance benchmarking into modern application development, and how to incubate an organizational culture that benchmarks applications from the get-go — and that continues to benchmark over time. As part of the Google Kubernetes Engine (GKE) engineering team, we also show you how we use the open-source Locust tool to simulate complex user behavior to use as part of your end-to-end benchmarking practice.

Why benchmarking is indispensable 

There are many reasons you should institute robust benchmarking practices as part of your application development process: 

Proactive performance management: Early and frequent benchmarking helps developers identify and address performance bottlenecks early in the development cycle, which in turn can help to conserve resources, accelerate time-to-market, and deliver smoother product launches. Additionally, integrating benchmarking into testing protocols can add a critical safety net, safeguarding code quality and user experience by promptly detecting and addressing performance regressions.

Continuous performance optimization: Applications are dynamic, subject to continuous evolution, scaling, and changes in user behavior. Regular benchmarking facilitates ongoing monitoring of performance trends, allowing developers to evaluate the impact of new features, updates, and system modifications, thus helping the application remain consistently performant and responsive in the face of change.

Bridging the gap between development and production: As part of a development process, benchmarking real-world workloads, images, and scaling patterns provides a realistic assessment of application performance in a production environment. This helps developers proactively address potential issues and smooth transitions from development to deployment.

Benchmarking scenarios to mimic real-world load patterns

As a developer, your goal should be to benchmark your applications in conditions that closely mirror real-world scenarios, including deployment, scaling, and load patterns. This approach tests how well applications handle unexpected traffic surges without compromising performance or user experience.

As part of the GKE engineering team, we perform extensive benchmarking across various scenarios to test and optimize our cluster and workload autoscalers. This helps us understand how our autoscaling mechanisms respond to varying demands, maximizing resource utilization and maintaining optimal application performance.

Locust for realistic load testing and performance benchmarking

Locust is a sophisticated yet simple-to-use load-testing tool that helps developers simulate complex user behavior through scripting, offering a comprehensive understanding of application performance under realistic conditions. By defining and instantiating “users” who perform specific tasks, Locust enables the creation of various load scenarios.

In one example benchmark, we employed Locust to simulate users accessing a web server and requesting the 30th Fibonacci number. This generated a consistent load of approximately 200 ms per request, with each connection closed and re-established to ensure load balancing across different pods.

Locust makes it relatively easy to simulate these kinds of intricate user interactions within your application. It can generate up to 10,000 requests per second on a single machine, and can scale further through distributed deployment out-of-the-box. It provides granular control over user count and spawn rate via custom load shapes, enabling you to emulate real-world load patterns with users who exhibit diverse load profiles. It natively supports HTTP/HTTPS protocols for web and REST requests, and is extensible to a wide array of systems, including XML-RPC, gRPC, and various request-based libraries/SDKs.

Accompanying this blog post, we provide a GitHub repository to illustrate an end-to-end benchmark of a pre-release autoscaling cluster environment. You are encouraged to adapt it to align with your specific requirements.

Benchmarking end-users’ perceived performance is not just a best practice; it’s imperative for delivering exceptional user experiences. By proactively integrating benchmarking into the development process, developers can discover whether their applications remain performant, responsive, and capable of meeting varying user demand.

Tools like Locust help simulate real-world scenarios, so you can gain deeper insights into your application performance under diverse conditions. Performance is an ongoing pursuit. Let benchmarking guide you toward delivering exceptional user experiences.

At Google Cloud, we’re always working to make GKE faster, more scalable and efficient under the hood. Upgrade your Kubernetes environments early to capitalize on the latest improvements in GKE, like the Cluster Autoscaler improvements introduced in GKE v1.29.

Today, we’re announcing a new tool to help Google Cloud partners better track their growth and opportunities, and new resources to help them increase their AI expertise and respond to growing customer demand.

Our partners are helping customers around the globe implement and generate business value with Google Cloud products. Generative AI in particular is having a significant impact on how businesses operate, with 74% of early adopters already seeing ROI through improved revenue, productivity, and customer experiences. We are committed to helping partners accelerate adoption of this powerful technology by providing the incentives, rewards, and education needed to grow their services practices and capitalize on the massive AI business opportunity that lies ahead. Now, we’re making it even easier for partners to understand these incentives with a new Earnings Hub platform.

Track and optimize incentives with Earnings Hub

Earnings Hub is a new platform that provides partners with a comprehensive overview of their earned incentives, unifying critical information like rebates, discounts, funds, and credits into a single, easy-to-use dashboard. Google Cloud partners can access this platform today to view Google Cloud incentive information, and in the coming weeks they’ll also be able to view both Workspace and Security Operations (SecOps) incentives information. With Earnings Hub, our goal is to provide partners with greater transparency into incentive calculations and eligibility, and to help them increase adoption of those incentives. 

Key benefits include: 

Unified dashboard: Partners can view a unified summary of all earned incentives, including rebates, discounts, funds, and credits, which can be sorted by specific time periods, product families, and customer segments.
Incentive analysis: Partners can analyze historical data and specific earnings to help inform decisions about where to invest in their businesses, with detailed overviews for each incentive type.
Filterable data: Partners can apply filters to view earnings by product family, customer, country, time period, and more, so they can easily find the information most relevant to them.
CSV download: Users can easily download incentive data in CSV format for offline analysis and record-keeping.

Early feedback on Earnings Hub from a select group of pilot partners has been very positive. They report that Earnings Hub has significantly improved their ability to make faster and more effective business decisions, enhancing incentive-based revenue and growth opportunities. Earnings Hub is also built with the same security measures as the Partner Advantage portal, including user authentication and authorization protocols. Access is strictly limited to authorized partner users. 

Here’s what partners had to say:

“Our team is excited for the improved clarity that Earnings Hub will bring to Google incentives, allowing us to more timely and accurately forecast and account for existing incentive programs. This will ultimately help us more precisely incorporate Google incentives into our ongoing financial decision-making processes.” – Aaron Billings, CFO, 66Degrees

“Earnings Hub enables our team to have much improved visibility into our earnings for the ever-changing portfolio of Google incentive programs. This gives us the ability to make more accurate accruals in our books and also helps us close the books faster.” – Stephanie Linton, Director of Alliances, Promevo

“Pythian appreciates the enhanced insights and opportunity analysis tools for Google Incentives that Earnings Hub will provide to partners, like us. This information will enable us to make better business decisions, improve sales processes, and enhance our ability to accurately forecast revenue and profit, which will allow us to serve our customers and stakeholders better.” – Michael Bray, SVP of Global Sales, Pythian

We’re just getting started with Earnings Hub. In the coming months, AI will play an increasingly important role by powering conversational chat experiences and providing more robust data analysis, including recommendations to help partners maximize their earnings potential. Next year, we’ll continue to build on the power of AI to offer personalized, actionable insights on earnings potential. 

AI services and skills enablement

Partners recognize the value of growing their AI services capacity to better assist customers who are looking for experts to guide them through the stages of an AI project. Since last year, our services partners have taken over 500,000 Google gen AI courses, and they are now achieving more than 1,500 skills badges a week. 

Today, we’re sharing multiple advancements in partner enablement for the most in-demand AI skills, including: 

An updated Professional Machine Learning Certification, which has been improved with the most current AI/ML best practices and is a great way for partners to validate their core AI skills. Existing certification holders do not need to re-test. 
New introductory AI courses to help partners start developing important skills like deploying apps with Firebase GenKit, using LangChain with Gemini, and training more reliable deep learning models. 
New gen AI courses and learning paths to equip technical professionals such as developers and data engineers, with real-world generative AI skills.
New gen AI workshops and webinars that we are facilitating regularly around the world for all knowledge levels. Topics include areas of rapid growth, such as developing apps with Vertex AI Agent Builder or boosting expertise using Google’s foundation models to generate code, images, and text. 

This month, we’re also running the Gen AI Skill Badge Challenge, which is a fun, engaging way to learn new gen AI skills, badges, and prizes. 

Other major opportunities for partners

Since Google Cloud Next ‘24, partners have moved quickly to support customer demand for AI and many other projects. Some of the biggest opportunities for partners currently include: 

Migrating workloads to VMware Cloud Foundation on Google Cloud: Modernizing infrastructure is often the first step towards realizing the benefits of AI. In June, we launched VMware Cloud Foundation on Google Cloud VMware Engine (GCVE) for customers looking to modernize critical workloads with minimal financial and technical friction. With funding provided by Google Cloud, partners have provided no-fee assessments and POCs to customers evaluating GVCE as part of their workload modernization efforts, and there remains a significant opportunity for partners to help customers modernize these workloads.
Combining Oracle database services with Google Cloud AI: Customers can now connect Oracle database services with Google Cloud AI services like Vertex AI and Gemini models to build new AI applications and agents, using the critical data they store with Oracle. Partners can play a major role helping customers deploy and manage these integrations, including the development of deep analytics capabilities enabled through BigQuery and Looker.  

Google Cloud is a partner-first organization, and we remain committed to investing in the tools, learning services, and technical resources that our ecosystem needs to accelerate gen AI adoption. We’re proud of what our partners have accomplished so far this year, and we’re excited to watch them continue to grow in the months ahead.

The modern workplace revolves around the browser. It’s where employees access critical applications, handle sensitive data, and collaborate with colleagues. This makes the browser a critical point for enforcing security. 

Chrome Enterprise, the most trusted enterprise browser, recently introduced powerful new capabilities for Chrome Enterprise Premium designed to further enhance security, threat detection, and usability. Let’s check them out. 

Enhanced data loss prevention with watermarking

Watermarking, now generally available, can help deter data leaks by overlaying a customizable, semi-transparent watermark on sensitive web pages. When a user visits a page that triggers a Data Loss Prevention (DLP) rule, the watermark appears, clearly identifying the content as confidential. 

This visual reminder can help increase security awareness, and can help prevent unauthorized sharing. For example, an organization might configure watermarks to appear on web pages containing sensitive financial data or confidential product designs. 

The watermark adds an extra layer of protection, making it clear to users that the information they are viewing is protected and should not be shared freely. You can learn how to set up watermarks here.

Advanced threat detection with URL filtering

URL filtering is a cornerstone of any secure enterprise browser. Chrome Enterprise Premium now offers an optional audit-only mode for URL filtering, allowing administrators to monitor activity in managed corporate browsers and user profiles without blocking access to sites in the filter. The audit-only mode provides valuable insights into risky user behavior and security threats, and can be used by enterprises to refine URL filtering policies, block malicious sites, and educate employees about safe browsing practices. 

Simplified security management with Chrome Security Insights

Chrome Security Insights simplifies security monitoring for Chrome Enterprise Core customers, providing insight into potential threats. With a few clicks, administrators can:

Enable Chrome Enterprise Connectors and security event logging automatically;

Activate 50 common DLP detectors to scan for sensitive data transfers; and,

Generate insightful reports on users, domains, and data types involved in high-risk activities.

Chrome Security Insights reduces the effort required to set up and manage DLP rules, which can help administrators identify and address potential insider risk and data exfiltration events more quickly. This proactive approach to security can help organizations stay ahead of threats and minimize the impact of potential breaches. 

By simplifying security management, Chrome Security Insights empowers IT teams to focus on other critical tasks while maintaining a high level of protection. This feature is currently in preview, and it is easy for any organization using Chrome Enterprise Core to turn it on directly in the Google Admin console home page.

Take the next step

You can get started with Chrome Enterprise Premium with pay-as-you-go pricing. Our flexible pricing model ensures you only pay for what you use. It’s a cost-effective way to access the advanced security features of Chrome Enterprise Premium without a long-term commitment. 

You can activate Chrome Enterprise Premium now on a self-service basis through the Google Cloud console.

You can also experience the advanced security features of Chrome Enterprise Premium first-hand with a risk-free, 60-day free trial. After the trial, you can smoothly transition to pay-as-you-go pricing, allowing you to scale your usage and only pay for what you need.

Learn how to set up your free Chrome Enterprise Premium trial and enable Chrome Security Insights here.

When it comes to purchasing technology for different departments and business units that operate across the globe, enterprise customers need flexibility and choice. This needs to extend to the technology, including generative AI solutions, that integrate with a customer’s Google Cloud environment and that are procured through Google Cloud Marketplace.

Google Cloud Marketplace offers a high-quality catalog of innovative ISV solutions, provides flexible ways to try and buy them, and maximizes the value of cloud investments by allowing purchases of ISV solutions to draw down on Google Cloud commitments. By bringing much of the traditional IT sales process online, we help streamline transactions between customers, technology vendors, and channel partners, whether through public listings or through custom, negotiated private offers. 

Private offers are an effective way for customers and partners to align on terms and payment structures that match an enterprise’s specific needs. Today, we are excited to announce additional private offer enhancements, and the enterprise use cases they unlock.

Support for enterprise AI purchasing models

Google Cloud Marketplace supports the buying and selling of generative AI SaaS solutions and key foundational models that are deployable to Vertex AI. These ground-breaking solutions are helping enterprises deliver innovative business applications ranging from high-quality code generation and summarizing large documents to generating content for products and services. Google Cloud Marketplace private offers make it possible for customers to transact third-party foundational models and LLMs through a variety of transaction models including committed use discounts (CUDs), usage-based discounts, and, now, provisioned throughput, a fixed-cost subscription service that reserves throughput for supported generative AI models on Vertex AI. 

We designed provisioned throughput to provide customers and partners the flexibility to transact and use any model from a partner-specified model family based on the capacity purchased. Provisioned throughput is suitable for customers that are building real-time generative AI applications such as chatbots and customer agents; enabling critical workloads that require consistently high throughput; and maintaining deterministic costs with control of overages. At the same time, customers can continue to take advantage of the spending optimization that procuring through Google Cloud Marketplace provides, such as the ability to draw down on Google Cloud commitments by spending on ISV solutions, including generative AI models.

Tailor offers and payment schedules with multiple orders

When purchasing an ISV’s SaaS solution on Google Cloud Marketplace, each individual business unit within an enterprise now has the flexibility to get different subscriptions and payment schedules based on their cost center’s needs. This capability allows customers to have multiple orders for the same product, which is particularly useful for large enterprises with different departments, subsidiaries, or international branches. Furthermore, each unit can have customized subscription plans for the same technology solution. This feature is generally available for fixed-fee SaaS solutions via private offer today. To enable this, partners can turn on multiple orders for their applicable products in Google Cloud Marketplace. Check out a video demo of how this works.

Customers and ISV partners are already enabling enterprise use cases and seeing the value this new feature unlocks.

“Support for multiple orders of the same product in Google Cloud Marketplace has been a game-changer for Quantum Metric and our customers. With multiple order support we’ve been able to meet the needs of our enterprise customers who require fast upsells as well as separate subscriptions to our analytics platform across different business units. This has enabled Quantum Metric to expand our footprint in our joint accounts and increase revenue, while providing an even higher level of customer support during the procurement cycle.” – Russell Efird, Head of NA Partnerships, Quantum Metric

“Spoon Guru’s integration with Google Cloud Marketplace has helped accelerate business growth, enabling us to rapidly expand our customer base and deliver exceptional value. The ability to offer multiple orders for our Nutrition Intelligence product has been instrumental in meeting the diverse needs of our enterprise clients, from fast-paced upsells to tailored subscriptions across different business units. This feature has not only accelerated revenue growth but also strengthened customer satisfaction by providing greater flexibility and support throughout the procurement process.” – Markus Stripf, Co-Founder, Spoon Guru

“Tackle exists to make things easier for our Cloud GTM customers. We’re constantly evolving our roadmap to help ISVs sell through and with Google Cloud Marketplace in a more flexible, streamlined, and effective way. That’s why we’re so happy to be a launch partner for Google Cloud Marketplace’s multiple orders for a single customer functionality. Greater flexibility for Google Cloud Marketplace customers helps them accelerate their Cloud GTM journey, accommodate the needs of customers, and grow revenue.” – John Jahnke, CEO, Tackle.io

Partners can also take advantage of simplified payment schedules for ISV-directed and reseller-initiated private offers to better deliver flexible payment options to end-customers. Google Cloud Marketplace private offers support monthly, quarterly, annual, and custom installment payment schedules, including prorated payments. This enhancement simplifies the user experience for managing variable payment schedules.

Upfront payments and extended contract lengths

In addition, customers have improved flexibility over how they manage their spend by making upfront payments for critical technology solutions up to five years in advance. The ability to make multi-year upfront payments is available for private offers and lets customers draw down on their Google Cloud commitment on an annual amortized basis. Furthermore, we are also introducing contract lengths of up to 7 years. These capabilities provide more flexible spend management for customers and improved control over financials for our partners.

Learn more about how customers can easily discover, try, and streamline procurement of innovative solutions from our partner ecosystem on Google Cloud Marketplace.

Government organizations and agencies face significant challenges from threat actors seeking to compromise data and critical infrastructure, and impact national and economic stability. These attacks target a broad range of industries and organizations, from water security to rural health care, via external targeting and insider threats. Government organizations are in the top five industries targeted by cyberattacks, according to this year’s M-Trends report, a special look at the evolving cyber threat landscape published by Mandiant. 

Google Cloud Security is committed to helping government agencies and organizations strengthen their defenses, including managing threats, and keeping staff trained and ready for any scenario, using generative AI to help expedite tasks, and helping to keep sensitive workloads secure. Google Cloud’s portfolio of security solutions, combined with Mandiant threat intelligence and expertise, positions us to help public sector organizations stay protected at every stage of their security lifecycles. Today at the Google Public Sector Summit, we’re proud to unveil several new announcements around our security solutions. Read about all of the details below.

Driving effective security operations 

Building on Google Cloud services that have achieved a FedRAMP High designation, we are pleased to announce today that Google Security Operations is now authorized for operation in FedRAMP High environments.  

Positioned as a Leader in the recent in the IDC MarketScape: Worldwide SIEM for Enterprise 2024 Vendor Assessment, the IDC report states, “Google Security Operations is a cloud-native SIEM that is integrating previous point products such as SOAR and ASM, enriching all alerts with Google Threat Intelligence, and offering a gen AI assistant to alleviate mundane security analyst tasks. It is built on the search, data visualization, and storage services of Google Cloud.”

Achieving FedRAMP authorization complements other government-focused efforts for Google Security Operations, including Google Cloud Cybershield™, which enables governments to build an enhanced cyber threat capability, get actionable insights in real time, and develop skills and processes that drive effective security operations.

New cloud compliance control packages and service expansion

Assured Workloads can help public sector customers run regulated workloads on Google’s public cloud infrastructure, without the tradeoffs of isolated government clouds. By enforcing data location, personnel access controls, and policy constraints, Assured Workloads can simplify the process of maintaining security and compliance while allowing customers to take advantage of the flexibility and innovation of Google Cloud services.

Today, we’re pleased to announce the expansion of services across several Assured Workloads compliance control packages: 

We recently added nine services for our control package supporting the FBI’s Criminal Justice Information Services (CJIS) information protection guidelines. These services unlock AI capabilities such as Text-to-Speech for applications, and enable users to build frontend and backend services with CloudRun and Spanner. The CJIS program now covers 16 accredited states.

Our new control package for Assured Workloads can support state and local government customers’ compliance with IRS 1075. The IRS 1075 control package maps to FedRAMP High controls, and includes Access Transparency control for U.S. personnel.

We now offer a Healthcare and Life Sciences control package, which can help simplify configuration and security for regulated workloads in healthcare and life sciences. It configures support for VPC Service Controls, Access Transparency, data residency, and customer-managed encryption keys. In addition, Healthcare and Life Sciences Controls can help organizations ensure that Google Cloud services used in their workload are covered by Business Associate Agreements and have passed HITRUST Common Security Framework (CSF) certification.

Simplify your compliance audit process

Compliance audits in the cloud can be time consuming and complex, and you need to generate evidence of compliance every audit cycle, each time you upgrade or deploy a new application, or whenever regulations change. 

We are excited today to introduce our new Google Cloud Audit Manager, so you can evaluate your resources against industry benchmarks such as NIST 800-53 Revision 4, NIST CSF, FedRAMP Moderate, ISO 27001, HIPAA, and Google-recommended AI controls. Generally available later this month, this powerful tool is designed to streamline and simplify your compliance audit process on Google Cloud by: 

Providing a shared responsibilities matrix, clearly articulating separation of duties and recommendations for you to execute your responsibilities.

Automating compliance assessments to evaluate compliance controls on workloads so you can understand their state of compliance.

Gathering evidence necessary for your compliance audits.

Audit Manager can be used by any Google Cloud customer and SCC Enterprise customers will have access to Audit Manager at no additional cost, which means customers can have a more complete CNAPP experience, streamlining their audit evidence generation to complement SCC’s monitoring and reporting capabilities.

Government-ready Gemini for Google Workspace

The shift towards generative AI presents an opportunity for governments to transform the lives of their citizens, through enhanced efficiencies, improved resilience of utilities and citizen services, and protection of government information. Our AI-powered agent, Gemini for Workspace, which has been submitted for FedRAMP High authorization, is built into the applications, such as Gmail, Drive, and Docs, to help your teams do more in less time.

We are also announcing general availability of Drive Inventory Reporting (DIR), which generates weekly snapshots of key metadata for files in Drive. DIR allows administrators to understand what data they have, how it is labeled, and who has access to it. DIR also includes Gemini logs (files accessed by or changes made by Gemini). 

DIR is mapped to the U.S. government’s Zero Trust Maturity Model Data pillar, which requires that agencies “inventory, categorize, and label data; protect data at rest and in transit; and deploy mechanisms to detect and stop data exfiltration.”

Immersive cyber range experience

Mandiant has long supported the government sector. In addition to recent announcements for Mandiant Custom Threat Hunt and Mandiant Managed Defense for Google Security Operations, we are announcing Mandiant ThreatSpace. 

Mandiant ThreatSpace facilitates a consequence-free, immersive environment to practice and learn without risk. Hosted on Google Cloud, this customized, live-fire, technical learning service is designed to help security teams keep their skills sharp and learn new ones by performing incident response and threat hunting against the latest real-world threat and attack scenarios.   

Cybersecurity assistance for rural hospitals

Google is committed to helping rural health systems strengthen their resilience to cyberattacks. We are partnering with government and industry to offer security services, support, and technologies, enabling these hospitals to defend their systems and keep their focus on providing safe patient care. 

This tailored initiative to improve cybersecurity is specially designed for rural hospitals and health clinics. The program delivers a growing set of secure-by-design Google technology for access and collaboration, consulting and support services, and security training resources at a discount or no cost. Interested organizations can learn more and sign up for more information.

Learn more about Google Cloud Security

Google Cloud Security is committed to empowering public sector organizations with the tools and resources they need to navigate today’s complex threat landscape. From FedRAMP-authorized solutions like Google Security Operations and Assured Workloads to innovative offerings such as Mandiant ThreatSpace and tailored support for rural hospitals, we’re dedicated to helping agencies strengthen their cybersecurity posture and protect sensitive data. With a focus on continuous innovation and collaboration, Google Cloud Security is your trusted partner in building a more secure and resilient future for the public sector.

If you’re attending our Google Public Sector Summit, you can learn more about what Google Cloud Security has to offer by visiting us today at the Security Station on the show floor, along with attending our session, “Protecting Critical Infrastructure and Resources,” at 1:45 p.m. ET.

The majority of organizations don’t feel ready for the AI era. In fact, 62% say they don’t have the expertise they need to unlock AI’s full potential.1 As the leader of learning for Google Cloud, the only thing that surprises me about that number is how low it is. I meet with customers every day, and 100% of them flag some kind of AI skills gap. 

Here’s the good news: that makes you — the developers, machine learning engineers, and data experts — invaluable. You are exactly the talent these organizations are looking for — but with the rate of change in AI, you have to stay on the cutting edge. A 2024 survey estimated that about 70% of AI talent needs to update their skills.2 And yet, many technical professionals don’t have the training they need to move from theory to practice and integrate AI into their everyday work. 

That’s why, today, I’m proud to share with you the latest learning offerings on generative AI from Google Cloud Skills Boost. Say hello to four new learning paths designed to equip developers with real-world generative AI skills to build applications, manage and secure machine learning models, generate impactful content, and analyze data like a pro. We’re talking in-depth courses that first guide you through building proficiency and then, ultimately, test your skills in a real-life challenge lab.

Get practical experience with gen AI use cases

Generative AI is powerful, but to actually see value from this technology in real-world use cases takes practical experience and technical knowledge. These new learning paths from Google Cloud, listed below, give you the generative AI skills you need to complete innovative work in your current roles — like improving customer experience or team productivity. This also opens up new career (and promotion) opportunities. Once you complete the hands-on training, you will receive a skill badge to showcase your expertise on your resume or social media channels: 

Learning Path: Build and modernize applications with generative AI: Learn how to enhance your projects and build end-to-end applications on Google Cloud with the power of generative AI. This path will guide you through essential techniques and tools to integrate gen AI capabilities seamlessly into your development workflow.

Learning Path: Integrate generative AI into your data workflow: Learn how to use BigQuery Machine Learning for inference, work directly with Gemini models in BigQuery, and enhance your data team’s efficiency with Gemini’s assistance. This path features a brand new course on boosting productivity with Gemini in BigQuery to aid in the data-to-AI pipeline.

Learning Path: Deploy and manage generative AI models: Learn how to manage the entire lifecycle of generative AI models, from development and deployment to monitoring — including introductions to responsible AI for developers. This path features a brand new course on security for AI models.

Learning Path: Generate smarter generative AI outputs: Learn how to build applications that generate text and visual content using generative AI. Develop an AI project on Google Cloud, use diffusion models for image generation, and build search applications with Vector Search and embeddings, then dive deeper into multimodal prompts and Multimodal RAG with Gemini.

Innovators get full, no-cost access to gen AI learning paths

Last year, we announced that every member of the Google Cloud Innovators community, our no-cost developer program, receives 35 unrestricted learning credits every month to use on courses and labs in Google Cloud Skills Boost. That’s enough to complete one of these new learning paths every month. Join Innovators today to activate your credits and start learning!

Innovators also get access to exclusive learning opportunities — like our latest challenge, AI Skills Quest, where you can immerse yourself in hands-on labs and earn skill AI badges alongside a cohort of like-minded peers.

The latest gen AI learning content — from security to productivity 

While you’re sharing your AI skills in the Innovators community, why not check out the very latest in learning? These new learning paths feature three brand new courses in security, data analytics and agent building — all hot off the presses in Google Cloud Skills Boost: 

Course: Introduction to security in the world of AI. Whether you’re a security engineer, an IT leader, an AI developer, or a less technical leader, this course will help you build a foundational understanding of Google’s approach to navigating the intersection of AI and security. 

Course: Boost productivity with Gemini in BigQuery. Accelerate your data-to-AI pipeline, write better code, and visualize workflows easily with Gemini in BigQuery. This course equips you or your team with practical skills to boost productivity and unlock the full potential of your data.

Course: Build generative AI agents with Vertex AI and Flutter. Whether you’re a seasoned app developer or just starting your journey with Flutter and Python, this course will help you build intelligent chat agents and fun, interactive experiences through generative AI. 

Go from gen AI theory to practice

Google has been at the forefront of AI innovation for over a decade. These learning paths are your direct line to that expertise, crafted by the very people who shaped the field. My team is committed to empowering you with the skills to lead in this exciting era. Because ultimately, we all have a shared goal: to build AI-driven solutions that are responsible, fun to use, and — above all — genuinely improve people’s lives. 

Join the Google Cloud Innovators program, get your 35 free credits every month, and dive head first into our new generative AI learning paths today.

1.  Help Net Security, “The cloud skills gap is digital transformation’s Achilles’ heel,” Nov 14, 2023
2. Pluralsight, “Pluralsight AI skills report,” 2024

Editor’s note: Today, we hear from Parthasarathy Ranganathan, Google VP and Technical Fellow and Amber Huffman, Principal Engineer. Partha delivered a keynote address today at the 2024 OCP Global Summit, an annual conference for leaders, researchers, and pioneers in the open hardware industry. Amber is on the board of directors at the Open Compute Project (OCP). Read on to hear about the past and future of hyperscale computing, and an overview of all of our activities in the OCP community.

We are in an exciting era of hyperscale computing, one where a new wave of innovations is building the foundation for AI/ML computing in the cloud. Building on Google’s rich 25-year history in hyperscale computing, we look ahead to how co-design and collaboration — across the hardware-software stack, disciplines, and communities — will be key to this exciting new future. 

From scrappy beginnings to societal infrastructure

When Google was founded in 1998, it was clear that successful web search would require enormous amounts of computing power and storage. This led to the design of the very first hyperscale computers specialized for search. These early makeshift systems included creative cost-reduction approaches like corkboard servers and off-the-shelf fans from Walmart, and they set the stage for the hardware-software co-design and workload-specific specialization principles that we follow to this day. 

Building on these first systems, over the subsequent decade, Google laid the groundwork for modern hyperscale computing, pioneering custom servers, custom networking, and custom data centers, and expanding our services beyond search to include Gmail, YouTube, and Android. All of this presaged the modern multi-workload cloud. During this period, we also developed essential systems software like Borg, Colossus, MapReduce, and Bigtable. In the following years, we focused on scaling these systems, while also prioritizing security, reliability, and power efficiency. The formation of the Open Compute Project (OCP) in 2011 marked the transition of hyperscale computing from niche discipline to more mainstream offering. In the current decade, hyperscale computing is characterized by innovations to counter the slowing of Moore’s law: specialized hardware to support machine learning and video processing as well as software-defined servers to manage heterogeneity. 

Today, hyperscale computing has truly come into its own, evolving into the crucial societal infrastructure that drives cloud and AI workloads.

Cross-disciplinary co-design: the heart of innovation

Across all these Google innovations over the past 25 years, one theme has remained constant: a strong commitment to cross-disciplinary systems innovation and co-design. Looking ahead to the AI era, we continue to take a holistic approach: from “mud to cloud” — starting at the very ground on which we build our data centers up to to broader cloud computing services; and from “chip to ship” — designing hardware that we then deploy and use in production. This philosophy has driven some incredible efficiency gains, delivering orders-of-magnitude improvements across multiple generations of systems.

Take our Tensor Processing Units (TPUs). Multiple generations of these purpose-built AI accelerators (including our latest Trillium TPU) have driven significant advances in machine learning, including large-language models like Gemini and Nobel-prize-winning scientific breakthroughs like AlphaFold. However, we’ve gone beyond just chip design to considering the entire system that surrounds them. We’ve coupled TPUs with innovations like liquid cooling, advanced networking systems featuring cutting-edge optics and topology awareness, and a commitment to sustainable power, all in the service of creating a truly amazing AI platform. We’ve then layered open software frameworks like JAX, TensorFlow, OpenXLA, and Kubernetes on top of this hardware foundation, creating what we call the AI Hypercomputer. This hypercomputer is further enhanced by integrating with model gardens and applications, creating a vertically integrated ecosystem that’s optimized for AI workloads.

Cross-industry collaboration: from ideas to impact

But there’s also another aspect of holistic co-design that has served us well: cross-industry collaborations, i.e., building standards and ecosystems. Our partnership with OCP is an important example of this. Since formally joining OCP in 2016, we’ve continued to grow our contributions year after year. Looking ahead, we want to highlight progress and opportunities in four key areas.

Sustainability
Last year, Google, along with fellow hyperscalers, rallied the industry to reduce carbon emissions with an ambitious roadmap towards greener concrete. We have since made good progress, collaborating to develop new metrics and benchmarks, identifying streamlined data center designs that minimize concrete use, and even using AI to research new materials. At a recent event, we demonstrated proof-of-concept concrete mixtures that can reduce carbon emissions by 20% to 40%. 

As we work towards net-zero emissions by 2030 across our operations and value chain, there’s a lot more we can do. At OCP this year, we are discussing how to develop product category rules (PCRs) to accurately measure hardware emissions across the lifecycle, make more high-quality carbon data available, and develop clean reliable power backup for our data centers. Further, we’re continuing to look holistically at all aspects of our energy consumption, carbon footprint, and water usage. 

Trusted silicon
Trusted silicon is a foundational element of hyperscaler systems. Over the past three years, we have collaborated on Caliptra, a re-usable IP block for root-of-trust management, and delivered an open-source implementation of Caliptra 1.0 that is being integrated by companies across the ecosystem. Google’s future TPUs and ARM SoCs will also include Caliptra. Leveraging Caliptra, the OCP L.O.C.K. project will provide layered open-source cryptographic key management for storage devices, improving both trust and sustainability. 

In the area of silicon reliability, we are continuing our industry-academia collaborations around a systems approach to addressing silicon faults and silent data errors, including funding six leading academic institutions for novel research. The Server Component Resilience (SDC) Specification discusses the opportunities ahead with standardized information exchange and test metrics and open frameworks for detecting and mitigating errors. 

AI accelerators
AI represents a fundamental platform shift requiring us to innovate across hardware and software. Google has played an active role in driving standardization efforts for AI accelerators, particularly in areas like low-precision data formats (e.g., OCP FP8 and MX), software frameworks (e.g., OpenXLA, JAX, TensorFlow), and networking (Falcon, Ultra Ethernet, Ultra Accelerator Link). Working with other hyperscalers and GPU suppliers, we have also aligned on common specifications for firmware updates, management interfaces, and RAS (reliability, availability, serviceability). 

But as AI continues to drive exponential demands on computing, we can do more. As part of the OCP AI Strategic initiative, we are sharing learnings from deploying over 1 GW of liquid cooled infrastructure to help the industry scale this capability. We are also identifying new power-delivery solutions, from chips to racks to data centers. Notably, akin to how Google led the industry with 48V racks, at OCP Summit this year, we are proposing 400V DC distribution and rack solutions that can significantly improve data center density and efficiency.

Systems infrastructure
Finally, we continue to make great progress on foundational systems infrastructure. Google’s contributions this past year span contributions to NVM Express for the data center (e.g., security enhancements, open test repositories), servers (e.g., OpenTitan platform root of trust), and networking (Falcon, SONiC advancements in telemetry and simulation, advanced PCIe enclosure compatible form factor), as well as new efforts such as the open-source random shock and vibration testing. At the same time, we’ve gone beyond technical contributions to form and co-chair the OCP Advisory Board as well as guide the formation of the OCP AI Strategic Initiative.

Looking ahead, we will continue to keep innovating in this space, particularly to meet the next level of scale required by AI infrastructure. Notably, at the OCP Summit this year, we are discussing the adoption of robotics and automation for data centers. Across a range of activities (material movement, monitoring/inspection, servicing/repair, media management), robotics enable data center operations to scale safely and sustainably, and present a fundamental shift in how we build these facilities. 

Innovating for the new intelligence revolution

We have a lot to be proud of over the past 25 years of hyperscale computing, but the best is yet to come. With AI, we are at an exciting inflection point in computing: the beginning of the new intelligence revolution. Akin to prior shifts — the industrial revolution for manufacturing or the information revolution with the mobile internet — this revolution will have a profound impact on both technology and society, and holistic system innovations will be key to enabling it. We look forward to collaborating with all of you on this exciting journey.

In the dynamic landscape of commerce, Shopify merchants rely on our platform’s ability to seamlessly and reliably deliver highly relevant products to potential customers. Therefore, a rich and intuitive search experience is an essential part of our offering.

Over the past year, Shopify has been integrating AI-powered search capabilities into our merchants’ storefronts. Shopify Storefront Search has transformed the way consumers can shop online. With Semantic Search, we went beyond keyword matching. We improved our understanding of the intent behind a consumer’s search, so that we could match a search with the most relevant products.

The net result is helping our merchants boost their sales while offering positive interactive experiences for their consumers. It’s a win-win!

Building ML assets with real-time Embeddings

Around the same time, Shopify also started investing in creating foundational machine learning (ML) assets. These assets are built as a shared repository of ML primitives which are used as reusable building blocks for more sophisticated AI systems. Shopify Storefront Search is the perfect use case for these ML assets. Complex systems like this need primitives that can transform both text and images into data formats it can process.

How do we do that?

Enter embeddings, which translate textual and visual content into numerical vectors in a high-dimensional space. This transformation allows us to measure the similarity between different pieces of content, whether text or images, enabling more accurate and context-aware search results.

Now that we have a clear idea of the ML primitives that Shopify Storefront Search relies on, let’s take a look at how we send these embedding updates to their systems.

Designing ML inference streaming pipelines

Currently, Shopify processes roughly 2,500 embeddings per second (or roughly 3.6 million per day) across our image and text pipelines in near real time. These embeddings include net-new or updated content from across our merchants. Since most of our data platform infrastructure is built on BigQuery, we use Google Cloud’s streaming analytics service Dataflow to power these pipelines. We chose Dataflow for its native streaming capabilities, easy integration with the overall Google Cloud ecosystem, and its ability to scale with our ever-growing AI and data needs.

The high-level architecture of our design is quite simple. Let’s walk through the image embeddings streaming pipeline as an example.

At startup, the embedding model is loaded.

The pipeline listens to new events from an input event topic that signals an image has been created or modified on a merchant’s website.

The new event is then preprocessed before running inference:

Download the image to the worker machine
Load the image to memory
Resize the image

Next, the embedding vector is generated for the image.

Some final postprocessing is applied to the embeddings.

The embedding is written to:

Data warehouse for offline analysis (reports, dashboards)
An output event topic for downstream real time ingestion (Shopify Storefront Search)

At this point, the curious reader may wonder why we chose to go with near real-time embeddings vs. a simpler batch solution. Doesn’t this increase the complexity of our pipelines? Merchants and their customers expect a seamless experience from Shopify’s platform. When a merchant edits their products or uploads a new image, they want these updates to be available on their website instantly. Additionally, the ultimate objective is to boost sales for our merchants and offer pleasant interactive experiences for their consumers. Our data suggests that up-to-date embeddings achieved through a streaming pipeline allows us to optimize for this, despite the additional complexity it incurs when compared with a batch solution.

Challenges of maintaining a streaming pipeline

The reality of maintaining a streaming pipeline can be quite challenging. Adding GPU-accelerated inference to the mix further raises the complexity. As a result, we constantly have to make meaningful trade-offs between cost, throughput and latency. Let’s go through a few examples of these technical decisions.

1. Managing data in memory
When deploying the image embedding pipeline, our initial choice of worker machines on Dataflow was the n1-standard-16 mounted with a T4 GPU. However, we started running into Out of Memory (OOM) errors. Images consume a significant amount of memory so this wasn’t surprising.

Our first solution was to increase the memory of the workers. So we switched to n1-highmem-16 machines, which increased the memory on the worker from 60GB to 104GB. This worked for quite some time but it also increased our costs by 14%. Ultimately, we needed a better long-term solution.

Thus began our journey to get a better understanding of Dataflow internals. To maximize the utilization of the worker’s hardware, Dataflow’s python runner will typically create one process per core on the machine. Therefore, on an n1-highmem-16, the runner spawns 16 processes for the highest possible parallelism on the machine. Furthermore, to boost concurrency, each of those processes creates 12 threads by default. Each thread is then assigned elements to run inference.

With 16 processes, this meant that we had 16 x 12 = 192 threads concurrently processing elements. This translates to roughly 192 concurrent images in memory (in reality, this gets a bit more complicated when we consider that elements are grouped into bundles, but we can ignore this for now).

Luckily, the Dataflow Runner allows us to set number_of_worker_harness_threads to control this. Doing some quick napkin math, we estimated that lowering the thread count to 4 would theoretically reduce our memory footprint by 3x (i.e. 4 x 16 = 64 threads; 192 / 64 = 3). Lowering the thread count also naturally reduces the throughput of a pipeline’s preprocessing steps due to lower concurrency. Since the pipeline is already GPU-bound, we were accumulating more images than the GPU could process. As a result, the inference step’s throughput was not significantly impacted. 

What is the result of this change? Beforehand, we were consistently hovering around the high-mem machine’s max memory of 104 GB. After the change, it’s now closer to 40 GB: ~2.6x memory footprint decrease! As a result, we switched back to n1-standard-16 machines, eliminating the extra 14% in cost in the process! 

2. Managing the model in memory
Apache Beam provides two concepts to enabling running inference in a batch or streaming pipeline:

ModelHandler defines the ML model that will be used for inference.

RunInference is the transform that generates embeddings based on the ModelHandler.

By default, each process on the worker loads its own instance of the embedding model to the GPU. Recall that we are using n1-standard-16. Therefore, the model was being loaded 16 times into the GPU’s memory. This increases throughput as more instances of the model can run in parallel but also increases memory consumption.

Initially, we explored reducing the model’s memory footprint by determining if we could control how the model is loaded to memory. The most straightforward solution to this is to enable the ModelHandler to share the model across processes.

This essentially loads the model once into a shared process. The python processes on the VM will then query it for inference.

This leads to a significant decrease in memory consumption but the throughput is also significantly degraded due to lower parallelism. We abandoned this approach due to the lower throughput.

Another option was to see if we can control how many python processes are created on each worker. Unfortunately, at the time of writing this, this is not currently possible. The only option is to enable experiments=no_use_multiple_sdk_containers. This forces the Dataflow Runner to create only one process per worker but leads to a significant degradation in throughput, which we opted against.

Ultimately, our embedding models are small enough that we don’t need to worry about this too much. We eventually opted to stick with Dataflow’s default configuration.

3. Batching
While CPU utilization above 80% should typically be avoided where possible to avoid thrashing, GPU utilization should still ideally be as close to 100% as possible. This may seem surprising initially but this is because switching between threads on a GPU Streaming Multiprocessor (SM) is basically free.

Due to the GPU’s high throughput, the data transfer between a host (CPU) and a device (GPU) is almost always the main bottleneck in a streaming data pipeline (in reality the data transfer between the GPU’s global memory and the SMs running the kernels is also a major bottleneck but this is beyond the scope of this blog). This is where batching becomes crucial to effectively saturate the workers’ GPU.

Thankfully, Apache Beam offers helpful semantics to batch data in your streaming pipeline. By overriding the batch_element_kwargs method in our Model Handler, we were able to define our desired batches to send to the GPU.

So we’re done right? Not so fast.

We are still sending batches of 1 to the GPU. What gives? 

RunInference uses the BatchElements transform under the hood to do batching. It essentially has two methods of batching data. It can attempt to batch data within the existing bundle or across bundles using a stateful implementation.

Bundles are a Dataflow concept to manage parallelism. Essentially, a bundle is a collection of elements that are processed together. This is determined by the Dataflow Runner — the user has no direct control of this value.

In our case, due to the burstiness of the pipelines’ input topics, elements were being organized in bundles of 1, meaning Dataflow was unable to batch the elements in any meaningful way. On the other hand, stateful batching (by adding max_batch_duration_secs) attempts to guarantee that you get your desired batch size but also incurs higher latency. This is mainly because the Dataflow Runner forces a shuffle. 

In the end, we are still able to saturate the GPU due to Dataflow loading a model instance per process (as we discussed previously). This creates enough parallelism to offset the higher number of data transfers between the host (CPU) and device (GPU). As a result, we chose to stick with the in-bundle batching option because the latency cost was too expensive. This is an area we are still actively investigating for best practices. 

Conclusion

Shopify’s real-time embedding pipelines, which deliver text and image embeddings as centralized ML assets, are a powerful building block for innovation, enabling us to continue building new and exciting products. Semantic Search is just one awesome example of this. We are currently partnering with a few internal teams at Shopify to identify what problems might be worth solving and what centralized ML asset we could build to help solve them.

What’s next?

Curious to learn more about Machine Learning at Shopify? Check out this blog on Building Shopify Inbox’s Message Classification Model.

Want to get started with Dataflow ML? Check out this docs page.

In today’s data-driven landscape, data unification plays a pivotal role in ensuring data consistency and accuracy across an organization. Reltio, a leading provider of AI-powered data unification and management solutions, recently undertook a significant step in modernizing its data infrastructure by migrating from self-managed Cassandra to Spanner on Google Cloud. The outcome of this endeavor showcases amplified performance capabilities and seamless scalability.Reltio provides AI-powered data unification and management solutions that deliver unified, trusted data where and when it’s needed so that enterprises can be responsive to changing business needs. Powered by the Reltio Connected Data Platform, Reltio’s offerings — 360 Data Products, Multidomain Master Data Management (MDM), and Entity Resolution — unify disparate data sources in milliseconds, creating a single source of truth for decision making. Reltio delivers AI-ready, high-quality data in real time, with built-in scalability, security, and flexibility. Leading enterprises across industries around the globe rely on Reltio to accelerate growth, drive efficiency, and manage risk.

Given such demanding workloads, Reltio needed a database that can scale, has low latency, and provides the reliability that the Reltio Connected Data Platform needs. Migration should also be straightforward. Reltio chose Spanner which satisfied all of its requirements.

“Implementing our logic for Spanner was incredibly straightforward,” said Reltio’s Principal Architect, Ivan Kliavin. “The code we wrote to support Spanner is simpler and more intuitive than for any other database we’ve worked with.”

Why Spanner Proved to be the Ideal Choice

Seamless Scalability and Performance: Spanner’s ability to scale effortlessly and seamlessly adapt to fluctuating workloads empowered Reltio to adapt to varying demands, ensuring optimal performance at all times.

Enhanced Reliability and Availability: Compared to self-managed Cassandra, Spanner offered built-in reliability enhancements that provide peace of mind with its high availability guarantees (99.99% for regional and 99.999% for multi-regional).

Unlocking Innovation: Spanner’s rich feature set, including integrated search capabilities and seamless integration with other Google Cloud services, paves the way for Reltio to explore new features and elevate their data unification offerings. ACID transactions provide unparalleled data consistency, while multi-regional instances enable greater availability for their services

Simplified Operations: Spanner’s fully managed nature eliminates the overhead of managing infrastructure, achieving streamlined operations and significant cost reductions.

Reltio’s Spanner Architecture and Migration Plan

Let’s take a closer look at the technical architecture and migration strategy that enabled Reltio’s smooth transition to Cloud Spanner.

Tenant Security and Isolation

Each tenant’s data can be  stored in a dedicated Spanner database, ensuring complete physical data plane isolation.

Access to each database is tightly controlled utilizing Spanner’s robust security framework. 

Compute plane utilizes a service account with permissions to ephemerally assume roles for seamless data access.

Autoscaling and Capacity Management

A hybrid autoscaling model leverages both automated scaling based on CPU and storage metrics, along with manual scaling triggered by client-side latency thresholds.

This approach ensures optimal performance and resource utilization while safeguarding against over-scaling due to badly throttled tenants.

Backup, Restore, and Disaster Recovery

Point-in-time recovery (PITR) backups are implemented using database export/import, stored in Google Cloud Storage with a custom lifecycle configuration for cost efficiency.

Regional disaster recovery backups are stored in secondary Spanner instances located in backup regions, enabling fast recovery in the event of a regional outage.

Tenant Provisioning and Monitoring

A Cloud Function automates tenant provisioning, service account creation, and instance assignment.

Another Cloud Function monitors tenant usage and instances, optimizing autoscaling configurations and cleaning up unused resources.

Throttling and Noisy Neighbor Prevention

Reltio’s internal traffic-shaping, based on credits, is adapted for Spanner to ensure fair resource allocation across tenants in shared instances.

The inbuilt Spanner autoscaler function analyzes per-database metrics to prevent over-provisioning and protect against workloads from one tenant impacting others.

This is a great example of resource management at the application and database layers working in harmony to deliver a consistent experience to the customer, and is a best practice for a well-architected application.

A Model for Success

Reltio’s success exemplifies the power of a joint design, well-executed database migration, and deep operational collaboration between application and data layers. Their experience highlights the possibilities for organizations seeking to modernize their data infrastructure, achieve greater agility, and unlock new opportunities for growth and innovation.

“Reltio” is a registered trademark, and “Reltio Connected Data Platform” is a trademark of Reltio, Inc. All other trademarks are property of their respective owners. All Rights Reserved.

We recently hosted our annual Google Cloud Startup Summit, and we were thrilled to showcase a wide range of AI startups leveraging Google Cloud, including Higgsfield AI, Click Therapeutics, Baseten, LiveX AI, Reve AI, and Vellum. As a former co-founder and venture capitalist, I was inspired by the remarkable solutions these startups are developing and the rapid pace at which they are transitioning from experimentation to production with our AI-optimized technology stack. You can watch the event on demand or tune into our sessions in Asia-Pacific region (October 17).

During the Startup Summit keynote, Google Cloud CEO Thomas Kurian and I discussed our ongoing commitment to help startups grow. Building on Alphabet’s latest infrastructure and AI advances, startups can accelerate AI development with the choice of more than 160 models, including first-party, third-party, and open models. They can leverage our single, integrated development platform, Vertex AI, harness the computing potential of both TPUs and GPUs, access Google’s AI-optimized infrastructure, and more. 

We’re proud that more than 60% of funded gen AI startups and nearly 90% of gen AI unicorns are Google Cloud customers. 

Here are the top five takeaways we heard from founders at the summit:

Build with the best infrastructure: During the “Driving impact with Google Cloud and NVIDIA” session, Jia Li (co-founder and chief AI officer at LiveX AI) highlighted the impact of using Google Cloud and NVIDIA technologies, such as Google Kubernetes Engine (GKE), NVIDIA A100 GPUs, and NVIDIA NIM microservices to scale their LiveX AI Agent and ChurnControl products effectively. This approach has enabled LiveX AI to reduce operational costs by approximately 66%. Working with partners like NVIDIA, Intel, and AMD, we provide customers with the widest range of AI-optimized compute options across TPUs, GPUs, and CPUs to provide performance and cost advantages for training and serving models. 
Stay flexible with Google Cloud’s open platform: During the “Choosing the right gen AI partner,” Akash Sharma, CEO at Vellum, talked about the value of having a diverse array of first-party, third-party, and open models available in Vertex AI to choose from depending on their unique needs. He also mentioned that an end-to-end platform that covers a range of technical requirements can speed up time to market. Our platform, Vertex AI, is built with an open and flexible approach, providing a coherent end-to-end solution to support the entire model-to-production lifecycle. Our open approach enables startups like Anthropic, Mistral, Cohere, and many others to connect with the developer ecosystem by bringing their models to Vertex AI. Startups are also utilizing our platform to build applications, leveraging and customizing the best foundation models from Google and the industry, including Gemini.
Accelerate development with Vertex AI: At the session “From idea to impact with Google Cloud & Workspace,” Alex Mashrabov, CEO at Higgsfield AI, talked about his company’s unique technical needs. Higgsfield AI utilizes Vertex AI and GPUs to quickly train its platform for gen-AI videos across a range of types, but especially personalized videos that were previously expensive and cumbersome to produce. In training, Gemini is the only option they’ve found that can capture the full expressiveness, action, and motion in their videos versus doing a simple frame-by-frame analysis. Our integrated development platform, Vertex AI, accelerates the creation of sophisticated generative AI agents and experiences. Built on world-class infrastructure, it uniquely allows customers to discover, tune, augment, and manage their AI agents seamlessly.
Strengthen data security: Startups can leverage Google Cloud’s advanced security measures to enhance their data security. During the session “From idea to impact with Google Cloud & Workspace,” Han Chiu, CTO of Click Therapeutics, noted how important data security is for his company. He mentioned that Click Therapeutics leverages Google Cloud’s intuitive solutions that are aligned with the company’s day-to-day workflows to help to improve the company’s data security posture.
Connect, collaborate, and scale: Google Cloud is committed to empowering startups through AI education and collaboration. We just launched Startup School: Gen AI, new Startup Hubs in London and São Paulo, and introduced programs to boost partner-sales and go-to-market capabilities. This includes the ISV Startup Springboard, partnerships with Y Combinator, and collaborations with top accelerators and incubators around the world such as 500 Global, StartX, Berkeley Skydeck and more.

We’re thrilled to continue supporting startups in their journey toward leveraging generative AI, and I look forward to seeing the innovative solutions that emerge from this vibrant community. Check out the Google for Startups Cloud Program to learn more about how Google Cloud can accelerate your startup journey.