Cloud

Distributed tracing is a critical part of an observability stack, letting you troubleshoot latency and errors in your applications. Cloud Trace, part of Google Cloud Observability, is Google Cloud’s native tracing product, and we’ve made numerous improvements to the Trace explorer UI on top of a new analytics backend.

The new Trace explorer page contains:

1. A filter bar with options for users to choose a Google Cloud project-based trace scope, all/root spans and a custom attribute filter.

2. A faceted span filter pane that displays commonly used filters based on OpenTelemetry conventions.

3. A visualization of matching spans including an interactive span duration heatmap (default), a span rate line chart, and a span duration percentile chart.

4. A table of matching spans that can be narrowed down further by selecting a cell of interest on the heatmap.

A tour of the new Trace explorer

Let’s take a closer look at these new features and how you can use them to troubleshoot your applications. Imagine you’re a developer working on the checkoutservice of a retail webstore application and you’ve been paged because there’s an ongoing incident.

This application is instrumented using OpenTelemetry and sends trace data to Google Cloud Trace, so you navigate to the Trace explorer page on the Google Cloud console with the context set to the Google Cloud project that hosts the checkoutservice.

Before starting your investigation, you remember that your admin recommended using the webstore-prod trace scope when investigating webstore app-wide prod issues. By using this Trace scope, you’ll be able to see spans stored in other Google Cloud projects that are relevant to your investigation.

You set the trace scope to webstore-prod and your queries will now include spans from all the projects included in this trace scope.

You select checkoutservice in Span filters (1) and the following updates load on the page:

• Other sections such as Span name in the span filter pane (2) are updated with counts and percentages that take into account the selection made under service name. This can help you narrow down your search criteria to be more specific.

• The span Filter bar (3) is updated to display the active filter.

• The heatmap visualization (4)  is updated to only display spans from the checkoutservice in the last 1 hour (default). You can change the time-range using the time-picker (5). The heatmap’s x-axis is time and the y-axis is span duration. It uses color shades to denote the number of spans in each cell with a legend that indicates the corresponding range.

• The Spans table (6) is updated with matching spans sorted by duration (default).

• Other Chart views (7) that you can switch to are also updated with the applied filter.

From looking at the heatmap, you can see that there are some spans in the >100s range which is abnormal and concerning. But first, you’re curious about the traffic and corresponding latency of calls handled by the checkoutservice.

Switching to the Span rate line chart gives you an idea of the traffic handled by your service. The x-axis is time and the y-axis is spans/second. The traffic handled by your service looks normal as you know from past experience that 1.5-2 spans/second is quite typical.

Switching to the Span duration percentile chart gives you p50/p90/p95/p99 span duration trends. While p50 looks fine, the p9x durations are greater than you expect for your service.

You switch back to the heatmap chart and select one of the outlier cells to investigate further. This particular cell has two matching spans with a duration of over 2 minutes, which is concerning.

You investigate one of those spans by viewing the full trace and notice that the orders publish span is the one taking up the majority of the time when servicing this request. Given this, you form a hypothesis that the checkoutservice is having issues handling these types of calls. To validate your hypothesis, you note the rpc.method attribute being PlaceOrder and exit this trace using the X button.

You add an attribute filter for key: rpc.method value:PlaceOrder using the Filter bar, which shows you that there is a clear latency issue with PlaceOrder calls handled by your service. You’ve seen this issue before and know that there is a runbook that addresses it, so you alert the SRE team with the appropriate action that needs to be taken to mitigate the incident.

Share your feedback with us via the Send feedback button.

Behind the scenes

This new experience is powered by BigQuery, using the same platform that backs Log Analytics. We plan to launch new features that take full advantage of this platform: SQL queries, flexible sampling, export, and regional storage.

In summary, you can use the new Cloud Trace explorer to perform service-oriented investigations with advanced querying and visualization of trace data. This allows developers and SREs to effectively troubleshoot production incidents and identify mitigating measures to restore normal operations.

The new Cloud Trace explorer is generally available to all users — try it out and share your feedback with us via the Send feedback button.

From transcribing customer calls and meetings, to analyzing research interviews and creating accessible content, audio transcription plays a vital role in extracting insights from spoken data. Our partners are collaborating with clients across industries to implement transcription solutions that enhance efficiency, accessibility, and data-driven decision-making.

Traditional audio transcription methods, such as manual transcription or basic speech-to-text tools can be time-consuming, error-prone, and expensive. In this blog post, we show how Gemini offers a cutting-edge solution for scalable audio transcription by automating the process and delivering results with high accuracy at a fast pace – all in a cost-effective way. 

The challenges of scaling audio transcription

As organizations scale their transcription needs, they might encounter challenges such as increasing costs, latency in handling large volumes of audio, and maintaining accuracy across diverse audio conditions. In particular, legacy solutions struggle with:

• Handling complex audio with multiple speakers, accents, or background noise.

• Maintaining accuracy in industry-specific terminology across healthcare, legal, and customer service domains.

• Adapting to multilingual needs, especially in global business environments.

• Optimizing processing time and cost, ensuring fast turnaround without excessive resource consumption.

A scalable solution must address these challenges efficiently, without compromising speed, accuracy, or customization — this is where Gemini excels.

How our partners put Gemini to work

Google Cloud Partners leverage audio transcription to help clients across various industries improve efficiency, compliance, and accessibility. Here are some examples:

• Media and entertainment: Transcribe interviews, podcasts, and webinars for content creation, and generate subtitles to enhance accessibility and engagement.

• Customer service: Transcribe customer calls in real time for quality assurance, sentiment analysis, and experience optimization. (Also see Conversational Insights within Customer Engagement Suite with Google AI).

• Legal and Compliance: Transcribe legal proceedings, contracts, and compliance-related communications to improve accuracy, streamline case management, and ensure regulatory adherence.

• Healthcare: Convert medical dictations and clinical notes into structured records for better documentation, electronic health record (EHR) integration, and regulatory compliance.

• Business and corporate: Transcribe meetings, interviews, and presentations to improve collaboration, knowledge sharing, and record-keeping.

Gemini redefines the possibilities of scalable audio transcription, offering a potent combination of advanced AI and seamless integration with Google Cloud. Here’s what sets it apart:

• Efficient processing of large datasets: Gemini can handle large volumes of audio data with ease, making it ideal for organizations with high-throughput transcription needs.

• Exceptional accuracy and contextual understanding: Backed by decades of Google research and development in speech recognition and natural language understanding, Gemini delivers highly accurate transcriptions that capture the nuances of conversations. This minimizes the need for manual review and correction, especially in cases with multiple speakers, accents, or challenging audio conditions

• Speaker diarization: Gemini can accurately identify and differentiate between speakers in an audio file, making it easier to follow conversations and attribute dialogue correctly

• Multilingual support: Gemini supports transcription in multiple languages and dialects, expanding its utility for global businesses and diverse content

• Customizable formatting: Gemini offers flexible formatting options, allowing users to tailor transcripts to their specific needs, including timestamps, speaker labels, and punctuation.

Introducing a differentiated solution

The Google Cloud Partner Engineering team worked together with System Integrators (SIs) to build a differentiated solution that allows customers to implement audio transcription at scale using Google’s Gemini on Google Cloud.

Gemini’s advanced multi-modal and reasoning capabilities have unlocked new possibilities for audio transcription. This solution allows audio files to be sent directly to Gemini for transcription. The reference architecture below illustrates how this solution is built;

This architecture demonstrates a robust and scalable approach to audio transcription using Gemini. It can be modified for any audio transcription use case. Here’s how it works:

1. File upload and sorting: The Upload Cloud Storage bucket is used to store source audio files like .wav, .mp3, .mp4, files etc. When these files are uploaded, Eventarc triggers the Sort Cloud Run function. This trigger event is passed using Cloud Pub/Sub.

The Sort Cloud Run function manages incoming files by sorting and filtering them based on their file types (e.g., .wav, .mp3). Depending on the file type, the files are then stored in either the Recordings Cloud Storage bucket or the Archive Cloud Storage bucket. 

2. Transcription: When audio files are placed in the Recordings Cloud Storage bucket, Eventarc uses Cloud Pub/Sub to trigger the Recording Cloud Run function. This Recording function then sends the audio files to the Gemini 1.5 Flash LLM model for audio transcription.

3. Gemini’s multi-faceted processing: Gemini performs three key tasks:

a. Analysis and formatting: It analyzes the audio file, extracting pertinent data and structuring it into JSON format based on the audio file schema. 

b. Transcription and summarization: Gemini transcribes the audio content into text and generates a concise summary. 

c. Output and evaluation: The summarized text is sent to a “TTS Output” Cloud Storage bucket, triggering the TTS Audio Generation function. This function executes a script from the “Golden Script” Cloud Storage bucket to generate sample audio, which is then used to evaluate the transcription quality against established metrics like Word Error Rate (WER), Character Error Rate (CER), Match Rate, etc.

This approach provides key benefits: dynamic scaling through a serverless, event-driven architecture (Cloud Run, Eventarc), simplified management via fully managed services (Cloud Storage), cost-effectiveness by consuming resources only when needed, and enhanced capabilities like advanced summarization and speaker diarization powered by Gemini.

Design considerations

When designing audio transcription applications and services on Google Cloud with Gemini, several factors are crucial for optimal performance and scalability:

1. Efficient audio file handling: Avoid loading large audio files directly into memory for serverless transcription on Google Cloud. Instead, use Google Cloud Storage URI to efficiently access and process audio without memory limitations.

2. Serverless function timeouts: To prevent premature termination when processing large audio files in Cloud run, increase the function timeout up to 60 minutes. Also set the Pub/Sub subscription acknowledgement deadline to 300 seconds for Eventarc.

3. Model selection and context window: For gen AI audio transcription, audio file size and duration dictate the model selection. Larger files and longer audio require models with large context windows like Gemini 1.5 Flash (1M tokens) and Gemini 1.5 Pro (2M tokens), overcoming prior LLM input limitations on the market today. The Gemini 1.5’s extended context window and near-perfect retrieval capabilities open up many new possibilities;

This shows that for audio transcription use cases, Gemini 1.5 Pro and Flash offer scalable audio transcription, processing up to 22 and 11 hours of audio respectively based on customer needs.

4. Optimizing speaker diarization: To leverage Gemini’s built-in speaker diarization capabilities effectively:

a. Use the latest Gemini SDK : Ensure your code utilizes the most up-to-date SDK for optimal diarization performance.

b. Design effective prompts: Craft prompts that clearly instruct Gemini on diarization and formatting requirements. The diagram below shows a code example of a diarization prompt

This sample code prompts Gemini to transcribe an audio file from Cloud Storage URI and displays the transcription.

5. Advanced diarization techniques: For complex scenarios with multiple speakers, accents, or overlapping speech,  design prompts efficiently to improve Gemini’s diarization accuracy. Consider separating diarization and transcription into separate functions, the snippet below shows an example of this;

From the screenshot above, the function highlighted in the red box is the prompt that instructs Gemini to transcribe. It also shows how we want the transcription to be formatted. This operation allows Gemini to focus first on transcribing the audio into text and summarizing it.

The transcription function is actually a straightforward function and zero shot prompt. For the diarization function, we recommend you design your prompt with a few short examples. The code block highlighted in blue shows the diarization function with some examples to help the model to diarize effectively and efficiently when there are multiple speakers.

6. Evaluating transcription quality: when building gen AI powered audio transcription systems on Google Cloud we recommend implementing a mechanism to evaluate the transcribed responses to further ensure better accuracy. Consider using tools like our Model Evaluation Service to assess and improve transcription quality.

Get started 

Ready to unlock the power of scalable audio transcription with Gemini? Explore Gemini’s API documentation and discover how easy it is to integrate its advanced capabilities into your solutions. By implementing the best practices and design considerations discussed in this post, you can deliver exceptional transcription experiences to clients and drive innovation across various industries. 

If you are an approved partner and require assistance, contact your Google Partner Engineer for deployment support.

Written by: Ashley Pearson, Ryan Rath, Gabriel Simches, Brian Timberlake, Ryan Magaw, Jessica Wilbur

Overview

Beginning in August 2024, Mandiant observed a notable increase in phishing attacks targeting the education industry, specifically U.S.-based universities. A separate investigation conducted by the Google’s Workspace Trust and Safety team identified a long-term campaign spanning from at least October 2022, with a noticeable pattern of shared filenames, targeting thousands of educational institution users per month.

These attacks exploit trust within academic institutions to deceive students, faculty, and staff, and have been timed to coincide with key dates in the academic calendar. The beginning of the school year, with its influx of new and returning students combined with a barrage of administrative tasks, as well as financial aid deadlines, can create opportunities for attackers to carry out phishing attacks. In these investigations, three distinct campaigns have emerged, attempting to take advantage of these factors. 

In one campaign, attackers leveraged phishing campaigns utilizing compromised educational institutions to host Google Forms. At this time, Mandiant has observed at least 15 universities targeted in these phishing campaigns. In this case, the malicious forms were reported and subsequently removed. As such, at this time none of the phishing forms identified are currently active. Another campaign involved scraping university login pages and re-hosting them on the attacker-controlled infrastructure. Both campaigns exhibited tactics to obfuscate malicious activity while increasing their perceived legitimacy, ultimately to perform payment redirection attacks. These phishing methods employ various tactics to trick victims into revealing login credentials and financial information, including requests for school portal login verification, financial aid disbursement, refund verification, account deactivation, and urgent responses to campus medical inquiries.

Google takes steps to protect users from misuse of its products, and create an overall positive experience. However, awareness and education play a big role in staying secure online. To better protect yourself and others, be sure to report abuse.  

Case Study 1: Google Forms Phishing Campaign

The first observed campaign involved a two-pronged phishing campaign. Attackers distributed phishing emails that contained a link to a malicious Google Form. These emails and their respective forms were designed to mimic legitimate university communications, but requested sensitive information, including login credentials and financial details.

The email is just the initial stage of the attack. While there are legitimate URLs contained within the phish, there is also a request to visit an external link to provide “urgent” information. This external link leads victims to a Google Form that has been tailored to the targeted university, including a color scheme in the school colors, a header with the logo or mascot, and references to the university name. Mandiant has observed the creation and staging of several different Google Forms, all with different methods employed to trick victims into providing sensitive information. In one instance, the social engineering pretext is that a student’s account is “associated with logins from two separate university portals”, a conflict which, if not resolved, will lead to interruption in service at both universities.

These Google Forms phishing campaigns are not just limited to targeting login credentials. In several instances, Mandiant observed threat actors attempting to obtain financial institution details.

Your school has collaborated with <redacted> to streamline fund 
distribution to students. <redacted> ensures the quickest, most 
dependable, and secure method for disbursing Emergency Grants 
to eligible students. Unfortunately, we've identified an outstanding 
issue regarding the distribution of your financial aid through <redacted>. 
We kindly request that you review and, if necessary, update your 
<redacted> information within the net 24 hours. Failing to address 
this promptly may result in delays in receiving your funds.

Figure 4: Example Google Form phish

After successfully compromising and propagating additional phishes using the compromised environment, the threat actor then uses the victim’s infrastructure to host a similar campaign targeting future victims. In some cases, the Google Form link was shut down and then repurposed to further the attacker’s objectives.

Case Study 2: Website Cloning and Redirection

This campaign involves a sophisticated phishing attack where threat actors cloned a university website, mimicking the legitimate login portal. However, this cloned website involved a series of redirects, specifically targeting mobile devices. 

The embedded JavaScript performs a “mobile check” and user-agent string verification and performed the following hex-encoded redirect:

if (window.mobileCheck()) {
	
window.location.href="x68x74x74x70x3ax2fx2fx63x75x74
x6cx79x2ex74x6fx64x61x79x2fx4ax4ex78x30x72x37";
		}

Figure 5: JavaScript Hex-encoded redirect

This JavaScript checks to determine if the user is on a mobile device. If they are, it redirects them to one of several possible follow-on URLs. These are two examples:

• hxxp://cutly[.]today/JNx0r7
• hxxp://kutly[.]win/Nyq0r4

Case Study 3: Two-Step Phishing Campaign Targeting Staff and Students

Google’s Workspace Trust and Safety team also observed a two-step phishing campaign targeting staff and students. First, attackers send a phishing email to faculty and staff. The emails are designed to entice faculty and staff to provide their login credentials in order to view a document about a raise or bonus.

Next, attackers use login credentials provided by faculty and staff to hijack their account and email phishing forms to students. These forms are designed to look like job applications, and phish for personal and financial information.

Understanding Payment Redirection Attacks

Payment redirection attacks via Business Email Compromise (BEC) are a sophisticated form of financial fraud. In these attacks, cyber threat actors gain unauthorized access to a business email account and exploit it to redirect payments meant for legitimate recipients into their own accounts. While these attacks often involve the diversion of large transfers, there have been instances where attackers divert small amounts (typically 5-10%) to lower the likelihood of detection. This outlier tactic allows them to steal funds gradually, making it more challenging to detect unauthorized transactions.

Initial Compromise: Attackers often begin by gaining access to a legitimate email account through phishing, social engineering, or exploiting vulnerabilities. A common phishing technique involves using online surveys or other similar platforms to create convincing but fraudulent login pages or forms. When unsuspecting employees enter their credentials, attackers capture them and gain unauthorized access.  

Reconnaissance: Once they have access to the email account, attackers closely monitor communications to understand the organization’s financial processes, the relationships with vendors, and the typical language used in financial transactions. This reconnaissance phase is crucial for the attackers to craft convincing fraudulent emails that appear authentic to their victims.  

Impersonation and Execution: Armed with the information gathered during reconnaissance, attackers impersonate the compromised user or create look-alike email addresses. The TA then sends emails to employees, vendors, or clients, instructing them to change payment details for an upcoming transaction. Believing these requests to be legitimate, recipients comply, and the funds are redirected to accounts controlled by the attackers.  

Withdrawal and Laundering: After the funds are diverted, attackers quickly withdraw or move the money across multiple accounts to make recovery difficult. The types of funds being stolen can vary widely and include financial aid such as FAFSA, refunds, scholarships, payroll, and other large transactions like vendor payments or grants. This diversity in targeted funds complicates efforts by organizations and law enforcement to trace and recover the stolen money, as each category may involve different institutions and processes.  

The Impact of Payment Redirection Attacks

The consequences of a successful payment redirection attack can be severe:

• Financial Losses: Organizations may lose substantial amounts of money, potentially running into millions of dollars, depending on the size of the transactions. 

• Reputational Damage: Clients and partners affected by these attacks may lose trust in the organization, which can harm long-term business relationships and brand reputation.  

• Operational Disruption: The aftermath of an attack often involves extensive investigations, coordination with financial institutions and law enforcement, and implementing enhanced security measures, all of which can disrupt normal business operations.  

Mitigating Payment Redirection Attacks

To protect against payment redirection attacks, Mandiant recommends a multi-layered approach focusing on prevention, detection, and response:

• Implement Multi-Factor Authentication (MFA): Requiring MFA for accessing email accounts adds an additional layer of security. Even if an attacker obtains a user’s credentials, they would still need the second factor to gain access, significantly reducing the risk of account compromise. Mandiant has observed many universities, which require MFA for current faculty/staff/students, but not for alumni accounts. While alumni accounts aren’t necessarily at risk of payment redirection attacks, Mandiant has identified instances where alumni accounts have been leveraged to access other user accounts in the environment.

• Conduct Employee Training: Regular training sessions can help employees recognize phishing attempts and suspicious emails. Training should emphasize vigilance against phishing forms hosted on platforms like Google Forms, and stress the importance of verifying unusual requests, especially those involving financial transactions or changes in payment details. If a Google Forms page seems suspicious, report it as phishing.

• Establish Payment Verification Protocols: Organizations should have strict procedures for verifying changes in payment information. For example, a policy that requires confirmation of changes via a known phone number or a separate communication channel can help ensure that any alterations are legitimate.  

• Use Canary Tokens for Detection: Deploying canary tokens, which are unique identifiers embedded in web pages or documents, can serve as an early warning system. If attackers scrape legitimate web pages to host them maliciously on their infrastructure, these tokens trigger alerts, notifying security teams of potential compromise or unauthorized data access.  

• Use Advanced Email Security Solutions: Deploying advanced email filtering and monitoring solutions can help detect and block malicious emails. These tools can analyze email metadata, check for domain anomalies, and identify patterns indicative of BEC attempts.

• Built-in Protections with Gmail: Employs AI, threat signals, and Safe Browsing to block 99.9% of spam, phishing, and malware, while also detecting more malware than traditional antivirus and preventing suspicious account sign-ins.

Detection

To assist the wider community in hunting and identifying activity outlined in this blog post, we have included a subset of these indicators of compromise (IOCs) in this post, and in a GTI Collection for registered users.

As a developer, there’s a lot to think about when you’re getting ready to launch an application. There’s the availability of the underlying database, of course, which stores application state, and determines how fast and you can recover if your application or web servers go down. Thankfully, if you’re running on Spanner, its 99.999% availability architecture means that application owners don’t have to worry about designing and operating active-active database deployments. While Spanner itself is designed to offer high availability, as with any other database, it’s important to have a checklist to ensure a successful launch of your workload as a whole and subsequent ongoing operation. We’ve captured a comprehensive list of items to keep in mind when launching workloads on Spanner in the Spanner Launch Checklist. In this blog post, we go over the key steps in the checklist, and provide additional color about each item. Once you’re done, you’ll be equipped to launch your Spanner-based application with confidence.

Let’s explore the key steps to focus on when gearing up for a Spanner launch.

1. Design, develop, test, optimize

Your Spanner schema and transactions are the blueprint for success. Designing for Spanner’s distributed architecture isn’t just a good idea — it’s the key to unlocking its full potential. Ensure your schema avoids bottlenecks and hotspots, and that transactions are tailored for minimal locking and maximum performance. But the real game-changer? Testing. Rigorous, real-world load testing, complete with traffic spikes and realistic concurrency, helps ensure your workload is ready for production scale and capable of handling unexpected spikes in traffic. 

2. Choose your deployments well

Spanner deployments entail more than just provisioning nodes; they’re about aligning infrastructure with your business goals. Choose configurations that balance availability and performance while accounting for traffic peaks with features like auto-scaling. Geography matters as well — place leader regions near your users for low-latency performance. And don’t forget the small stuff, like setting up tags and labels for resource tracking or warming up your database for a smooth start.

3. Implement backup and disaster recovery

Downtime isn’t an option when critical data is on the line. A solid backup and recovery strategy helps your business weather unexpected failures. Define recovery objectives, automate backups, and test restore procedures regularly. Features like incremental backups and Point-in-Time Recovery (PITR) make Spanner a reliable ally in your data protection arsenal.

4. Stay secure

Protecting your data goes beyond locking it up. Spanner’s security features, like fine-grained access controls and least-privilege permissions, let you limit access to only what’s necessary and for the right principals. Add an extra layer of defense with deletion protection and carefully designed IAM policies. With these safeguards, your Spanner instance remains both robust and secure.

5. Deploy logging, monitoring, and observability

To keep Spanner running smoothly, you need visibility. Audit logs, distributed tracing with OpenTelemetry, and real-time dashboards give you a clear view of what’s happening under the hood. Proactive alerts ensure you’re ready to act before small issues turn into big problems. We also invite you to check out Database Center, which provides a centralized view across your entire database fleet, including Spanner instances.

6. Optimize your configuration with the client library: 

Spanner’s client library is more than a connection — it’s a tool for optimization. Add transaction tags for enhanced debugging, fine-tune session pooling to match workloads, and tweak retry policies to handle errors with grace. These configurations boost both performance and resilience.

7. Cut costs without compromising performance 

Managing costs isn’t about cutting corners. Take advantage of committed use discounts for predictable savings and let auto-scaling handle spikes while minimizing idle resources. For non-production environments, scale down or shut off instances when they’re not needed. 

8. Plan your Migration

If you’re moving to Spanner, a well-planned migration makes all the difference. From orchestrating a seamless cutover to preparing a reliable fallback strategy, every step counts. Communicate with stakeholders, validate data, and monitor the process to ensure success.

Ready, set, scale!

Launching a successful workload or application is the result of careful planning, rigorous testing, and proactive management. And it’s no different for workloads running on Spanner.

Deploying mission-critical workloads on Spanner is about building a solid foundation for scalability, resilience and peak performance. A well-structured launch checklist ensures you’ve covered all the bases, from design and deployment to backup strategies and cost management. You can review the Spanner Launch Checklist in the Spanner documentation.

Generative AI diffusion models such as Stable Diffusion and Flux produce stunning visuals, empowering creators across various verticals with impressive image generation capabilities. However, generating high-quality images through sophisticated pipelines can be computationally demanding, even with powerful hardware like GPUs and TPUs, impacting both costs and time-to-result.

The key challenge lies in optimizing the entire pipeline to minimize cost and latency without compromising on image quality. This delicate balance is crucial for unlocking the full potential of image generation in real-world applications. For example, before reducing the model size to cut image generation costs, prioritize optimizing the underlying infrastructure and software to ensure peak model performance.

At Google Cloud Consulting, we’ve been assisting customers in navigating these complexities. We understand the importance of optimized image generation pipelines, and in this post, we’ll share three proven strategies to help you achieve both efficiency and cost-effectiveness, to deliver exceptional user experiences.

A comprehensive approach to optimization

We recommend having a comprehensive optimization strategy that addresses all aspects of the pipeline, from hardware to code to overall architecture. One way that we address this at Google Cloud is with AI Hypercomputer, a composable supercomputing architecture that brings together hardware like TPUs & GPUs, along with software and frameworks like Pytorch. Here’s a breakdown of the key areas we focus on:

1. Hardware optimization: Maximizing resource utilization

Image generation pipelines often require GPUs or TPUs for deployment, and optimizing hardware utilization can significantly reduce costs. Since GPUs cannot be allocated fractionally, underutilization is common, especially when scaling workloads, leading to inefficiency and increased cost of operation. To address this, Google Kubernetes Engine (GKE) offers several GPU sharing strategies to improve resource efficiency. Additionally, A3 High VMs with NVIDIA H100 80GB GPUs come in smaller sizes, helping you scale efficiently and control costs.

Some key GPU sharing strategies in GKE include:

• Multi-instance GPUs: In this strategy, GKE divides a single GPU in up to 7 slices, providing hardware isolation between the workloads. Each GPU slice has its own resources (compute, memory, and bandwidth) and can be independently assigned to a single container. You can leverage this strategy for inference workloads where resiliency and predictable performance is required. Please review the documented limitations of this approach before implementing and note that currently supported GPU types for multi-instance GPUs on GKE are NVIDIA A100 GPUs (40GB and 80GB), and NVIDIA H100 GPUs (80GB).
• GPU time-sharing: GPU time-sharing lets multiple containers access full GPU capacity using rapid context switching between processes; this is made possible by instruction-level preemption in NVIDIA GPUs. This approach is more suitable for bursty and interactive workloads, or for testing and prototyping purposes where full isolation is not required. With GPU time-sharing, you can optimize GPU cost and utilization by reducing GPU idling time. However, context switching may introduce some latency overhead for individual workloads.
• NVIDIA Multi-Process Service (MPS): NVIDIA MPS is a version of the CUDA API that lets multiple processes/containers run at the same time on the same physical GPU without interference. In this approach, you can run multiple small-to-medium-scale batch-processing workloads on a single GPU and maximize the throughput and hardware utilization. While implementing MPS, you must ensure that workloads using MPS can tolerate memory protection and error containment limitations.

2. Inference code optimization: Fine-tuning for efficacy

When you have an existing pipeline written natively in PyTorch, you have several options to optimize and reduce the pipeline execution time.

One way is to use PyTorch’s compile method, which enables Just-in-time (JIT) compiling PyTorch code into optimized kernels for faster execution, especially for the forward pass of the decoder step. You can do this through various backend computers such as NVIDIA TensorRT, OpenVINO or IPEX depending on the underlying hardware. You can also use certain compiler backends at training time. A similar JIT compilation capability is also available for other frameworks such as JAX.

Another way to improve code latency is to enable Flash Attention. By enabling the torch.backends.cuda.enable_flash_sdp attribute, PyTorch code natively runs Flash Attention where it is helpful in speeding up a given computation, while automatically selecting another attention mechanism if Flash isn’t optimal based on the inputs.

Additionally, to reduce latency, you also need to minimize data transfers between the GPU and CPU. Operations such as tensor loading, or comparing tensors and Python floats, incur significant overhead due to the data movement. Each time a tensor is compared with a floating-point value, it must be transferred to the CPU, incurring latency. Ideally, you should only load and offload a tensor on and off the GPU once in the entire image generation pipeline; this is especially important for image generation pipelines that utilize several models, where latency cascades with each model that is run. Tools such as PyTorch Profiler help us observe the time and memory utilization of a model.

3. Inference pipeline optimization: Streamlining the workflow

While optimizing code can help speed up individual modules within a pipeline, you need to take a look at the big picture. Many multi-step image-generation pipelines cascade multiple models (e.g., samplers, decoders, and image and/or text embedding models) one after the other to generate the final image, often on a single container that has a single GPU attached. 

For Diffusion-based pipelines, models such as decoders can have significantly higher computational complexity and hence take longer to execute, especially as compared with embedding models, which are generally faster. That means that certain models can cause a bottleneck in the generation pipeline. To optimize GPU utilization and mitigate this bottleneck, you may consider employing a multi-threaded queue-based approach for efficient task scheduling and execution. This approach enables parallel execution of different pipeline stages on the same GPU, allowing for concurrent processing of several requests. Efficiently distributing tasks among worker threads minimizes GPU idle time and maximizes resource utilization, ultimately leading to higher throughput.

Furthermore, by maintaining tensors on the same GPU throughout the process, you can reduce the overhead of CPU-to-GPU (and vice-versa) data transfers, further enhancing efficiency and reducing costs.

Final thoughts

Optimizing image-generation pipelines is a multifaceted process, but the rewards are significant. By adopting a comprehensive approach that includes hardware optimization for maximized resource utilization, code optimization for faster execution, and pipeline optimization for increased throughput, you can achieve substantial performance gains, reduce costs, and deliver exceptional user experiences. In our work with customers, we’ve consistently seen that implementing these optimization strategies can lead to significant cost savings without compromising image quality.

Ready to get started?

At Google Cloud Consulting, we’re dedicated to helping our customers build and deploy high-performing AI solutions. If you’re looking to optimize your image generation pipelines, connect with Google Cloud Consulting today, and we’ll work to help you unlock the full potential of your AI initiatives.

^{We extend our sincere gratitude to Akhil Sakarwal, Ashish Tendulkar, Abhijat Gupta, and Suraj Kanojia for their invaluable support and guidance throughout the crucial experimentation phase.}

Organizations use multiple clouds to gain agility, use resources more efficiently, and leverage the strengths of different cloud providers. However, managing application traffic across these environments is challenging. To support predictable services, organizations need a system that intelligently selects the optimal backend for each request. The selection takes into account both the real-time health of those backends and the user’s location. This entire process of dynamic traffic routing depends on continuous monitoring of application endpoints across all clouds, which delivers the real-time insights needed for informed decisions.

Today, we’re announcing the general availability of Cloud DNS routing policies with public IP health checking, which provides the automated, health-aware traffic management that you need to build resilient applications, no matter where your workloads reside.

Running on multiple cloud providers often leads to fragmented traffic management strategies. Cloud DNS now lets you intelligently route traffic across multiple cloud providers based on application health from a single interface. Cloud DNS supports a variety of routing policies, including weighted round robin (WRR), geolocation, and failover, giving you the flexibility to tailor your traffic management strategy to your specific needs.

Cloud DNS uses routing policies and health checks to direct traffic to healthy backends. These health checks probe internet-based endpoints — any public IP address on other cloud providers, on-premises environments, or even other load balancers. To help improve outage detection in multicloud deployments, health checks are regionalized, originating from points of presence near Google Cloud regions. A backend is considered healthy when a majority of these regional probes report a successful connection. Based on these health checks, Cloud DNS routing policies automatically direct traffic away from failing backends. This automated process happens at the DNS level, providing a crucial layer of control and traffic steering across your infrastructure.

Here are the steps to building a resilient multicloud architecture with Cloud DNS routing policies and public IP health checking:

1. Set up health checks

Configure a HealthCheck resource in Compute Engine, specifying the application’s port on the public IP address. You must select three geographically diverse Google Cloud regions as the origin points for the health-check probes. Good practice is to select regions that are most representative of the user base. For example, if an application services clients from North America and Europe, then a good choice is to include regions from those locations as origins for health checks.

2. Configure a failover routing policy and link it to the health check. 

Create a routing policy in Cloud DNS. Define the primary and backup endpoints, specifying the public IP addresses of your applications in different cloud environments.

3. Fail over automatically

If an application instance becomes unhealthy (two or three out of three regions are reporting a failure), Cloud DNS can switch traffic to the healthy instance in another cloud, depending on how the routing policy is configured and the health of the backup endpoint. The routing decision happens at the DNS level before traffic reaches the applications, helping support failover across your multi-cloud infrastructure.

Because health checks test internet-based endpoints, they can be located anywhere on the internet, letting you build cross-cloud and on-prem failover scenarios. Services can be located in other clouds, and traffic can be switched between providers or to on-prem locations during an outage. This lets you as a multicloud customer standardize on Cloud DNS for workloads, helping streamline traffic management and reduce the operational overhead of managing multiple DNS configurations. Furthermore, with health check logging, you can validate that your routing policies are performing as expected and identify any infrastructure issues with specific backends.

Multicloud deployments are increasingly common. This new Cloud DNS capability provides the automated, health-aware traffic management needed to navigate the complexities of multicloud deployments and strive for positive user experiences.