GCP – Investigate fast with AI: Gemini Cloud Assist for Dataproc & Serverless for Apache Spark

Apache Spark is a fundamental part of most modern lakehouse architectures, and Google Cloud’s Dataproc provides a powerful, fully managed platform for running Spark applications. However, for data engineers and scientists, debugging failures and performance bottlenecks in distributed systems remains a universal challenge.

Manually troubleshooting a Spark job requires piecing together clues from disparate sources — driver and executor logs, Spark UI metrics, configuration files and infrastructure monitoring dashboards. 

What if you had an expert assistant to perform this complex analysis for you in minutes? 

Today, we are excited to introduce the public preview of Gemini Cloud Assist Investigations for troubleshooting Spark workloads. Available for both Dataproc on Google Compute Engine and Google Cloud Serverless for Apache Spark, Gemini Cloud Assist identifies underlying issues and provides clear, actionable recommendations. 

Accessible directly in the Google Cloud console — either from the resource page (e.g., Serverless for Apache Spark Batch job list or Batch detail page) you are investigating or from the central Cloud Assist Investigations list — Gemini Cloud Assist offers several powerful capabilities: 

• For data engineers: Fix complex job failures faster. A prioritized list of intelligent summaries and cross-product root cause analyses helps in quickly narrowing down and resolving a problem.

• For data scientists and ML engineers: Solve performance and environment issues without deep Spark knowledge. Gemini acts as your on-demand infrastructure and Spark expert so you can focus more on models.  

• For Site Reliability Engineers (SREs): Quickly determine if a failure is due to code or infrastructure. Gemini finds the root cause by correlating metrics and logs across different Google Cloud services, thereby reducing the time required to identify the problem.

• For big data architects and technical managers: Boost team efficiency and platform reliability. Gemini helps new team members contribute faster, describe issues in natural language and easily create support cases.

Gemini Cloud Assist is also accessible through a direct API and other interfaces.

The inherent challenges of debugging Spark jobs

Debugging Spark applications is inherently complex because failures can stem from anywhere in a highly distributed system. These issues generally fall into two categories. First are the outright job failures. Then, there are the more insidious, subtle performance bottlenecks. Additionally, cloud infrastructure issues can cause workload failures, complicating investigations.

Gemini Cloud Assist is designed to tackle all these challenges head-on:

Gemini Cloud Assist: Your AI-powered operational expert

Let’s explore how Gemini transforms the investigation process in common, real-world scenarios.  

Example 1: The slow job with performance bottlenecks

Some of the most challenging issues are not outright failures but performance bottlenecks. A job that runs slowly can impact service-level objectives (SLOs) and increase costs, but without error logs, diagnosing the cause requires deep Spark expertise. 

Say a critical batch job succeeds but takes much longer than expected. There are no failure messages, just poor performance.  

Manual investigation requires a deep-dive analysis in the Spark UI. You would need to manually search for “straggler” tasks that are slowing down the job. The process also involves analyzing multiple task-level metrics to find signs of memory pressure or data skew. 

With Gemini assistance

By clicking Investigate, Gemini automatically performs this complex analysis of performance metrics, presenting a summary of the bottleneck.

Gemini acts as an on-demand performance expert, augmenting a developer’s workflow and empowering them to tune workloads without needing to be a Spark internals specialist.

Example 2: The silent infrastructure failure

Sometimes, a Spark job or cluster fails due to issues in the underlying cloud infrastructure or integrated services. These problems are difficult to debug because the root cause is often not in the application logs but in a single, obscure log line from the underlying platform.

Say a cluster configured to use GPUs fails unexpectedly. 

The manual investigation begins by checking the cluster logs for application errors. If no errors are found, the next step is to investigate other Google Cloud services. This involves searching Cloud Audit Logs and monitoring dashboards for platform issues, like exceeded resource quotas.

With Gemini assistance

A single click on the Investigate button triggers a cross-product analysis that looks beyond the cluster’s logs. Gemini quickly pinpoints the true root cause, such as an exhausted resource quota, and provides mitigation steps.

Gemini bridges the gap between the application and the platform, saving hours of broad, multi-service investigation.

Get started today!

Spend less time debugging and more time building and innovating. Let Gemini Cloud Assist in Dataproc on Compute Engine and Google Cloud Serverless for Apache Spark be your expert assistant for big data operations.

Get Gemini Cloud Assist today!

Learn more about Gemini Cloud Assist in Dataproc on Compute Engine and Google Cloud Serverless for Apache Spark.