Tibor Kiss

Modern consumers demand a seamless, personalized shopping journey, from initial product discovery all the way to final purchase. With the rise of agentic AI, merchants now have an opportunity to deliver a truly assistive and cohesive experience across every touchpoint.

That’s why today, building on our goal of transforming commerce, PayPal and Google Cloud are thrilled to announce that we’re bringing agentic shopping experiences to life with a new offering that combines Google Cloud’s Conversational Commerce agent with payments powered by PayPal. 

This combination will allow merchants to rapidly deploy agentic commerce experiences directly on their own digital surfaces to drive more consumer engagement, personalization, and conversion. Merchants are able to maintain full control over the agent’s tone, look, and the customer relationship.

How it works

The PayPal Agent will communicate securely with the merchant’s agent over the open  Agent2Agent (A2A) Protocol, as well as being integrated with the Agent Payments Protocol (AP2) — a payments layer built on top of A2A and the Model Context Protocol (MCP) that provides trust, accountability, and fraud controls.

A2A Protocol is an open standard designed to enable AI agents to communicate, collaborate and delegate tasks to one another across organizations. AP2 provides a set of requirements, including Verifiable Digital Credentials, which secure agentic transactions.

Smooth, simple shopping journeys: The power of agent collaboration

With this new offering, merchants will have the option to adopt Google Cloud’s Conversational Commerce Agent or build their own agents using Google’s Agent Development Kit (ADK). Fully brand-compliant and acting as an intelligent sales associate for the merchant, the Conversational Commerce Agent is designed to engage shoppers in natural, human-like conversations, guiding them all the way from initial intent and product discovery to a completed purchase. 

Once deployed, the merchant’s commerce agent can understand complex requests, suggest relevant products, answer questions, and personally assist the user through their shopping journeys. During product discovery and selection, the merchant’s commerce agent engages the PayPal Agent through A2A to provide context on the user’s shopping history, based on permissioned data, to help improve product recommendations. 

Once a customer is ready to check out, the PayPal Agent, in line with AP2, will provide a seamless and secure checkout experience within the conversational interface. The PayPal Agent can also surface payment method recommendations and check “buy now, pay later” eligibility.  With the shopper’s consent, merchant agents will then connect to the PayPal Agent in an authenticated manner, and authorize the transaction on a trusted surface.

Consumer trust at the core 

Agentic commerce holds massive opportunity, but also exposes potential challenges around control, risk, and fraud, which Google Cloud and PayPal are proactively addressing.

AP2 is an open protocol that’s payment-method agnostic, thanks to its development by Google in collaboration with more than 100 industry partners. AP2 provides a common, secure language for AI agents to transact on behalf of users, extending the core constructs of the A2A Protocol and MCP to establish the essential foundation for secure, accountable, and authorized commerce. 

AP2 uses mandates — tamper-proof, cryptographically-signed digital contracts that provide verifiable proof of user intent. These mandates are signed by Verifiable Digital Credentials (VDCs), creating a non-repudiable audit trail.

For example:

• Cart Mandate: The foundational credential used when the user is present to authorize a purchase. Cart Mandates are generated by the merchant and cryptographically signed by the user (typically via their device), binding authorization to a specific transaction.

• Payment Mandate: A separate VDC shared with the payment network and issuer to provide visibility into the agentic nature of the transaction, helping the network and issuer build trust and assess risk. This credential contains signals for AI agent presence and the transaction modality (e.g. Human Present vs. Not Present).

Essentially, AP2 provides the critical foundation for trusted, agent-led payments, providing verifiable intent and establishing clear transaction accountability. Instead of inferring action, trust is anchored to deterministic, non-repudiable proof of intent from the user, which directly addresses the risk of agent error. Payment mandates act as the foundational evidence for every transaction, creating a secure, unchangeable audit trail that helps payment networks to establish clear and fair principles for accountability and dispute resolution. 

For example, with PayPal’s AP2-compliant agent, merchants will be able to have the assurance that a user was present to authorize the payment. Instead of using APIs, it will connect agents using AP2, helping ensure users, merchants, and payment providers can confidently initiate and transact with agent-driven payments.

With today’s announcement, Google Cloud and PayPal are proud to work together to provide a largely out-of-the-box solution for merchants who want to deploy agentic commerce experiences without building the complex framework from scratch, all while owning the experience and relationship with the consumer. Building the solution using A2A and AP2 protocols ensures  safety and security throughout the process.

To learn more, contact your Google Cloud sales representative or reach us here.

_{Disclaimer: The video shown in this post is for informational purposes only and contains forward-looking statements, projections, and assumptions. These are not guarantees of future performance, and actual results and experiences may vary.}

Today, we’re pleased to introduce a new Bigtable storage tier for efficient management of massive datasets, now available in preview. This fully managed, cost-effective system automatically moves less frequently accessed data from high-performance SSDs to infrequent access storage, lowering your total cost of ownership. With tiered storage in Bigtable, you can access and modify data across both hot and cold tiers via a single interface. You don’t have to sacrifice data to cost controls, you can afford to keep the full picture of the application and you no longer have to compromise on finding critical historical insights.

Bigtable, Google Cloud’s key-value and wide-column store is ideal for fast access to structured, semi-structured, or unstructured data, including time-series data from sensors, equipment, and operations in industries such as manufacturing and automotive. 

High-volume data streams — including electric vehicle (EV) battery data, factory-floor machine status, and automotive telemetry from software-defined vehicles (SDVs) and in-vehicle infotainment (IVI) systems — are essential for driving business and technical objectives. These objectives range from driver personalization and optimized equipment maintenance schedules to logistics optimization and predictive maintenance. However, efficiently storing such vast quantities of data can become costly, particularly when it’s not frequently accessed.

Introducing Bigtable tiered storage

Bigtable’s new tiered storage feature can help you manage your storage costs while meeting regulatory data storage requirements. It automatically moves older, infrequently used data to a less expensive storage tier — where it remains available when needed — without impacting access to your more recent, frequently used data.

Bigtable’s new “infrequent access” storage tier works alongside your existing SSD storage, allowing you to store both frequently and infrequently used data in the same table and manage it all in one place. This feature works with Bigtable’s autoscaling to optimize your Bigtable instance resource utilization. Moreover, data in the infrequent access storage tier is still accessible alongside existing SSD storage through the same Bigtable API.

Key benefits of Bigtable tiered storage

• Unified management: Manage data in a single Bigtable instance without manually exporting infrequently accessed data to archival storage. With Bigtable tiered storage, you can reduce operational overhead and avoid manual data organization and migration.

• Automatic tiering: Set an age-based tiering policy, and Bigtable automatically moves data between SSD and infrequent access tiers. Retain data for longer to meet regulatory compliance requirements while retaining data access. 

• Cost optimization: Move and store historical data to infrequent access to lower storage costs. Infrequent access storage is up to 85% less expensive than SSD storage. This can significantly reduce overall storage expenses, as well as the operational overhead of manual data migrations.

• Increased storage capacity: Infrequent access storage increases the total storage space of your Bigtable node. This lets you store more data per node than you can with the standard Bigtable SSD node. A Bigtable node with tiered storage has 540% more capacity than a regular SSD node.

• Data accessibility for analytics and reporting: Use Bigtable SQL to query infrequently used data. You can then build Bigtable logical views to present this data in a format that can be queried when needed. This feature is useful for giving specific users access to historical data for reports, without giving them complete access to the table.

Operational time-series data: an example

Bigtable is well-suited for time-series data such as sensor readings or vehicle telemetry, and this data’s variety, speed, and volume makes it suitable for Bigtable tiered storage. This data pattern includes:

• Varying schema: Systems often have multiple data sources with different structures. Bigtable’s flexible structure is helpful for managing these different sources.

• Time-based access patterns: The most recent data is often required for real-time operations and dashboards, while historical data is valuable for analysis and long-term trends.

• Archival needs: Data needs to be stored for long periods for compliance or analysis.

Consider a manufacturing plant that uses Bigtable for sensor data:

• The challenge: The plant collects data from sensors every second. This information is important, but storing everything on an SSD device is expensive.

• The solution: The plant uses Bigtable tiered storage with an age-based rule:

• Last 30 days: Data is stored on SSD for quick access.

• 30 days to 1 year: Data is moved to the infrequent access storage tier for analysis.

• Older than 1 year: Data is deleted due to the garbage collection policy on the table. This period is fully configurable and can be extended, for example, to six years.

  Note: You can access your infrequent access storage tier through the same Bigtable API that you use to access SSD storage.

• Monitor performance: Use Bigtable’s monitoring tools to track storage use, speed, and data flow for both SSD and infrequent access tiers.

• Adjust policy: Change the tiering policy based on your needs.

• Structure the relevant sensor data as a logical view: Use SQL on the infrequent access storage, providing a relational data model to the historical sensor information.

• The results:

• Simplified operations by managing all data in one Bigtable instance

• Historical data is stored for compliance

• Reduced storage costs

Best practices when using tiered storage

• Write your data with timestamps: Include accurate timestamps in your data to enable age-based tiering.
• Read your data using timestamp range filters: Use timestamp range filters to ensure your reads go to the correct storage tier. For SSD-only reads, timestamp range filters are required to maintain SSD performance.
• Monitor performance: Check performance metrics to find bottlenecks and adjust your tiering policy.
• Use autoscaling: Use autoscaling to change resources automatically based on your needs.

Get started today

Bigtable tiered storage helps manage costs and simplifies data management, especially for time-series data. It lets you keep important data accessible while managing the expenses of storing large historical datasets. This is helpful for businesses using large amounts of time-series data, such as those in manufacturing, automotive, and IoT. To learn more and get started, enable Bigtable tiered storage for your table.

Building and scaling generative AI models demands enormous resources, but this process can get tedious. Developers wrestle with managing job queues, provisioning clusters, and resolving dependencies just to ensure consistent results. This infrastructure overhead, along with the difficulty of discovering the optimal training recipe and navigating the endless maze of hyperparameter and model architecture choices, slows the path to production-grade model training.

Today, we’re announcing expanded capabilities in Vertex AI Training that simplify and accelerate the path to developing large, highly differentiated models. 

Our new managed training features, aimed at developers training with hundreds to thousands of AI accelerators, builds on the best of Google Cloud’s AI infrastructure offerings, including Cluster Director for a fully managed and resilient Slurm environment, and adds sophisticated management tools. This includes pre-built data science tooling and optimized recipes integrated with frameworks like NVIDIA NeMo for specialized, massive-scale model building.

Built for customization and scale

Vertex AI Training delivers choice across the full spectrum of model customization. This range extends from cost-effective, lightweight tunings like LoRA for rapid behavioral refinement of models like Gemini, all the way to large-scale training of open-source or custom-built models on clusters for full domain specialization.

The Vertex AI training capabilities are organized around three areas: 

1. Flexible, self-healing infrastructure

With Vertex AI Training, you can create a production-ready environment in minutes. By leveraging our included Cluster Director capabilities, customers benefit from a fully managed and resilient Slurm environment that simplifies large scale training.

Automated resiliency features proactively check for and avoid stragglers, swiftly restart or replace faulty nodes, and utilize performance-optimized checkpointing functionality to maximize cluster uptime.

To achieve optimal cost efficiency, you can provision Google Cloud capacity using our Dynamic Workload Scheduler (DWS). Calendar Mode provides fixed, future-dated reservations (up to 90 days), similar to a scheduled booking. Flex-start provides flexible, on-demand capacity requests (up to 7 days) that are fulfilled as soon as all requested resources become simultaneously available. 

2. Comprehensive data science tooling

Our comprehensive data science tooling removes much of the guesswork from complex model development. It includes capabilities such as hyperparameter tuning (which automatically finds the best model settings), data optimization, and advanced model evaluation – all designed to ensure your specialized models are production-ready faster.

3. Integrated recipes and frameworks

Maximize training efficiency out-of-the-box with our curated, optimized recipes for the full model development lifecycle, from pre-training and continued pre-training to supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). We also provide seamless integration of standardized frameworks like NVIDIA NeMo and NeMo-RL.

How customers are seeing impact with Vertex AI Training

Salesforce: The Salesforce AI Research team leveraged Vertex AI Training to expand the capabilities of their large action models. By fine-tuning these models for their unique business operations, Salesforce’s Gemini models now outperform industry-leading LLMs against key CRM benchmarks. This allows customers to more accurately and reliably automate complex, multi-step business processes, providing the reliable foundation for building AI agents.

• “In the enterprise environment, it’s imperative for AI agents to be highly capable and highly consistent, especially for critical use cases. Together with Google Cloud, we are setting a new standard for building the future of what’s possible in the agentic enterprise down to the model level.” – Silvio Savarese, Chief Scientist at Salesforce

AI Singapore (AISG): AISG utilized Vertex AI Training’s managed training capabilities on reserved clusters to launch their 27-billion parameter flagship model. This extensive specialization project demanded peak infrastructure reliability and performance tuning to achieve precise language and contextual customization for diverse Southeast Asian markets.

• “AI Singapore recently launched SEA-LION v4, an open source foundational model incorporating Southeast Asian contexts and languages. Vertex AI and its managed training clusters were instrumental in our development of SEA-LION v4. Vertex AI delivered a stable, resilient environment for our large scale training workloads that was easy to set up and use. Its optimized training recipes helped increase training throughput performance by nearly 30%.” – William Tjhi, Head of Applied Research, AI Products Pillar, AI Singapore

Looking for more control?

For customers seeking maximum flexibility and control, our AI-optimized infrastructure is available via Google Compute Engine or through Google Kubernetes Engine, both of which include Cluster Director to provision and manage highly scalable AI training accelerators and clusters. Cluster Director provides the deep control over hardware, network optimization, capacity management, and operational efficiency that these advanced users demand.

Elevate your models today

Vertex AI Training provides the full range of approaches, the world-class infrastructure, and the expertise to make your AI your most powerful competitive asset. Interested customers should contact their Google Cloud sales representative for access and to gain access and learn more about how Vertex AI Training can help deliver their unique business advantage.

The conversation around generative AI in the enterprise is getting creative. 

Since launching our popular Nano Banana model, consumers have created 13 billion images and 230 million videos¹. Enterprises can combine Gemini 2.5 Pro with our generative media models – Lyria, Chirp, Imagen, and Veo – to bring their ideas to life. 

To us, generative media is a canvas to explore ideas that were previously constrained by time, budget, or the limits of conventional production. To test this, we briefed several top agencies to use Google’s AI to create an “impossible” ad — a campaign that pushes the boundaries of what’s creatively and technically feasible.

This is what they created.

Impossible retro audio experiences: Reimagine yesterday’s greatest hits, today

Brand: Slice healthy soda

Agency: BarkleyOKRP

Challenge: Slice needed to relaunch a nostalgic soda brand with a new focus on probiotic benefits. They aimed to create a distinct brand experience that resonated with both long-time fans and a new generation, creatively showcasing its retro appeal and health-focused features.

Approach: “106.3 The Fizz,” an AI-generated retro radio station, marketed Slice’s relaunch. Gemini wrote 80s/90s pop lyrics, lore, and DJ banter, all infused with “fizz” themes, and powered the global streaming site. Imagen and Veo 3 created visual assets like album covers and music videos. Lyria composed lo-fi instrumentals for a “Chill Zone,” and Chirp provided voices for radio hosts. This approach combined nostalgia with AI innovation, matching Slice’s retro-meets-modern identity.

Results:

• +119 million PR and paid media impressions

• 45,700 online streams

• A 33% view rate

• A 60% more efficient CPM

• 180,000 analog radio listeners

Tech stack:

• Google Cloud (Gemini 2.5 Pro, Lyria, Veo, Imagen)

• YouTube channel

Impossible personalization: Message the future with personalized trip previews

Brand: Virgin Voyages

Agency: In-house at Virgin Voyages 

Challenge: Virgin Voyages wanted to improve its digital advertising by creating highly personalized and engaging ad experiences. The goal was to re-engage prospective cruisers with compelling visuals and messaging that directly reflected their on-site browsing behavior, turning potential bookings into actual conversions.

Approach: Virgin Voyages launched “Postcards from your future self.” This campaign used Google AI to create personalized “postcard” ads based on users’ browsing behavior on virginvoyages.com. Gemini interpreted on-site signals, such as viewed itineraries or ship pages, to generate tailored messaging, taglines, and calls to action. Imagen then created static postcard visuals matching the destinations and cruise themes each user explored, while Veo produced dynamic video versions for more immersive ad formats. These unique AI-generated creatives were used to retarget users, showing them a “Postcard from your future self” specific to their browsing session.

Tech stack:

• Google Cloud (Gemini 2.5 Pro, Imagen, Veo 2, Vertex AI)

• Google Ads (YouTube, Search, Display)

Impossible experiences: Unlock endless, unique party themes & bespoke cocktails

Brand: Smirnoff

Agency: McCANN 

Challenge: Smirnoff aimed to become the preferred vodka brand for LDA Gen Z’s house party culture. While popular for casual home use, the brand wanted to elevate its status and become linked with the unique, personalized gatherings favored by this generation, being the go-to option for bringing people together over delicious drinks. To lead in the LDA Gen Z home party market, Smirnoff needed an innovative way to connect and prove its relevance, making every at-home celebration an unforgettable experience to enjoy responsibly and with moderation.

Approach: Smirnoff introduced Party Engine, an AI-powered co-host that designs unique house parties. Gemini powered a conversational co-host that chatted with each guest to understand their preferences and personalities. As more guests interacted, Gemini combined their inputs with cultural data to develop a unique party theme in real-time. The engine recommended specific party details, including the theme, music, decor, and a personalized Smirnoff cocktail. This approach blended guest personalities with cultural trends, down to the dress code and playlist, creating tailored, one-of-a-kind experiences, all designed to deliver the collective effervescence that Smirnoff brings to every occasion.

Impossible world building: Crowdsource mascots for the lesser traveled parts of Orlando

Brand: Visit Orlando

Agency: razorfish 

Challenge: To attract visitors to Orlando’s unique, lesser-known destinations beyond major theme parks, Visit Orlando needed to create compelling awareness. They required an innovative strategy to differentiate these local attractions and their distinct personalities from dominant parks like Walt Disney World and Universal Studios, encouraging travelers to explore the city’s hidden attractions.

Approach: Visit Orlando launched “The Morelandos,” a group of AI-generated characters inspired by real Google reviews. Vertex AI powered a custom agent that gathered and organized Google reviews into distinct personality traits and descriptors for each location. Gemini then turned this information into creative prompts and character backstories, while Imagen visualized these unique mascots. Veo brought the characters to life through animated video stories, featured in YouTube pre-roll and Performance Max campaigns. The characters are available on a Google Maps-integrated experience on VisitOrlando.com, allowing users to explore them online or in real life through AR.

Tech stack:

• Gemini 2.5 Pro 

• Vertex AI, Veo, Imagen

• Google Maps

• Google Reviews

• AR Core

• Firebase Studio

• YouTube Pre-roll ads

Impossible consistency: Achieve cinematic quality and brand consistency

Brand: Moncler

Agency: R/GA

Challenge: Moncler sought innovative ways to produce high-quality, cinematic visual content at scale while maintaining its distinctive luxury aesthetic and brand consistency across diverse creative inputs. The goal was to show how advanced AI could serve as a powerful creative partner for high-end storytelling through an experimental brand film.

Approach: Moncler partnered with R/GA to create “A Journey from Mountains to the City,” an experimental AI-driven film. Gemini powered a tool called Shotflow, which converted creative direction, style, and references into consistent, production-ready prompts. Veo 2 then used these prompts to create high-quality, cinematic visuals that perfectly matched Moncler’s luxury aesthetic. R/GA’s development of Shotflow also enabled global collaboration and maintained visual continuity throughout the project. This film was not intended for media distribution.

The results: The project was finished in four weeks, establishing Veo as a strong creative partner for high-end, brand-forward storytelling and demonstrating AI’s ability to produce cinematic, consistent visuals for luxury brands.

Tech stack:

• Gemini 2.5 Pro

• Google Cloud (Veo 2, Imagen)

Get started

If you’re interested in learning how to apply these AI-driven approaches to your own brand challenges, explore Gemini 2.5 Pro and our generative media solutions:

• Gemini 2.5 Pro 

• Veo

• Imagen

^{1. Source: Google internal data, Oct 2025}

Effective monitoring and treatment of complex diseases like cancer and Alzheimer’s disease depends on understanding the underlying biological processes, for which proteins are essential. Mass spectrometry-based proteomics is a powerful method for studying these proteins in a fast and global manner. Yet the widespread adoption of this technique remains constrained by technical complexity as mastering these sophisticated analytical instruments and procedures requires specialized training. This creates an expertise bottleneck that slows research progress.

To address this challenge, researchers at the Max Planck Institute of Biochemistry collaborated with Google Cloud to build a Proteomics Lab Agent that assists scientists with their experiments. This agent simplifies performing complex scientific procedures through personalized AI guidance, making them easier to execute, while automatically documenting the process.

“A lab’s critical expertise is often tacit knowledge that is rarely documented and lost to academic turnover. This agent addresses that directly, not only by capturing hands-on practice to build an institutional memory, but by systematically detecting experimental errors to enhance reproducibility. Ultimately, this is about empowering our labs to push the frontiers of science faster than ever before.”, said Prof. Matthias Mann, a pioneer in mass spectrometry-based proteomics who leads the Department of Proteomics and Signal Transduction at the Max Planck Institute of Biochemistry.

The agent was built using the Agent Development Kit (ADK), Google Cloud infrastructure, and Gemini models, which offer advanced video and long-context understanding uniquely suited to the needs of advanced research. 

One of the agent’s core capabilities is to detect errors and omissions by analyzing a video of a researcher performing lab work and comparing their actions against a reference protocol. This process takes just over two minutes and catches about 74% of procedural errors with high accuracy, although domain-specific knowledge and spatial recognition should still be improved.Our Ai-assisted approach is more efficient compared to the current manual approach, which relies on a researcher’s intuition to either spot subtle mistakes during the procedure or, more commonly, to troubleshoot only after an experiment has failed.

By making it easier to spot mistakes and offering personalized guidance, the agent can reduce troubleshooting time and build towards a future where real-time AI guidance can help prevent errors from happening.

The potential of the Proteomics AI agent goes beyond life sciences, addressing a universal challenge in specialized fields: capturing and transferring the kind of expertise that is learned through hands-on practice, not from manuals. To enable other researchers and organizations to adapt this concept to their own domains, the agentic framework has been made available as an open-source project on GitHub.

In this post, we will detail the agentic framework of the Proteomics Lab Agent, how it uses multimodal AI to provide personalized laboratory guidance, and the results from its deployment in a real-world research environment.

The challenge: Preserving expert knowledge in a high-turnover environment

Imagine it’s a Friday evening in the lab. A junior researcher needs to use a sophisticated analytical instrument, a mass spectrometer, but the senior expert who is responsible for it has already left for the weekend. The researcher has to search through lengthy protocols, interpret the instrument’s performance, which depends on multiple factors reflected in diverse metrics, and proceed without guidance. A single misstep could potentially damage the expensive equipment, waste a unique and valuable sample, or compromise the entire study.

Such complexity is a regular hurdle in specialized research fields like mass spectrometry-based proteomics. Scientific progress often depends on complex techniques and instruments that require deep technical expertise. Laboratories face a significant bottleneck in training personnel, documenting procedures, and retaining knowledge, especially with the high rate of academic turnover. When an expert leaves, their accumulated knowledge often leaves with them, forcing the team to partially start over. Collectively, this creates accessibility and reproducibility challenges, which slows down new discoveries.

A solution: an AI agent for lab guidance

The proteomics lab agent addresses these challenges by connecting directly to the lab’s collective knowledge – from protocols and instrument data to past troubleshooting decisions. With this it provides researchers with personalized AI guidance for complex procedures across the entire experimental workflow. Examples include regular wet-lab work such as pipetting or the interactions with specialized equipment and software as required for operating a mass spectrometer. A further feature of the agent is the ability to automatically generate detailed protocols from videos of experiments, detect procedural errors, and provide guidance for correction, reducing troubleshooting and documentation time.

An AI agent architecture for the lab

The underlying multimodal agentic AI framework uses a main agent that coordinates the work of several specialized sub-agents, as shown in Figure 1. Built with Gemini models and the Agent Development Kit, this main agent acts as an orchestrator. It receives a researcher’s query, interprets the request, and delegates the task to the appropriate sub-agent.

The sub-agents are designed for specific functions and connect to the lab’s existing knowledge systems:

• Lab Note and Protocol Agents: These agents handle video-related tasks. When a researcher provides a video of an experiment, these agents upload videos to Google Cloud Storage to allow the analysis of the visual and spoken content of a video. Following, the agent can check for errors or generate a new protocol.

• Lab Knowledge Agent: This agent connects to the laboratory’s knowledge base (MCP Confluence) to retrieve protocols or save new lab notes, making knowledge accessible to the entire team.

• Instrument Agent: To provide guidance on using complex analytical instruments, this agent retrieves instrument performance metrics from a self-build MCP server that monitors the lab’s mass spectrometers (MCP AlphaKraken).

• Quality Control Memory Agent: This agent captures all instrument-related decisions and their outcomes in a database (e.g. MCP BigQuery). This creates a searchable history of what has worked in the past and preserves valuable troubleshooting experience.

Together, these agents can provide guidance adapted to the current instrument status and the researcher’s experience level while automatically documenting the researcher’s experience.

A closer look: Catching experimental errors with video analysis

While generative AI has proven effective for digital tasks in science – from literature analysis to controlling lab robots through code – it has not addressed the critical gap between digital assistance and hands-on laboratory execution. Our work demonstrates how to bridge this divide by automatically generating lab notes and detecting experimental errors from a video.

The process, illustrated in Figure 2, unfolds in several steps:

1. A researcher records their experiment and submits the video to the agent with a prompt like, “Generate a lab note from this video and check for mistakes.”.

2. The main agent delegates the task to the Lab Note Agent, which uploads the video to Google Cloud Storage and analyzes the actions performed in the video.

3. The main agent asks the Lab Knowledge Agent to find the protocol that matches these actions. The Lab Knowledge Agent then retrieves it from the lab’s knowledge base, Confluence.

4. With both the video analysis and the baseline protocol, the task is passed on to the Lab Note Agent again, which has the knowledge how to perform a step-by-step comparison of video and protocol. It flags any potential mistakes, such as missed steps, incorrectly performed actions, added steps not in the protocol, or steps completed in the wrong order.

5. The main agent returns the generated lab notes to the researcher with these potential errors flagged for review. The researcher can accept the notes or make corrections.

6. Once finalized, the corrected notes are saved back to the Confluence knowledge base via the Lab Knowledge Agent, preserving a complete and accurate record of the experiment.

Building institutional memory

To support a lab in building a knowledge base, the Protocol Agent can generate lab instructions directly from a video. A researcher can record themselves performing a procedure while explaining the steps aloud. The agent analyzes the video and audio to produce a formatted, publication-ready protocol. We found that providing the model with a diverse set of examples, step-by-step instructions, and relevant background documents produced the best results.

The agent can also support instrument operations (see Figure 3). A researcher may ask, “Is instrument X ready so that I can measure my samples?”. The agent retrieves the latest instrument metrics via the Instrument Agent and compares it with past troubleshooting decisions from the Quality Control Memory Agent. It then provides a recommendation, such as “Yes, the instrument is ready,” or “No, calibration is recommended first”. It can even provide the relevant calibration protocol from the Lab Knowledge Agent. Subsequently, it saves the final researcher’s decision and actions with the Quality Control Memory Agent. With this, every reasoning and its outcome is saved, creating a continuously improving knowledge base for operating specialized equipment and software. 

More technical details are described in our full publication.

Real-world impact: Making complex scientific procedures easier

To measure the AI agent’s value in a real-world setting, we deployed it in our department at the Max Planck Institute of Biochemistry, a group with 40 researchers. We evaluated the agent’s performance across three key laboratory functions: detecting procedural errors, generating protocols, and providing personalized guidance.

The results showed strong gains in both speed and quality. Key findings include:

• AI-assisted error detection: The agent successfully identified 74% of all procedural errors (a metric known as recall) with an overall accuracy of 77% when comparing 28 recorded lab procedures against their reference protocols. While precision (41%) is still a limitation at this early stage, the results are highly promising.

• Fast, expert-quality protocols: From lab videos, the agent generated standardized, publication-ready protocols in about 2.6 minutes. This was approximately 10 times faster than manual creation and achieved an average quality score of 4.4 out of 5 across 10 diverse protocols.

• Personalized, real-time support: The agent successfully integrated real-time instrument data with past performance decisions to provide researchers with tailored advice on equipment use.

A deeper analysis of the error-detection results revealed specific strengths and areas for improvement. As shown in Figure 4, the system is already effective at recognizing general lab equipment and reading on-screen text. The main limitations were in understanding highly specialized proteomics equipment (27% of these errors were unrecognized) and perceiving fine-grained details, such as the exact placement of pipette tips on a 96-well grid (47%) or small text on pipettes (41%) (see Appendix of corresponding paper). As multimodal models advance, we expect their ability to interpret these details will improve, strengthening this critical safeguard against experimental mistakes.

Our agent already automates documentation and flags errors in recorded videos, but its future potential lies in prevention, not just correction. We envision an interactive assistant that uses speech to prevent mistakes in real-time before they happen. By making this project open source, we invite the community to help build this future.

Scaling for the future

In conclusion, this framework addresses critical challenges in modern science, from the reproducibility crisis to knowledge retention in high-turnover academic environments. By systematically capturing not just procedural data but also the expert reasoning behind them, the agent builds an institutional memory.

“This approach helps us capture and share the practical knowledge that is often lost when a researcher leaves the lab”, notes Matthias Mann. “This collected experience will not only accelerate the training of new team members but also creates the data foundation we need for future innovations like predictive instrument maintenance for mass spectrometers and automated protocol harmonization within individual labs and across different labs”.

The principles behind the Proteomics Lab Agent are not limited to one field. The concepts outlined in this study are a generalizable solution for any discipline that relies on complex, hands-on procedures, from life sciences to manufacturing.

Dive deeper into the methodology and results by reading our full paper. Explore the code on GitHub and adapt the Proteomics Lab Agent for your own research. Follow the work of the Mann Lab at the Max Planck Institute to see what comes next either on LinkedIn, BlueSky or X.

_{This project was a collaboration between the Max Planck Institute of Biochemistry and Google. The core team included Patricia Skowronek and Matthias Mann from Department of Proteomics and Signal Transduction at the Max Planck Institute for Biochemistry and Anant Nawalgaria from Google. P.S. and M.M. want to thank the entire Mann Lab for their support.}