Tibor Kiss

At Google Cloud, we empower businesses to accelerate their generative AI innovation cycle by providing a path from prototype to production. Palo Alto Networks, a global cybersecurity leader, partnered with Google Cloud to develop an innovative security posture control solution that can answer complex “how-to” questions on demand, provide deep insights into risk with just a few clicks, and guide users through remediation steps. 

Using advanced AI services, including Google’s Gemini models and managed Retrieval Augmented Generation (RAG) services such as Google Cloud’s Vertex AI Search, Palo Alto Networks had an ideal foundation for building and deploying gen AI-powered solutions. 

The end result was Prisma Cloud Co-pilot, the Palo Alto Networks Prisma Cloud gen AI offering. It helps simplify cloud security management by providing an intuitive, AI-powered interface to help understand and mitigate risks.

Technical challenges and surprises 

The Palo Alto Networks Prisma Cloud Co-pilot journey began in 2023 and launched in October 2024. During this time, Palo Alto Networks witnessed Google’s AI models evolve rapidly, from Text Bison (PaLM) to Gemini Flash 1.5. That rapid pace of innovation meant that each iteration brought new capabilities, necessitating a development process that could quickly adapt to the evolving landscape. 

To effectively navigate the dynamic landscape of evolving gen AI models, Palo Alto Networks established robust processes that proved invaluable to their success:

• Prompt engineering and management: Palo Alto Networks used Vertex AI to help manage prompt templates and built a diverse prompt library to generate a wide range of responses. To rigorously test each new model’s capabilities, limitations, and performance across various tasks, Palo Alto Networks and Google Cloud team systematically created and updated prompts for each submodule. Additionally, Vertex AI’s Prompt Optimizer helped streamline the tedious trial-and-error process of prompt engineering.
• Intent recognition: Palo Alto Networks used the Gemini Flash 1.5 model to develop an intent recognition module, which efficiently routed user queries to the relevant co-pilot component. This approach provided users with many capabilities through a unified and lightweight user experience.

• Input guardrails: Palo Alto Networks created guardrails as a first line of defense against unexpected, malicious, or simply incorrect queries that could compromise the functionality and experience of the chatbot. These guardrails maintain the chatbot’s intended functionality by preventing known prompt injection attacks, such as circumventing system instructions; and restricting chatbot usage to its intended scope. Guardrails were created to detect if user queries are restricted to responses within the predefined domain of general cloud security, risks, and vulnerabilities to prevent unintended use. Any topics outside this scope did not receive a response from the chatbot. Additionally, since the chatbot was designed for proprietary code generation for Palo Alto Networks systems to query internal systems, requests for general-purpose code generation similarly did not receive a response.

• Evaluation dataset curation: A robust and representative evaluation dataset serves as a foundation to accurately and quickly assess the performance of gen AI models. The Palo Alto Networks team took great care to choose high-quality evaluation data and keep it relevant by constantly refreshing it with representative questions and expert-validated answers. The accuracy and reliability of the evaluation dataset was sourced and validated directly from Palo Alto Networks subject matter experts.

• Automated evaluation: In collaboration with Google Cloud, Palo Alto Networks developed an automated evaluation pipeline using Vertex AI’s gen AI evaluation service. This pipeline allowed Palo Alto Networks to rigorously scale their assessment of different gen AI models, and benchmark those models using custom evaluation metrics while focusing on key performance indicators such as accuracy, latency, and consistency of responses. 

• Human evaluator training and red teaming: Palo Alto Networks invested in training their human evaluation team to identify and analyze specific loss patterns and provide detailed answers on a broad set of custom rubrics. This allowed them to pinpoint where a model’s response was inadequate and provide insightful feedback on model performance, which then guided model selection and refinement.

  The team also conducted red teaming exercises focused on key areas, including:

  • Manipulating the co-pilot: Can the co-pilot be tricked into giving bad advice by feeding it false information?
  • Extracting sensitive data: Can the co-pilot be manipulated into revealing confidential information or system details?
  • Bypassing security controls: Can the co-pilot be used to craft attacks that circumvent existing security measures?
• Load testing: To ensure the gen AI solutions met real-time demands, Palo Alto Networks actively load tested them, working within the pre-defined QPM (query per minute) and latency parameters of Gemini models. They simulated user traffic scenarios to find the optimal balance between responsiveness and scalability using provisioned throughput, which helped ensure a smooth user experience even during peak usage.

Operational and business challenges 

Operationalizing gen AI can introduce complex challenges across multiple functions, especially for compliance, legal, and information security. Evaluating ROI for gen AI solutions also requires new metrics. To address these challenges, Palo Alto Networks implemented the following techniques and processes:

• Data residency and regional ML processing: Since many Palo Alto Networks customers need a regional approach for ML processing capabilities, we prioritized regional machine learning processing to help enable customer compliance with data residency needs and regional regulations, if applicable.

  Where Google does not offer an AI data center that matched Prisma Cloud data center locations, customers were able to choose having their data processed in the U.S. before gaining access to the Prisma Cloud Co-pilot. We implemented strict data governance policies and used Google Cloud’s secure infrastructure to help safeguard sensitive information and uphold user privacy.

• Deciding KPIs and measuring success for gen AI apps: The dynamic and nuanced nature of gen AI applications demands a bespoke set of metrics tailored to capture its specific characteristics and comprehensively evaluate its efficacy. There are no standard metrics that work for all use cases. The Prisma Cloud AI Co-pilot team relied on technical and business metrics to measure how well the system was operating. 

• Technical metrics, such as recall, helped to measure how thoroughly the system fetches relevant URLs when answering questions from documents, and to help increase the accuracy of prompt responses and provide source information for users. 

• Customer experience metrics, such as measuring helpfulness, relied on explicit feedback and telemetry data analysis. This provided deeper insights into user experience that resulted in increased productivity and cost savings.

Given customer concerns, enterprises must prioritize clear communication around data usage, storage, and protection.  By collaborating  with legal and information security teams early on to create transparency in marketing and product communications, Palo Alto Networks was able to build customer trust and help ensure they have a clear understanding of how and when  their data is being used. 

Ready to get started with Vertex AI ?

The future of generative AI is bright, and with careful planning and execution, enterprises can unlock its full potential. Explore your organization’s AI needs through practical pilots in Vertex AI, and rely on Google Cloud Consulting for expert guidance.

• Learn more about Vertex AI customer use cases and stories.

• Dive into our generative AI repository and explore tuning notebooks and samples.

Your customers might not all speak the same language. If you operate internationally or serve a diverse customer base, you need your chatbot to meet them where they are – whether they’re searching for something in Spanish or Japanese. If you want to give your customers multilingual support with chatbots, you’ll need to orchestrate multiple AI models to handle diverse languages and technical complexities intelligently and efficiently. Customers expect quick, accurate answers in their language, from simple requests to complex troubleshooting.

To get there, developers need a modern architecture that can leverage specialized AI models – such as Gemma and Gemini – and a standardized communication layer so your LLM models can speak the same language, too.  Model Context Protocol, or MCP, is a standardized way for AI systems to interact with external data sources and tools. It allows AI agents to access information and execute actions outside their own models, making them more capable and versatile. Let’s explore how we can build a powerful multilingual chatbot using Google’s Gemma, Translation LLM and Gemini models, orchestrated via MCP.

The challenge: Diverse needs, one interface

Building a truly effective support chatbot might be challenging for a few different reasons:  

1. Language barriers: Support needs to be available in multiple languages, requiring high-quality, low-latency translation.

2. Query complexity: Questions range from simple FAQs (handled easily by a basic model) to intricate technical problems demanding advanced reasoning.

3. Efficiency: The chatbot needs to respond quickly without getting bogged down, especially when dealing with complex tasks or translations.

4. Maintainability: As AI models evolve and business needs change, the system must be easy to update without requiring a complete overhaul.

Trying to build a single, monolithic AI model to handle everything is often inefficient and complex. A better approach? Specialization and smart delegation.

MCP architecture for harnessing different LLMs

The key to making these specialized models work together effectively is MCP. MCP defines how an orchestrator (like our Gemma-powered client) can discover available tools, request specific actions (like translation or complex analysis) from other specialized services, pass necessary information (the “context”), and receive results back. It’s the essential plumbing that allows our “team” of AI models to collaborate. Here’s a framework for how it works with the LLMs:

• Gemma: The chatbot uses a versatile LLM like Gemma to manage conversations, understand user requests, handle basic FAQs, and determine when to utilize specialized tools for complex tasks via MCP.

• Translation LLM server: A dedicated, lightweight MCP server exposing Google Cloud’s Translation capabilities as a tool. Its sole focus is high-quality, fast translation between languages, callable via MCP.

• Gemini: A specialized MCP server uses Gemini Pro or similar LLM for complex technical reasoning and problem-solving when invoked by the orchestrator.

• Model  Context Protocol:This protocol will allow Gemma to discover and invoke the Translation and Gemini “tools” running on their respective servers.

How it works

Let’s walk through an example non-English language scenario:

1. A technical question arrives: A customer types a technical question into the chat window, but it’s in French.

2. Gemma receives the text: The Gemma-powered client receives the French text. It recognizes the language isn’t English and determines translation is needed.

3. Gemma calls on Translation LLM: : Gemma uses the MCP connection to send the French text to the Translation LLM Server, requesting an English translation.

4. Text is translated: The Translation LLM Server performs the translation via its MCP-exposed tool and sends the English version back to the client.

This architecture offers broad applicability. For example, imagine a financial institution’s support chatbot where all user input, regardless of the original language, must be preserved in English in real time for fraud detection. Here, Gemma operates as the client, while Translation LLM, Gemini Flash, and Gemini Pro function on the server. In this configuration, the client-side Gemma manages multi-turn conversations for routine inquiries and intelligently directs complex requests to specialized tools. As depicted in the architectural diagram, Gemma manages all user interactions within a multi-turn chat. A tool leveraging Translation LLM can translate user queries and concurrently save them for immediate fraud analysis. Simultaneously, Gemini Flash and Pro models can generate responses based on the user’s requests. For intricate financial inquiries, Gemini Pro can be employed, while Gemini Flash can address less complex questions.

Let’s look at this sample GitHub repo that illustrates how this architecture works.

Why this is a winning combination

This is a powerful combination because it’s designed for both efficiency and how easily you can adapt it.

The main idea is splitting up the work. The Gemma model based client that users interact with stays light, handling the conversation and sending requests where they need to go. Tougher jobs, like translating or complex thinking, are sent to separate LLMs built specifically for those tasks. This way, each piece does what it’s best at, making the whole system perform better.

A big plus is how this makes things easier to manage and more flexible. Because the parts connect with a standard interface (the MCP), you can update or swap out one of the specialized LLMs – maybe to use a newer model for translation – without having to change the Gemma client. This makes updates simpler, reduces potential headaches, and lets you try new things more easily. You can use this kind of setup for things like creating highly personalized content, tackling complex data analysis, or automating workflows more intelligently.

Get started 

Ready to build your own specialized, orchestrated AI solutions?

• Explore the code: Clone the GitHub repository for this project and experiment with the client and server setup.

• Learn more about the models and MCP: 

• Gemma open models

• Gemini models on Vertex AI

• Google Cloud Translation LLM

• Model Context Protocol (MCP) documentation

We’re thrilled to share that Google Cloud Spanner has been recognized by Gartner in the Critical Capabilities for Cloud Database Management Systems for Operational Use Cases report, where it was ranked #1 in the Lightweight Transactions Use Case and was ranked #3 in the OLTP Transactions Use Case and the Application State Management Use Case. This recognition showcases Spanner’s strength and versatility to handle the most demanding workloads.

Check out the full Gartner report.

Beyond traditional transactions: Expanding capabilities

We believe the Gartner recognition isn’t just about raw performance. We feel it’s about Spanner’s comprehensive feature set, which is designed to address the complex needs of modern enterprises. Beyond its renowned transactional consistency and global scalability, Spanner offers a powerful multi-model experience, seamlessly integrating the graph, full-text, and vector search functionality required by modern applications. 

• Graph database functionality: Spanner’s ability to model and query relationships makes it a strong fit for applications requiring graph analysis, such as social networks, fraud detection, and recommendation engines.

• Full-text search: Integrated full-text search capabilities enable efficient retrieval of unstructured data, powering features like product catalogs, content management systems, and knowledge bases.

• Vector search: With the rise of AI and machine learning, Spanner’s vector search capabilities facilitate similarity searches, enabling applications like image recognition, semantic search, and personalized recommendations.

This flexibility allows developers to build diverse applications on a single platform that provides dynamic elasticity combined with operational efficiency without the complexity of managing multiple specialized databases.

A truly global service: Transactions and analytics combined

Spanner’s global footprint helps ensure low latency and high availability for transactional workloads, regardless of a user’s location. But its power extends beyond transactions. Spanner’s deep integration with BigQuery allows for federated queries, enabling real-time analytics on transactional data without the need for complex ETL processes. This integration also supports reverse ETL from BigQuery, allowing you to push analytical insights back into Spanner for operational use.

Real-world impact: Customer success stories

The true testament to Spanner’s capabilities is its impact on our customers. Here’s a sampling of how it’s being used in the field:

• Spanner’s high availability, external consistency, and infinite horizontal scalability made it the ideal choice for Deutsche Bank’s business critical application for online banking.

• By consolidating all user data with the exception of logs to a single database for development, COLOPL has eliminated the scalability constraints that occurred when using horizontally and vertically partitioned databases for large-scale services.

• With Spanner’s fully-managed relational database, Kroger has been able to build a true event-driven ledger, which enables the company to capture unique events to make better-informed decisions about how to direct associates to be more productive.

Looking ahead

We believe Spanner’s recognition in the Gartner Critical Capabilities report reinforces Google’s position in the Cloud Database Management Systems market. We’re committed to continuing to innovate and expand Spanner’s capabilities, empowering our customers to build the next generation of mission-critical applications.

Whether you need a database for global transactions, multi-model applications or real-time analytics, Spanner is the solution you can rely on. Sign up for a free Spanner trial account and experience the power of multi-model Spanner today.

^{Gartner Critical Capabilities for Cloud Database Management Systems for Operational Use Cases, Ramke Ramakrishnan, Henry Cook, Xingyu Gu, Masud Miraz, Aaron Rosenbaum, 18 December, 2024.}

^{GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. in the U.S. and internationally and is used herein with permission. All rights reserved. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.}