GCP – Integrating MedGemma into clinical workflows just got easier!

Our team, Google Heath AI Developer Foundations, introduced MedGemma earlier this year in May and later followed up in July with a 27-billion-parameter multimodal variant plus MedGemma’s vision encoder: MedSigLIP. We’ve been humbled by the wide variety of model adaptations, research papers, and applications MedGemma has created across academia and industry!

Our aim is to meet you where you are in your research, development, and clinical integration journey. In the earlier releases, we prioritized simplicity, where image prompts were constructed from pixels decoded from non-medical image formats such as JPEG and PNG, and medical record snippets were fed into the model in JSON format or as plain text.

However, we acknowledge the complexities of integrating MedGemma into clinical workflows in an interoperable way. That’s why standard protocols like Digital Imaging and Communications in Medicine (DICOM) and Fast Healthcare Interoperability Resources (FHIR) are crucial for integration into clinical workflows. Today, we’re pleased to announce that we have made it simpler for developers who are working with these data formats. 

DICOMweb integration 

We are releasing a new Docker container for MedGemma which accepts medical images as DICOMweb links: 

• Public container 

• Container source code  

• API spec

You can use this new Docker container or source code directly to deploy DICOM-aware MedGemma services on any compute platform. However, if you’re a user of Google Cloud Platform (GCP) with data stored in Cloud DICOM Store, visit get started section in this post to get up and running in minutes using pre-built resources on Vertex Model Garden.

Note that since inception, MedSigLIP container has had native understanding of DICOM; here’s the public container, the container source code, and API spec.

When your interactive user-facing applications have to deal with complex modalities such as digital pathology Whole Slide Imaging (WSI) or multi-dimensional radiology imaging such as Computed Tomography (CT) or Magnetic Resonance Imaging (MRI), reading images server-side optimizes network performance and bypasses API payload restrictions. Furthermore, this architecture hardens security in transit and ensures consistent, deterministic data preprocessing.

Note that as a GCP user, you are not limited to deployment via Model Garden which currently deploys the model and its custom container using a fixed configuration to a Vertex online inference endpoint for you. Please refer to the serving architecture document to understand your deployment options. 

FHIR navigation agent demonstration

In our FHIR integration approach, we configured MedGemma and the GCP FHIR Store as executable tools for an agent. We show how the agent formulates prompts requiring a patient’s full medical records without feeding their entire FHIR history into MedGemma’s context window, leveraging MedGemma’s awareness of the FHIR standard to intelligently navigate patient data. We demonstrate an implementation using LangGraph, a popular agentic framework, though the same can be achieved using other agentic frameworks including the GCP’s Agent Development Kit (ADK). Visit the get started section in this post and see how an agent can intelligently navigate patient data.

Get started

• For DICOM-aware MedGemma, start with the model on Model Garden and use the new drop down options to deploy either 4B or 27B variants. Once deployed, use this tutorial notebook to see how to prompt the model with links to the medical images instead of the image pixels.

• For FHIR navigation demonstration, start with the illustrative app and then look into the technical details in the demo notebook. 

What’s next

If you are looking to build advanced agentic solutions, systems that use LLMs to perform complex, multi-step tasks, Model Context Protocol (MCP) is a reliable way to manage and deliver all the necessary context and data. To leverage MCP on GCP, you should take advantage of the open MCP Toolbox for Databases and its integration with GCP Healthcare API. If you are working with medical imaging, you can use the new DICOM-aware MedGemma to achieve more efficient server-side DICOM processing in your MCP configuration, accelerating the preparation of clinical context for your agentic applications.

Resources and support 

Our mission is to enable your success. Here are the best ways to engage with our team and the community:

• Seek technical support on the HAI-DEF developer forum.

• File technical issues directly in the MedGemma or MedSigLIP GitHub repos.

• Help shape our roadmap by sharing your use cases via our feedback form. This helps us align our engineering efforts with the industry’s most common needs.

• Stay updated on new tools and models by signing up for our newsletter.

• Access all resources by bookmarking goo.gle/hai-def, your one-stop shop for everything we offer.

This post includes additional contributions from Liron Yatziv – Software Engineer, Kenneth Philbrick – Software Engineer, Bram Sterling – Software Engineer, and Tiffany Chen – Software Engineer.