GCP – Announcing Gemma 3 on Vertex AI

Today, we’re sharing the new Gemma 3 model is available on Vertex AI Model Garden, giving you immediate access for fine-tuning and deployment. You can quickly adapt Gemma 3 to your use case using Vertex AI’s pre-built containers and deployment tools. 

In this post, you’ll learn how to fine-tune Gemma 3 on Vertex AI and deploy it as a production-ready endpoint.

Gemma 3 on Vertex AI: PEFT and vLLM deployment

Tuning and deploying large language models can be computationally expensive and time-consuming. That’s why we’re excited to announce Gemma 3 support for Parameter-Efficient Fine-Tuning (PEFT) and optimized deployment using vLLM on Vertex AI Model Garden. 

Gemma 3 fine-tuning allows you to achieve performance gains with significantly less computational resources compared to full fine-tuning. Our vLLM-based deployment is easy-to-use and fast. vLLM’s optimized inference engine maximizes throughput and minimizes latency, ensuring a responsive and scalable endpoint for your Gemma 3 applications on Vertex AI.

Let’s look at how you can fine-tune and deploy your Gemma 3 model on Vertex AI.

Fine-tuning Gemma 3 on Vertex AI

In Vertex AI Model Garden, you can fine-tune and deploy Gemma 3 using PEFT (LoRA) from Hugging Face in only a few steps. Before you run the notebook make sure you complete all of the initial steps as described in the notebook. 

Fine-tuning Gemma 3 on Vertex AI for your use case requires a custom dataset. The recommended format is a JSONL file, where each line is a valid JSON string. Here’s an example inspired by the timdettmers/openassistant-guanaco dataset:

The JSON object has a key text, which should match train_column; The value should be one training data point, i.e. a string. You can upload your dataset to Google Cloud Storage (preferred) or to Hugging Face datasets.

Choose the Gemma 3 variant that best suits your needs. For example, to use the 1B parameter model:

You have the flexibility to customize model parameters and job arguments. Let’s explore some key settings. LoRA (Low-Rank Adaptation) is a PEFT technique that significantly reduces the number of trainable parameters. The following parameters control LoRA’s behavior. lora_rank controls to control dimensionality of the update matrices (smaller rank = fewer parameters), lora_alpha that scales the LoRA updates, and lora_dropout to add regularization. The following settings are a reasonable starting point.

When fine-tuning large language models (LLMs), precision is a key consideration, impacting both memory usage and performance. Lower precision training, such as 4-bit quantization, reduces the memory footprint. However, this can come with a slight performance trade-off compared to higher precisions like 8-bit or float16. The train_precision parameter dictates the numerical precision used during the training process. Choosing the right precision involves balancing resource limitations with desired model accuracy.

Optimizing model performance involves tuning training parameters that impact speed, stability, and capabilities. Essential parameters include per_device_train_batch_size, which determines the batch size per GPU, with larger sizes accelerating training but demanding more memory. gradient_accumulation_steps allows simulating larger batch sizes by accumulating gradients over smaller batches, providing a memory-efficient alternative at the cost of increased training time. The learning_rate dictates the optimization step size, where a rate that is too high can lead to divergence, while a rate that is too low can slow down convergence. The lr_scheduler_type dynamically adjusts the learning rate throughout training, such as through linear decay, fostering better convergence and accuracy. And, the total training duration is defined by either max_steps, specifying the total number of training steps, or num_train_epochs, with max_steps taking precedence if both are specified. Below you have the full training recipe you find in the official notebook.

Finally, create and run the CustomContainerTrainingJob to start the fine-tuning job.

You can monitor the fine-tuning progress using Tensorboard. Once the job is complete, you can upload the tuned model to the Vertex AI Model Registry and deploy it as an endpoint for inference. Let’s dive into deployment next. 

Deploying Gemma 3 on Vertex AI

Deploying Gemma 3 on Vertex AI requires only three steps as described in this notebook. 

First, you need to provision a dedicated endpoint for your Gemma 3 model. This provides a scalable and managed environment for hosting your model. You use the create function to set the endpoint name (display_name), and ensure dedicated resources for your model (dedicated_endpoint_enabled).

Next, register the Gemma 3 model within the Vertex AI Model Registry. Think of the Model Registry as a central hub for managing your models. It keeps track of different versions of your Gemma 3 model (in case you make improvements later), and is the central place from which you’ll deploy.

This step involves a few important configurations including the serving container to deploy Gemma 3. 

To serve Gemma 3 on Vertex AI, use the Vertex AI Model Garden vLLM pre-built Docker image for fast and efficient model serving. The vLLM recipe to set how vLLM will serve Gemma 3 which includes  --tensor-parallel-size lets you spread the model across multiple GPUs if you need extra computation resources, --gpu-memory-utilization controls how much of the GPU memory you want to use and --max-model-len sets the maximum length of text the model can process at once. You also have some advanced settings like --enable-chunked-prefill, and --enable-prefix-caching to optimize performance, especially when dealing with longer pieces of text. 

There are also some of deployment configuration Vertex AI requires to serve the model including the port (8080 in our case) that the serving container will listen on, defines the URL path for making prediction requests (e.g., “/generate”) and the URL path for health checks (e.g., “/ping”), allowing Vertex AI to monitor the model’s status.

Finally, use upload() to take this configuration – the serving container, your model-specific settings, and instructions for how to run the model – and bundle them up into a single, manageable unit within the Vertex AI Model Registry. This makes deployment and version control much easier.

Now you’re ready to deploy the model. To deploy the registered model to the endpoint, use the deploy method as shown below.

This is where you choose the computing power for our deployment including the type of virtual machine (like “a3-highgpu-2g”, machine_type), the kind of accelerator (e.g., “NVIDIA_L4” GPUs, accelerator_type), how many accelerators to use (accelerator_count). 

Deploying the model requires some time and you can monitor the status of the deployment in Cloud Logging. Once you get the endpoint running, you can use the ChatCompletion API to call the model and integrate it within your applications as shown below.

Depending on the Gemma model you deploy, you can use the ChatCompletion API to call the model with multimodal inputs (images). You can find more in the “Deploy Gemma 3 4B, 12B and 27B multimodal models with vLLM on GPU” section of the model card notebook.

What’s next?

Visit the Gemma 3 model card on Vertex AI Model Garden to get started today. For a deeper understanding of the model’s architecture and performance, check out this developer guide on Gemma 3.