fbpx
Menu
Empower your teams with self-service Kubernetes using GKE fleets and Argo CD

GCP – Empower your teams with self-service Kubernetes using GKE fleets and Argo CD

,

Managing applications across multiple Kubernetes clusters is complex, especially when those clusters span different environments or even cloud providers. One powerful and secure solution combines Google Kubernetes Engine (GKE) fleets and, Argo CD, a declarative, GitOps continuous delivery tool for Kubernetes. The solution is further enhanced with Connect Gateway and Workload Identity.

This blog post guides you in setting up a robust, team-centric multi-cluster infrastructure with these offerings. We use a sample GKE fleet with application clusters for your workloads and a control cluster to host Argo CD. To streamline authentication and enhance security, we leverage Connect Gateway and Workload Identity, enabling Argo CD to securely manage clusters without the need to manage cumbersome Kubernetes Services Accounts.

On top of this, we incorporate GKE Enterprise Teams to manage access and resources, helping to ensure that each team has the right permissions and namespaces within this secure framework.

aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud containers and Kubernetes’), (‘body’, <wagtail.rich_text.RichText object at 0x3e0eb810d220>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectpath=/marketplace/product/google/container.googleapis.com’), (‘image’, None)])]>

Finally, we introduce the fleet-argocd-plugin, a custom Argo CD generator designed to simplify cluster management within this sophisticated setup. This plugin automatically imports your GKE Fleet cluster list into Argo CD and maintains synchronized cluster information, making it easier for platform admins to manage resources and for application teams to focus on deployments.

Follow along as we:

  • Create a GKE fleet with application and control clusters

  • Deploy Argo CD on the control cluster, configured to use Connect Gateway and Workload Identity

  • Configure GKE Enterprise Teams for granular access control

  • Install and leverage the fleet-argocd-plugin to manage your secure, multi-cluster fleet with team awareness

By the end, you’ll have a powerful and automated multi-cluster system using GKE Fleets, Argo CD, Connect Gateway, Workload Identity, and Teams, ready to support your organization’s diverse needs and security requirements. Let’s dive in!

Set up multi-cluster infrastructure with GKE fleet and Argo CD

Setting up a sample GKE fleet is a straightforward process:

1

1. Enable the required APIs in the desired Google Cloud Project. We use this project as the fleet host project.

a. gcloud SDK must be installed, and you must be authenticated via gcloud auth login.

code_block
<ListValue: [StructValue([(‘code’, ‘export FLEET_PROJECT_ID={{_YOUR_FLEET_PROJECT_ID_}}rnexport FLEET_PROJECT_NUMBER = “{{_YOUR_PROJECT_NUMBER_}}”rnrngcloud config set project $FLEET_PROJECT_IDrngcloud services enable container.googleapis.com \rngkehub.googleapis.com \rncloudresourcemanager.googleapis.com \rniam.googleapis.com \rnconnectgateway.googleapis.com \rnanthos.googleapis.com \rncloudbuild.googleapis.com’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0eb810d2e0>)])]>

2. Create application clusters and register them under your fleet host project.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud container clusters create app-cluster-1 –enable-fleet –region=us-central1 rngcloud container clusters create app-cluster-2 –enable-fleet –region=us-central1’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0eb810dca0>)])]>

3. Set up teams on your fleet. Let’s say you have one frontend team with a webserver namespace. 

a. With fleet teams and fleet Namespace, you can control which team accesses specific namespaces on specific clusters.

code_block
<ListValue: [StructValue([(‘code’, ‘# Create a frontend team. rngcloud container fleet scopes create frontendrnrn# Add your application clusters to the frontend team. rngcloud container fleet memberships bindings create app-cluster-1-b \rn –membership app-cluster-1 \rn –scope frontend \rn –location us-central1rnrngcloud container fleet memberships bindings create app-cluster-2-b \rn –membership app-cluster-2 \rn –scope frontend \rn –location us-central1rnrn# Create a fleet namespace for webserver.rngcloud container fleet scopes namespaces create webserver –scope=frontendrnrn# [Optional] Verify your fleet team setup.rn# Check member clusters in your fleet.rngcloud container fleet memberships list rn# Verify member clusters have been added to the right team (`scope`). rngcloud container fleet memberships bindings list –membership=app-cluster-1 –location=us-central1rngcloud container fleet memberships bindings list –membership=app-cluster-2 –location=us-central1’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52760>)])]>

4. Now, set up Argo CD and deploy it to the control cluster. Create a new GKE cluster as your application and enable Workload Identity on it.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud container clusters create control-cluster –region=us-central1 \rn–workload-pool=$FLEET_PROJECT_ID.svc.id.goog’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52a00>)])]>

5. Install the Argo CD CLI to interact with the Argo CD API server. Version 2.8.0 or higher is required. Detailed installation instructions can be found via the CLI installation documentation. 

6. Deploy Argo CD on the control cluster.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud container clusters get-credentials control-cluster –region=us-central1rnkubectl create namespace argocdrnkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb529d0>)])]>

Customize the Argo CD generator

Now you’ve got your GKE fleet up and running, and you’ve installed Argo CD on the control cluster. In Argo CD, application clusters are registered with the control cluster by storing their credentials (like API server address and authentication details) as Kubernetes Secrets within the Argo CD namespace. We’ve got a way to make this whole process a lot easier!

The fleet-argocd-plugin is a customized Argo CD plugin generator that takes the hassle out of cluster management by: 

  • Automatically importing your GKE fleet cluster list into Argo CD and setting up the cluster secret objects for each application cluster 

  • Keeping an eye on your fleet’s status on Google Cloud, making sure your Argo CD cluster list is always in sync and up-to-date

Now, let’s see how to build and configure the Argo CD generator. 

7. Install fleet-argocd-plugin on your control cluster. 

a. In this demo, we use Cloud Build to build and deploy the fleet-argocd-plugin.

code_block
<ListValue: [StructValue([(‘code’, ‘git clone https://github.com/GoogleCloudPlatform/gke-fleet-management.git rncd gke-fleet-management/fleet-argocd-pluginrnrngcloud artifacts repositories create argocd-fleet-sync \rn –repository-format=docker \rn –location=us-central1 \rn –description=”Docker repository for argocd fleet sync”rngcloud builds submit –region=us-central1 –config=cloudbuild.yamlrnexport PATH_TO_IMAGE=us-central1-docker.pkg.dev/$FLEET_PROJECT_ID/argocd-fleet-sync/plugin:v1.0’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52640>)])]>

8. To make sure the fleet-argocd-plugin works as it should, give it the right permissions for fleet management. 

a. Create an IAM service account in your Argo CD control cluster and grant it the appropriate permissions. The setup follows the official onboarding guide of GKE Workload Identity Federation.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud iam service-accounts create argocd-fleet-admin \rn –project=$FLEET_PROJECT_IDrnrngcloud projects add-iam-policy-binding $FLEET_PROJECT_ID \rn–member “serviceAccount:argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com” \rn–role “roles/container.developer”rnrngcloud projects add-iam-policy-binding $FLEET_PROJECT_ID \rn–member “serviceAccount:argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com” \rn–role “roles/gkehub.gatewayEditor”rnrngcloud projects add-iam-policy-binding $FLEET_PROJECT_ID \rn–member “serviceAccount:argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com” \rn–role “roles/gkehub.viewer”rnrn# Allow ArgoCD application controller and fleet-argocd-plugin to impersonate this IAM service account.rngcloud iam service-accounts add-iam-policy-binding argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com \rn–role roles/iam.workloadIdentityUser \rn–member “serviceAccount:$FLEET_PROJECT_ID.svc.id.goog[argocd/argocd-application-controller]”rngcloud iam service-accounts add-iam-policy-binding argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com \rn–role roles/iam.workloadIdentityUser \rn–member “serviceAccount:$FLEET_PROJECT_ID.svc.id.goog[argocd/argocd-fleet-sync]”rnrn# Annotate the Kubernetes ServiceAccount so that GKE sees the link between the service accounts.rnkubectl annotate serviceaccount argocd-application-controller \rn –namespace argocd \rn iam.gke.io/gcp-service-account=argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52fd0>)])]>

b. You also need to allow the Google Compute Engine service account to access images from your artifacts repository.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud projects add-iam-policy-binding projects/$FLEET_PROJECT_ID \rn–role=”roles/artifactregistry.reader” \rn–member=serviceAccount:$FLEET_PROJECT_NUMBER-compute@developer.gserviceaccount.com’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52f10>)])]>

9. Run the fleet plugin on your Argo CD control cluster!

code_block
<ListValue: [StructValue([(‘code’, ‘export FLEET_SA=”argocd-fleet-admin@$FLEET_PROJECT_ID.iam.gserviceaccount.com”rnrnenvsubst ‘$FLEET_SA $FLEET_PROJECT_NUMBER $PATH_TO_IMAGE’ < fleet-sync-install.yaml | kubectl apply -f – -n argocd’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52c40>)])]>

Demo time

Let’s do a quick check to make sure the GKE fleet and Argo CD are playing nicely together. You should see that the secrets for your application clusters have been automatically generated.

code_block
<ListValue: [StructValue([(‘code’, ‘kubectl get secret -n argocdrnrn# Example Output: TYPE DATA AGErn# app-cluster-1.us-central1.141594892609 Opaque 3 64mrn# app-cluster-2.us-central1.141594892609 Opaque 3 64m’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52b50>)])]>

Demo 1: Automatic fleet management in Argo CD

Okay, let’s see how this works! We’ll use the guestbook example app. First, we deploy it to the clusters that the frontend team uses. You should then see the guestbook app running on your application clusters, and you won’t have to deal with any cluster secrets manually!

code_block
<ListValue: [StructValue([(‘code’, “export TEAM_ID=frontendrnenvsubst ‘$FLEET_PROJECT_NUMBER $TEAM_ID’ < applicationset-demo.yaml | kubectl apply -f – -n argocdrnrnkubectl config set-context –current –namespace=argocdrnargocd app list -o name rn# Example Output:rn# argocd/app-cluster-1.us-central1.141594892609-webserverrn# argocd/app-cluster-2.us-central1.141594892609-webserver”), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb52ca0>)])]>

Demo 2: Evolving your fleet is easy with fleet-argocd-plugin

Suppose you decide to add another cluster to the frontend team. Create a new GKE cluster and assign it to the frontend team. Then, check to see if your guestbook app has been deployed on the new cluster.

code_block
<ListValue: [StructValue([(‘code’, ‘gcloud container clusters create app-cluster-3 –enable-fleet –region=us-central1rngcloud container fleet memberships bindings create app-cluster-3-b \rn –membership app-cluster-3 \rn –scope frontend \rn –location us-central1rnrnargocd app list -o namern# Example Output: a new app shows up!rn# argocd/app-cluster-1.us-central1.141594892609-webserverrn# argocd/app-cluster-2.us-central1.141594892609-webserverrn# argocd/app-cluster-3.us-central1.141594892609-webserver’), (‘language’, ”), (‘caption’, <wagtail.rich_text.RichText object at 0x3e0ebbb521f0>)])]>

Closing thoughts

In this blog post, we’ve shown you how to combine the power of GKE fleets, Argo CD, Connect Gateway, Workload Identity, and GKE Enterprise Teams to create a robust and automated multi-cluster platform. By leveraging these tools, you can streamline your Kubernetes operations, enhance security, and empower your teams to efficiently manage and deploy applications across your fleet.

However, this is just the beginning! There’s much more to explore in the world of multi-cluster Kubernetes. Here are some next steps to further enhance your setup:

As you continue your journey with multi-cluster Kubernetes, remember that GKE fleets and Argo CD provide a solid foundation for building a scalable, secure, and efficient platform. Embrace the power of automation, GitOps principles, and team-based management to unlock the full potential of your Kubernetes infrastructure.

Read More for the details.

Related Posts