GCP – Waze’s journey to Infrastructure as Code with Google Cloud’s KCC

In 2023, the Waze platform engineering team transitioned to Infrastructure as Code (IaC) using Google Cloud’s Config Connector (KCC) — and we haven’t looked back since. We embraced Config Connector, an open-source Kubernetes add-on, to manage Google Cloud resources through Kubernetes. To streamline management, we also leverage Config Controller, a hosted version of Config Connector on Google Kubernetes Engine (GKE), incorporating Policy Controller and Config Sync. This shift has significantly improved our infrastructure management and is shaping our future infrastructure.

The shift to Config Connector

Previously, Waze relied on Terraform to manage resources, particularly during our dual-cloud, VM-based phase. However, maintaining state and ensuring reconciliation proved challenging, leading to inconsistent configurations and increased management overhead.

In 2023, we adopted Config Connector, transforming our Google Cloud infrastructure into Kubernetes Resource Modules (KRMs) within a GKE cluster. This approach addresses the reconciliation issues encountered with Terraform. Config Sync, paired with Config Connector, automates KRM synchronization from source repositories to our live GKE cluster. This managed solution eliminates the need for us to build and maintain custom reconciliation systems.

The shift helped us meet the needs of three key roles within Waze’s infrastructure team: 

1. Infrastructure consumers: Application developers who want to easily deploy infrastructure without worrying about the maintenance and complexity of underlying resources.

2. Infrastructure owners: Experts in specific resource types (e.g., Spanner, Google Cloud Storage, Load Balancers, etc.), who want to define and standardize best practices in how resources are created across Waze on Google Cloud.

3. Platform engineers: Engineers who build the system that enables infrastructure owners to codify and define best practices, while also providing a seamless API for infrastructure consumers.

First stop: Config Connector

It may seem circular to define all of our Google Cloud infrastructure as KRMs within a Google Cloud service, however, KRM is actually a great representation for our infrastructure as opposed to existing IaC tooling.

Terraform’s reconciliation issues – state drift, version management, out of band changes – are a significant pain. Config Connector, through Config Sync, offers out-of-the-box reconciliation, a managed solution we prefer. Both KRM and Terraform offer templating, but KCC’s managed nature aligns with our shift to Google Cloud-native solutions and reduces our maintenance burden. 

Infrastructure complexity requires generalization regardless of the tool. We can see this when we look at the Spanner requirements at Waze:

• Consistent backups for all Spanner databases

• Each Spanner database utilizes a dedicated Cloud Storage bucket and Service Account to automate the execution of DDL jobs.

• All IAM policies for Spanner instances, databases, and Cloud Storage buckets are defined in code to ensure consistent and auditable access control.

To define these resources, we evaluated various templating and rendering tools and selected Helm, a robust CNCF package manager for Kubernetes. Its strong open-source community, rich templating capabilities, and native rendering features made it a natural fit. We can now refer to our bundled infrastructure configurations as ‘Charts.’ While KRO has since emerged that achieves a similar purpose, our selection process predated its availability.

Under the hood

Let’s open the hood and dive into how the system works and is driving value for Waze.

1. Waze infrastructure owners generically define Waze-flavored infrastructure in Helm Charts. 

2. Infrastructure consumers use these Charts with simplified inputs to generate infrastructure (demo).

3. Infrastructure code is stored in repositories, enabling validation and presubmit checks.

Code is uploaded to a Artifact Registry where Config Sync and Config Connector align Google Cloud infrastructure with the code definitions.

Approaching our destination

So why does all of this matter? Adopting this approach allowed us to move from Infrastructure as Code to Infrastructure as Software. By treating each Chart as a software component, our infrastructure management goes beyond simple code declaration. Now, versioned Charts and configurations enable us to leverage a rich ecosystem of software practices, including sophisticated release management, automated rollbacks, and granular change tracking.

Here’s where we apply this in practice: our configuration inheritance model minimizes redundancy. Resource Charts inherit settings from Projects, which inherit from Bootstraps. All three are defined as Charts. Consequently, Bootstrap configurations apply to all Projects, and Project configurations apply to all Resources.

Every change to our infrastructure – from changes on existing infrastructure to rolling out new resource types – can be treated like a software rollout.

Now that all of our infrastructure is treated like software, we can see what this does for us system-wide:

Reaching our destination

In summary, Config Connector and Config Controller have enabled Waze to achieve true Infrastructure as Software, providing a robust and scalable platform for our infrastructure needs, along with many other benefits including: 

• Infrastructure consumers receive the latest best practices through versioned updates.

• Infrastructure owners can iterate and improve infrastructure safely.

• Platform Engineers and Security teams are confident our resources are auditable and compliant

• Config Connector leverages Google’s managed services, reducing operational overhead.