Cloud

Transportation leaders are facing challenges in providing safe and effective mobility. Road fatalities have reached a 16-year high, extreme weather and climate risks are calling for a more resilient infrastructure, and there’s a growing urgency to address equity gaps. Furthermore, agencies are struggling to attract and retain talent, meaning teams need to be empowered to do more with less. 

Government organizations need modern tools and technology that can deliver insights, speed and scale, so they can focus on bringing their expertise, taking action, and making an impact.

Here are some examples of how these organizations are future-proofing infrastructure in their regions and transforming their day-to-day operations thanks to cloud technology.

Workforce transformation and streamlining business processes is a top priority at the Utah Department of Transportation (UDOT). 

“We’ve taken a cloud-first mentality to drive cost efficiency from the top,” says Executive Director Carlos Braceras. “We are setting big goals, like making our operations 100% paperless, and implementing them with digital transformation. This will improve the ease of doing business with UDOT.” UDOT has also centralized all of its data into an advanced data analytics platform.

 This will make it easier for the teams across the UDOT to access the data and insights they need.

Hawaii’s Department of Transportation Highways Division (HIDOT) is leveraging Google Earth Engine and Google Cloud to deploy a Climate Resilience Platform, which helps them assess risk and prioritize investment decisions based on multiple climate risks, asset conditions, and community impact. Deputy Director Ed Sniffen says, “We are taking an integrated approachto climate resilience, safety, and equity. By using data-driven approaches across agencies, levels of government, and with the public, we can empower and enable decision making.”

The Colorado Department of Transportation (CDOT) has built an advanced analytics platform to bring together a wide range of data sources and apply analytics and machine learning to safety, traffic operations, and performance measurement. “CDOT has undergone significant digital infrastructure modernization to enable interoperability, access and analytics using Google Cloud,” says Ashley Nylen, Assistant Director for Mobility Technology within the Office of Innovative Mobility. “We have been able to break down data and organizational silos to make data and information more accessible and usable for our teams. Now we can leverage that information to enable a more reliable, efficient, and clean transportation system as we strive to achieve ambitious statewide greenhouse gas reduction goals.” 

The District Department of Transportation (DDOT) defines equity as the shared and just distribution of transportation infrastructure and services and works to incorporate equity into all stages of project delivery. “Through collaboration with our technology partners and meaningful engagement with stakeholders, DDOT is working to build equity into decision-making for delivery of transportation services and in contracting for projects,” says DDOT Director Everett Lott. “DDOT’s goal is to address gaps in transportation policy and delivery, access to jobs and opportunities, and inclusion of minority-owned businesses in awarding contracts. With better data and insight, we can improve access to jobs and opportunities and streamline business processes to lower the barrier to entry for Minority Business Enterprises (MBEs) and Disadvantaged Business Enterprises (DBEs).”

While technology is not the whole answer, it is an important enabler for mission-driven transformation. Each agency may be different in where they start and what areas they tackle. The important thing is to take the first steps in an area that is meaningful to the organization.

To learn more from these transportation leaders about how they are using digital tools to improve safety, climate resilience, and organizational efficiency in their departments, attend our panel on “Making the AI Connection for Next-generation Transportation” at the ITS World Congress on September 18, 2022 or contact our team. Also, pop by meeting room 304C at the Los Angeles Convention Center to meet with our Google Public Sector leadership team during the conference. Click here to learn more about Google Cloud sustainability offerings.

As part of our ongoing efforts to enable developers to further style and customize the Google basemap, today we’re announcing the launch of advanced polyline styling for the Maps SDK for Android. This feature builds on the polylines capability we currently offer, which allows developers to create stylized lines rendered on the map, and are often used to represent information about directions, parade routes, and street closures. Polylines are one of the most common features Google Maps Platform developers use to customize maps for their users. 

Advanced polyline styling gives you new ways to make this core feature more informational and expressive by introducing three new polyline customizations: multicolored polylines, gradient polylines, and stamped polylines.

Advanced polyline styling is available today when you use version 18.1.0 or later of the Maps SDK for Android with the latest renderer enabled, and version 5.2.0 of the Maps SDK for iOS.

Multicolored polylines

Multicolored polylines allow you to segment polylines, and represent information by applying different colors to each segment. This customization type has many use cases, such as conveying varied information along a route, matching a company’s branding for a seasonal campaign, or drawing a user’s attention to an area.

To use a multicolored polyline, create `StyleSpan` objects and add them to `PolylineOptions` using the `addSpan()` or `addSpans()` methods. By default, each item in the array will set the color of the corresponding line segment. The following example shows setting segment colors to create a polyline with green and purple segments:

Gradient polylines 

Gradient polylines apply a two-color gradient over the entire polyline, which can be useful for representing changing intensities, such as traffic flow or a change in elevation.

Here’s an example that creates a gradient polyline from San Francisco to Rancho Corral de Tierra:

Stamped polylines

With stamped polylines, you can create a polyline using a repeating bitmap image of your choice. This allows you to customize the shape and appearance of the polyline.

To use a stamped polyline, set the appearance of a polyline to a repeating bitmap texture by creating a `StampStyle` of `TextureStyle`. You can set this property on the shape’s options object by calling `PolylineOptions.addSpan()`:

Styling and customization features are one of the best and easiest ways to improve the mapping experience for your users, convey critical information, and make the map your own. Along with the recent release of other styling and customization features like Cloud-based maps styling, Data-driven styling, and WebGL-powered maps features, our team is working hard to give you the tools you need to create great mapping experiences that have the flexibility to meet your individual needs. We hope you enjoy this new way to customize and style polylines on your maps.

For more information on Google Maps Platform, visit our website.

As part of our efforts to democratize Zero Trust, Google Cloud has designed our BeyondCorp Enterprise solution to be an extensible platform where customers can choose to integrate signals from other technology vendors and incorporate these into their Zero Trust access policies. Following our integrations announcements earlier this year, we are excited to announce a new BeyondCorp Enterprise integration with Microsoft Intune, now available in Preview.

This integration allows organizations to craft Zero Trust access policies and protect private applications and SaaS applications, including Office 365, based on data collected from the Intune graph API, including device posture and other trust signals. It can also be leveraged to configure context-aware access policies for Workspace applications. 

These policies can be applied across end-user devices, no matter where they are located. The ability to leverage device information to make access decisions is a critical component of a Zero Trust approach. Intune is a widely used mobile device management (MDM) tool and many of our customers will be able to benefit from this integration to help ensure that their distributed workforce can appropriately access corporate resources. 

The BeyondCorp Enterprise integration with Microsoft Intune collects data from Intune using the Microsoft Graph API. End-user device information collected by the connector is then fed into Access Context Manager, a component of BeyondCorp Enterprise, to gate access to resources based on policies and access levels.

Customers are already seeing the benefits of the BeyondCorp Enterprise integration with Microsoft Intune. For example, one of our customers, a global ecommerce vendor, uses it to ensure their corporate-owned devices comply with internal policies before they are able to connect to corporate resources. With the Intune integration, they are then able to quickly and easily configure context-aware access policies with an attribute that company-owned devices must be in compliance in order to access the specific applications. 

Previously, without the integration, they would need to set up a custom integration and manage both the code and the infrastructure where it was running. This integration not only alleviates the need to create custom code, but the customer has also seen a reduction in the time it takes to onboard new devices and build these policies.

If you’re interested in learning more or joining the preview, a full reference guide to the Intune integration can be found in our documentation here.

We believe that customers should be able to leverage their existing technology investments to build a more secure ecosystem. BeyondCorp Enterprise can help ensure that the right people have access to the right resources—only authorized users should be able to access only the resources that they have been approved for, based on their identity and device information. Google Workspace customers can also incorporate signal information from other vendors, including Intune, to create context-aware access policies for securing Workspace applications. 

Earlier this year, we announced Netskope as a new member of the BeyondCorp Alliance to enable integration of a user’s risk score between Netskope Cloud Exchange and Google Cloud. We also announced new integrations with Jamf Pro for MacOS, which shares the Jamf-determined compliance state with BeyondCorp Enterprise so admins can incorporate this information into context-aware policies to restrict or allow access to protected applications.

You can learn more about BeyondCorp Enterprise integrations by registering for Google Cloud Next ‘22 on October 11-13, and attending the “What’s New in Zero Trust” session. Google Cloud will also be featured at the upcoming Jamf Nation User Conference (JNUC), September 27-29, to discuss BeyondCorp Enterprise integrations with Jamf.

As enterprises look to accelerate cloud adoption, it is critical to not only upskill your technical talent, but to focus on skilling your non-technical teams too. Investing in your collective workforce’s cloud proficiency helps ensure you fully embrace everyone’s potential, and make the most of your cloud investment.

According to research shared in a recent IDC paper1, comprehensively trained organizations saw a bigger impact vs. narrowly trained organizations, with 133% greater improvement in employee retention, a 47% reduction in business risk and a 22% increase in innovation. 

This is where Cloud Digital Leader training and certification comes in. Most cloud training and certification is geared toward technical cloud practitioners, leaving non-technical (tech-adjacent) teams with little understanding of cloud technologies. Cloud Digital Leader bridges this gap, providing easy-to-understand training that enables everyone to understand the capabilities of cloud so that they can contribute to digital transformation in their organizations.

In  a recent fireside chat with Google Cloud Partner Kyndryl, who have achieved over 1,000 Cloud Digital Leader certifications across their organization, they shared how the Cloud Digital Leader training and certification has led to significant time reduction within their pre-sales cycle:

“Our sales teams who work with customers and learn about their challenges were able to apply the know-how from their Cloud Digital Leader education and certification. They can now guide the technical solution teams in the right direction, without having to pull them into the discovery phases of their customer interactions. As a result, we operated more quickly and efficiently,  as the sales teams were able to speak to the Google Cloud solutions very early on in the sales cycle. This accelerated the sales process, as the sales teams were therefore more confident in their Google Cloud knowledge, saving time and money for us, and the customer.” — Christoph Schwaiger, Google Cloud Business Development Executive, Global Strategic Alliances, Kyndryl.

Empower your team’s cloud fluency, and discover your next phase of digital transformation. Invite your teams to jump start their cloud journey with no-cost Cloud Digital Leader training on Google Cloud Skills Boost.

Join our live webinar to access a time-limited certification offer

Register for our upcoming webinar, “Getting started with Google Cloud Digital Leader training and certification” to learn more. 

Those that register for the webinar before broadcast on September 15, 9am PT will get access to a time-limited discount voucher for the Cloud Digital Leader certification exam. That’s an offer that you won’t want to miss.

1. IDC Paper, sponsored by Google Cloud Learning: “To Maximize Your Cloud Benefits, Maximize Training” – Doc #US48867222, March 2022

DevOps is a concept that allows software development teams to release software in an automated and stable manner. DevOps itself is not just one thing; it’s a combination of culture and technology, which together make the implementation of DevOps successful.

In this blog, we will be focusing on the tools and technology side of DevOps. At the core of the technical aspect of DevOps, the concept is Continuous Integration and Continuous Delivery (CI/CD). The idea behind CI/CD concept is to create an automated software delivery pipeline that continuously deploys the new software releases in an automated fashion.

The flow begins with the developers committing the code changes to a source code repository, which automatically triggers the delivery pipeline (henceforth called CI/CD pipeline) by building and deploying the code changes into various environments, from non-prod environments to production environments.

Also, as we build the CI/CD pipelines for faster and more reliable software delivery, the security aspect should not be ignored and must be incorporated into the pipeline right from the beginning. When we build our source code, we typically use various open-source libraries and container images. Having some security safeguards within the CI/CD pipeline is imperative to ensure that the software we are building and deploying is free from any vulnerability. Additionally, it’s equally important to control what type of code/container image should be allowed to be deployed on your target runtime environment.

Security is everyone’s responsibility. Shifting left on security is a DevOps practice that allows you to address security concerns early in the software development lifecycle. Vulnerability scanning of container images, putting security policies in place through Binary Authorization, and allowing approved/trusted images to be deployed on GKE are a couple of ways to implement this policy to make your CI/CD pipelines more secure.

What are we building?

This blog post will show how to build a secure CI/CD pipeline using Google Cloud’s built-in services. We will create a secure software delivery pipeline that builds a sample Node.js application as a container image and deploys it on GKE clusters.

How are we building the CI/CD pipeline?

We’re going to use the following Google Cloud built-in services to build the pipeline:

Cloud Build – Cloud Build is an entirely serverless CI/CD platform that allows you to automate your build, test, and deploy tasks.

Artifact Registry – Artifact Registry is a secure service to store and manage your build artifacts.

Cloud Deploy – Cloud Deploy is a fully managed Continuous Delivery service for GKE and Anthos.

Binary Authorization – Binary Authorization provides deployment time security controls for GKE and Cloud Run deployments.

GKE – GKE is a fully managed Kubernetes platform.

Google Pub/Sub – Pub/Sub is a serverless messaging platform.

Cloud Functions – Cloud Functions is a serverless platform to run your code.

We use GitHub as a source code repository and Sendgrid APIs to send email notifications for approval and error logging.

The CI/CD pipeline is set up so that a Cloud Build trigger is configured to sense any code pushed to a particular repository and branch in a GitHub repository and automatically starts the build process.

Below is the flow of how the CI/CD pipeline is set up without any security policy enforcement:

Developer checks in the code to a GitHub repo.

A Cloud Build trigger is configured to sense any new code pushed to this GitHub repo and starts the ‘build’ process. A successful build results in a docker container image.

The container image is stored in the Artifact Registry.

The Build process kicks off a Cloud Deploy deployment process that deploys the container image to three different GKE clusters, pre-configured as the deployment pipeline mimicking the test, staging, and production environments.

Cloud Deploy is configured to go through an approval step before deploying the image to the Production GKE cluster.

A Cloud Function sends an email to a pre-configured email id, notifying you that a Cloud Deploy rollout requires your approval. The email receiver can approve or reject the deployment to the production GKE cluster. Cloud Function code can be found here

To secure this CI/CD pipeline, we will use a couple of Google Cloud’s built-in features and services. First, we will enable vulnerability scans on Artifact Registry, an out-of-the-box feature. Then finally, we will create a security policy using the Binary Authorization service, which only allows a specific image to be deployed to your GKE cluster.

Below is the flow when we try to build and deploy a container image that has vulnerabilities present:

Developer checks in the code to a GitHub repo.

A Cloud Build trigger is configured to sense any new code pushed to this GitHub repo and start the ‘build’ process.

The build process fails with the error message that vulnerabilities were found in the image.

Below is the flow when we try to deploy a container image to GKE, which violates a Binary Authorization policy:

Developer checks in the code to a GitHub repo.

A Cloud Build trigger is configured to sense any new code pushed to this GitHub repo and start the ‘build’ process. A successful build results in a docker container image.

The container image is stored in Artifact Registry.

The Build process kicks off a Cloud Deploy deployment process that deploys the container image to three different GKE clusters, pre-configured as the deployment pipeline mimicking the test, staging, and production environments.

Cloud Deploy fails as the GKE clusters reject the incoming image as it violates the existing Binary Authorization policy. Please note that an approval email is still triggered before the production deployment via the Cloud Function; the email receiver is expected to reject this release based on the failures in the previous stages.

Once the deployment fails due to the Binary Authorization policy violation, Cloud Function sends an email to a pre-configured email id about the deployment failure. Cloud Function code can be found here.

Note: The deployment fails after the timeout value is exceeded, set for Cloud Deploy, which is 10 minutes by default, but you can change this value according to your requirements, see here for more details.

Note: The Cloud Function code provided for the rollout approval email and deployment failure notification is under the folder cloud-functions in this repo. You will still have to create these cloud functions with this code in your Google Cloud project to receive email notifications.

Solution Architecture

The CI/CD pipeline is constructed by combining the aforementioned Google Cloud services. Cloud Build is at the center of automating the pipeline, which contains all the steps we need to build and deploy our container image. Cloud Build executes the steps defined in a YAML file sequentially. It’s quite flexible in terms of how you want to define your ‘build’ and ‘deploy’ process, and the service ensures to execute those steps reliably every time.

Below are solution diagrams of how the CI/CD pipeline is set up :

As the last step of our CI process, the Cloud Build YAML triggers the Cloud Deploy service, and the container image is deployed to three different GKE clusters. Cloud Deploy automatically emits multiple notifications to pub/Sub topics throughout the deployment process. We are using Cloud Functions to listen to these Pub/Sub topics to send appropriate email notifications about the deployment status and required approvals.

Step-by-Step instructions for creating the CI/CD pipeline

I. Prerequisites

These steps are required to set up and prepare your GCP environment. We highly recommend you create a new GCP Project as you will run multiple cloud services within the region “us-central1”.

Fork the following GitHub Repo: https://github.com/sysdesign-code/dev-sec-ops-demo

Create a new GCP Project, follow the steps here around how to provision and create one: https://cloud.google.com/resource-manager/docs/creating-managing-projects

Once your new project is created, enable Cloud SDK to allow CLI access for gcloud either in Cloud Shell or your local workstation. Follow the steps here: https://cloud.google.com/sdk/docs/install

Once you’ve enabled CLI access, either through your Cloud Shell or local workstation, validate or set your project ID:
gcloud config set project YOUR_PROJECT_ID

Run the following one-time script /scripts/gcp_env_setup.sh, which creates and provisions the necessary GCP cloud services required to create the DevSecOps CI/CD pipeline for deploying a sample docker application.

Here are all the service deployments that will occur once the script finishes:

a) Enables all the required cloud service APIs such as Cloud Build, Binary Authorization, Kubernetes Service, Artifact Registry, Cloud Deploy, and many more.

b) Create three (3) GKE clusters for test, staging, and production to show image rollout deployments across these clusters using Cloud Deploy.

c) Bind all the necessary IAM roles and permissions for Cloud Build and Cloud Deploy.

d) Create a Binary Authorization attestor, associated container note, cryptographic KMS key, and all the associated IAM roles and permissions to allow container note access for the attestor.

By default, the binary authorization policy allows for all images to be deployed to GCP. Later, we will update this policy only to allow attestor-approved images to be deployed to specific GKE clusters.

e) Create the Artifact Registry repository where the docker image will be stored.

6. To execute the script, run the following command: sh /scripts/gcp_env_setup.sh

g) This script will approximately take 20-22 minutes to complete. Once finished, the output should look similar to something like this.

7. Create a SendGRID API Key. Follow the instructions: https://app.sendgrid.com/guide/integrate to create a free “Web API” email integration for cURL and its associated API key. Take note and save your key value and verify the integration. The key details will be needed when you create the Cloud Deploy approval process later in this blog. Note: Using SendGRID APIs DOES require you to create a user account.

II. Configure Cloud Build

This step requires integrating your git repository (from Pre-Requisites, Step 1) as a managed repository to GCP’s cloud build service and creating the necessary Trigger. The goal of this integration is that any updates you make to your application within your GitHub repository will automatically kick off a Cloud Build deployment which will create, enable and deploy your application to GKE.

Create the GitHub Repository Integration for Cloud Build :

To start, from your GCP Console homepage, type “Cloud Build” within the search bar and select this service.

From the left-hand panel, click on “Triggers”. And click on “Connect Repository.”

Select the source as “GitHub (Cloud Build GitHub App)

Authenticate the connection with your GitHub credentials, select the forked repository, and click “Connect”.

Once the integration is done, you will see your newly added repository under “Triggers” -> “Manage Repositories.”

Create a Trigger for Cloud Build

From the “Triggers” page, click on “+ Create Trigger”

Enter/Select the following values for the Trigger:

Name: CI/CD-blog-trigger

Region: us-central1

Description: Deploy Docker Image using GCP CI/CD cloud services.

Event: Push to a branch.

Repository: Select your forked repository

Branch: ^main$

Configuration: Cloud Build Configuration File (YAML or JSON)

Location: Repository

Cloud Build configuration file location: / cloudbuild.yaml

Under “Advanced”, add the following TWO environment variables and their values: _CONTAINER_REPO_NAME: test-repo _SEVERITY: CRITICAL
NOTE: The value of these env variables is case sensitive.

3. After the environment values are entered/selected, click “Create”.

Once the Trigger is created, it will look like the following:

III. Create Cloud Deploy Pipeline

Now that we have created GitHub integration and Cloud Build Trigger, the next step is to create the Cloud Deploy pipeline. This will deploy the container image to the three GKE environments: “test,” “staging,” and “prod” once the image release for all three environments is created through Cloud Build. The requirement for image release requires a Cloud Deploy pipeline.

Edit the clouddeploy.yaml file with your GCP project ID.

Within the file, update lines 22, 32, and 42 with your respective GCP project ID

3. Once this is updated, save the file.

4. Either through Cloud Shell or your local workstation, run the following GCP command to create the environment variables and the Cloud Deploy pipeline called ci-cd-test:

NOTE: If you run into issues with a failed Cloud Deploy pipeline creation, delete the pipeline using the following gcloud command:

5. Once the pipeline is created, here is what the output will look like:

6. From your GCP Console homepage, type “Cloud Deploy” within the search bar and select this service. From the main page, you will see the newly created pipeline.

IV. Configure email notifications for GKE production cluster deployment

As part of a typical CI/CD process, any deployment of production workloads requires some form of approval process by DevOps engineers. Cloud Deploy allows you to inject an ‘approval’ step before deploying a rollout to the next target. We have created this approval check in our pipeline before the deployment to the ‘prod’ GKE cluster. Once the pipeline reaches the step to deploy the rollout to the ‘prod’ GKE cluster, it emits a message in the clouddeploy-approvals Pub/Sub topic. We have created a Cloud Function to listen to this topic and implement logic to send email notifications via Sendgrid. You can use any other library to send emails via Cloud Functions.

The one-time script has created a Pub/Sub topic and Cloud Function, allowing your cloud build release to send an approver email.

To validate that the Pub/Sub topics and Cloud Function was created, go to those respective services and ensure they were created.

From your GCP Console homepage, type “Pub/Sub” within the search bar and select this service. There will be two Pub/Sub topics, and they’re called clouddeploy-approvals and clouddeploy-operations.

From your GCP Console homepage, type “Cloud Functions” within the search bar and select this service. There will be two Cloud Functions, called cd-approval and cd-deploy-notification.

Click on cd-approval and select “Variables”.

Click the “Edit” button and expand the `Runtime, build, connections and security settings.

Scroll down until you get to the “Runtime environment variables.” Here you will update the following three variables.

For FROM_EMAIL, enter a secondary email account; it could be @gmail or any other domain of your choice. For TO_EMAIL, select a primary email. For instance, the email of a DevOps Engineer who will be the approver of all production workload deployments to GKE. For SENDGRID_API_KEY, you will enter your API Key, starting with “SG.”. If you haven’t already, refer to the Prerequisites section above, step 6, around creating this key.

6. After you’ve updated the cloud function environment variables, click “Next” and “Deploy” the updated function. It will take about 1-2 minutes. Once completed, the function will have a green check mark to validate its running.

7. Repeat steps 4-6 from above for the other cloud function of cd-approval.

Step-by-step instructions of testing and validating the GCP CI/CD pipeline

Now that all the GCP prerequisites and environment setup is complete for Cloud Build, Cloud Deploy, and Email approvals, we’ll next deploy the image to GKE and initiate the pipeline testing.

A couple of items to note during this test, we’re going to show a “Happy” and “Vulnerable” Image deployment path to GKE.

The “Happy” path will show a successful deployment of the end-to-end pipeline across nine steps for a clean image deployment to GKE. “Clean” refers to the docker image with non-critical vulnerabilities. This path will also update the Binary Authorization policy that allows only the “Happy” image to be deployed to GKE’s “test”, “staging”, and eventually “production” environments, which a DevOps engineer will approve.

The “Vulnerable” docker path will show a failed deployment of the end-to-end pipeline across seven steps. The pipeline will fail in 2 of these steps because the image has:

Certain vulnerabilities must be addressed before the image can be stored in the Artifact Registry.

A failed deployment to GKE because this is a non-approved image without attestation, violating the updated Binary Authorization policy from the “Happy” path.

When Binary Authorization is enabled, its default policy allows all images to be deployed to the GKE target environments without attestation. In the “Happy” path, we will update the default Binary Authorization policy where only a specific docker image is approved for deployment to GKE. GKE will reject any other image not approved by the Binary Authorization policy at the deployment time.

To allow other images to be deployed to GKE through an active binary authorization policy, update the following script /scripts/create_binauthz_policy.sh where you can sign the image digest to the existing attestor and allow for that image deployment to GKE.

In the following sections, we’ll go into further detail describing both paths of image deployment to GKE.

I. Run Cloud Build configuration file for “Happy” path

Ensure your GitHub repo is connected as a repository in Cloud Build. Refer to the “Create the GitHub Repository Integration for Cloud Build” section on how to do this.

Ensure your Cloud Build trigger called CI/CD-blog-trigger is created. Refer to the section “Create a Trigger for Cloud Build” on how to do this.

Since the Trigger is already enabled, any updates to your repository will trigger this Cloud Build deployment.

Open up the cloudbuild.yaml from your GitHub repo. This is the cloud build configuration file for the “Happy” Docker path.

To kick off the build, make any update to your codebase such as update the /src/static/js file for any cosmetic change.

After you’ve made the change, push the changes to your GitHub repo.

From the GCP Console, go to the Cloud Build service and click on “History”.

Since the Trigger is enabled and integrated with your GitHub page, the build is automatically kicked off, and you can click the custom build number to see the log details.

II. Validate image deployment for “Happy” path

Within that build, steps 7-9 highlight the image deployment to GKE through Cloud Deploy. If you click on step 9, the result of the build states that the deployment to “prod” is awaiting approval.

2. Go to the Cloud Deploy homepage from the GCP Console and click on the ci-cd-test pipeline.

3. Within the pipeline, click on the release associated with the latest cloud build deployment. Here you see that the “Happy” image is deployed successfully to both “test” and “staging”, but there’s an approval process required for the “prod” cluster.

4. From the GCP Console, search for Kubernetes Engine; from the left-hand navigation, click on “Workloads.” Here you can see that the image deployment is successful in the two “test” and “staging” GKE environments.

5. Now that the deployment is queued for production, check your primary email and validate that you received a notification for approval. It will look something like this.

6. From the email, click the here hyperlink and it will take you to the Cloud deploy pipeline page.

7. From the Pipeline page, approve or reject the release so the deployment can be pushed to “prod” in GKE. In this case, we will approve.

7. If you go back to the Kubernetes workload page, you’ll see that the image rollout to prod was successful.

In parallel, validate your Cloud Deploy, continuous deployment pipeline also confirms a successful rollout.

III. Run Cloud Build configuration file for “Vulnerable” path (container image has vulnerabilities)

We will show two failure paths with this deployment: image vulnerabilities and Binary Authorization policy enforcement.

A. First, failed deployment to push docker image to Artifact Registry because of severity-specific vulnerabilities –

1. Ensure your GitHub Repo is connected as a repository in Cloud Build. Refer to the “Create the GitHub Repository Integration for Cloud Build” section on how to do this.

2. Ensure your Cloud Build Trigger called CI/CD-blog-trigger is created. Refer to the section “Create a Trigger for Cloud Build” on how to do this.

3. Since the Trigger is already enabled, any updates to your repository will trigger this cloud build deployment.

4. View the cloudbuild-vulnerable.yaml file from your GitHub repo. This is the cloud build configuration file for the “Vulnerable” Docker path.

5. Edit the existing Trigger with the following:

Click on the ellipses next to “RUN” and update the “Cloud Build configuration file location” to be: cloudbuild-vulnerable.yaml

Update the “_SEVERITY” environment variable value to be HIGH. We’re changing the severity of the vulnerabilities because the vulnerability check will either PASS or FAIL a cloud build deployment if the image contains ANY HIGH vulnerabilities.

Save the Trigger and validate its status as “Enabled”.

6. To kick off the build, make any update to your codebase, such as updating the /src/static/js file for any cosmetic change. After you’ve made the change, push the changes to your GitHub repo.

7. From the GCP Console, go to the Cloud Build service and click on “History”.

8. The build will fail in Step 2: Check For Vulnerabilities within the Image because this image contains HIGH vulnerabilities, and cloud build will NOT push this image to be stored in the artifact registry.

B. Second, a failed image deployment to GKE because of Binary Authorization policy enforcement –

Go back to the Trigger configuration for this build and change the “_SEVERITY” environment variable value to CRITICAL instead of “HIGH”.

To kick off the build, make any update to your codebase such as update the /src/static/js file for any cosmetic change. After you’ve made the change, push the changes to your GitHub repo.

From the GCP Console, go to the Cloud Deploy pipeline ci-cd-test and check the results of this latest release.

From the Cloud Deploy pipeline page, approximately 10 minutes later, the build for “test” and “staging” will eventually fail because the Kubernetes manifest file for this docker image timed out.

You can change the timeout period to be shorter; additional details can be found here

5. From the GCP Console, go to the GKE page and click on “Workloads”. Here you will see the image deployments to both the “test” and “staging” GKE environments failed. The reason being is binary authorization policy enforcement. The “vulnerable” docker image is not approved for deployment.

6. In parallel to a failed deployment to any of the GKE staging environments, Cloud Function cd-deploy-notification will send the following email to check the logs for the pipeline.

7. From the email, click on here to see deployment logs, and it will take you to the log files within cloud build around additional details on the failure of the release rollout to GKE.

Conclusion and further reading

In this blog post, we built a secure CI/CD pipeline using Google Cloud’s built-in services.

We learned how we can secure a CI/CD pipeline using Google Cloud’s built-in services, such as Binary Authorization and Vulnerability scanning of the container images. We only saw one way to put some control on specific images that can be deployed to a GKE cluster. Binary Authorization also offers Build Verification, in which Binary Authorization uses attestations to verify that an image was built by a specific build system or continuous integration (CI) pipeline such as Cloud Build.

Additionally, Binary Authorization also writes all the events where the deployment of a container image is blocked due to the constraints defined by the security policy to the audit logs. You can create alerts on these log entries and notify the appropriate team members about the blocked deployment events.

Lastly, all of the services used to build and secure the CI/CD pipelines are serverless, which makes it very easy to spin up the whole infrastructure within a few minutes without worrying about maintaining or managing it, so that your teams can focus on building and releasing software in a faster, reliable and cost efficient manner.

You’re working on a new machine learning problem, and the first environment you use is a notebook. Your data is stored on your local machine, and you try out different model architectures and configurations, executing the cells of your notebook manually each time. This workflow is great for experimentation, but you quickly hit a wall when it comes time to elevate your experiments up to production scale. Suddenly, your concerns are more than just getting the highest accuracy score.

Sound familiar?

Developing production applications or training large models requires additional tooling to help you scale beyond just code in a notebook, and using a cloud service provider can help. But that process can feel a bit daunting. 

To make things a little easier for you, we’ve created the Prototype to Productionvideo series, which covers all the foundational concepts you’ll need in order to build, train, scale, and deploy machine learning models on Google Cloud using Vertex AI.

Let’s jump in and see what it takes to get from prototype to production!

Getting started with Notebooks for machine learning

Episode one of this series shows you how to create a managed notebook using Vertex AI Workbench. With your environment set up, you can explore data, test different hardware configurations, train models, and interact with other Google Cloud services.

Storing data for machine learning

When working on machine learning problems, it’s easy to be laser focused on model training. But the data is where it all really starts.

If you want to train models on Vertex AI, first you need to get your data into the cloud. In episode 2, you’ll learn the basics of storing unstructured data for model training and see how to access training data from Vertex AI Workbench.

Training custom models on Vertex AI

You might be wondering, why do I need a training service when I can just run model training directly in my notebook? Well, for models that take a long time to train, a notebook isn’t always the most convenient option. And if you’re building an application with ML, it’s unlikely that you’ll only need to train your model once. Over time, you’ll want to retrain your model to make sure it stays fresh and keeps producing valuable results. 

Manually executing the cells of your notebook might be the right option when you’re getting started with a new ML problem. But when you want to automate experimentation at scale, or retrain models for a production application, a managed ML training option will make things much easier.

Episode 3 shows you how to package up your training code with Docker and run a custom container training job on Vertex AI. Don’t worry if you’re new to Docker! This video and the accompanying codelab will cover all the commands you’ll need.

CODELAB: Training custom models with Vertex AI

How to get predictions from an ML model 

Machine learning is not just about training. What’s the point of all this work if we don’t actually use the model to do something? 

Just like with training, you could execute predictions directly from a notebook by calling model.predict. But when you want to get predictions for lots of data, or get low latency predictions on the fly, you’re going to need something more than a notebook. When you’re ready to use your model to solve a real world problem with ML, you don’t want to be manually executing notebook cells to get a prediction.

In episode 4, you’ll learn how to use the Vertex AI prediction service for batch and online predictions.

CODELAB: Getting predictions from custom trained models

Tuning and scaling your ML models

By this point, you’ve seen how to go from notebook code, to a deployed model in the cloud. But in reality, an ML workflow is rarely that linear. A huge part of the machine learning process is experimentation and tuning. You’ll probably need to try out different hyperparameters, different architectures, or even different hardware configurations before you figure out what works best for your use case.

Episode 5, covers the Vertex AI features that can help you with tuning and scaling your ML models. Specifically, you’ll learn abouthyperparameter tuning, distributed training, and experiment tracking.

CODELAB: Hyperparameter tuning on Vertex AI
CODELAB: Distributed Training on Vertex AI

We hope this series inspires you to create ML applications with Vertex AI! Be sure to leave a comment on the videos if you’d like to see any of the concepts in more detail, or learn how to use the Vertex AI MLOps tools. 

If you’d like try all the code for yourself, check out the following codelabs:

Training custom models with Vertex AI

Getting predictions from custom trained models

Hyperparameter tuning on Vertex AI

Distributed training on Vertex AI

There’s an explosion in the number of digital images generated and used for both personal and business needs. On e-commerce platforms and online marketplaces for example, product images and visuals heavily influence the consumer’s perception, decision making and ultimately conversion rates. In addition, there’s been a rapid shift towards user-generated visual content for ecommerce — think seller-generated product imagery, host-generated rental property photos and influencer-generated social media content. The challenge? These user-generated images are often captured using mobile cameras and vary greatly in terms of their size, quality, compression ratios and resolution, making it difficult for companies to provide consistent high-quality product images on their platforms.

That’s exactly the problem that Let’s Enhance, a computer vision startup with teams across the US and Ukraine, set out to solve with AI. The Let’s Enhance platform improves the quality of any user-generated photo automatically using AI-based features to enhance images through automatic upscaling, pixelation and blur fixes, color and low-light correction, and removing compression artifacts — all with a single click and no professional equipment or photo-editing chops. 

“Let’s Enhance.io is designed to be a simple platform that brings AI-powered visual technologies to everyone — from marketers and entrepreneurs to photographers and designers,” said Sofi Shvets, CEO and Co-founder of Let’s Enhance.

To date, Let’s Enhance has processed more than 100M photos for millions of customers worldwide for use-cases ranging from digital art galleries, real estate agencies, digital printing, ecommerce and online marketplaces. With the introduction of Claid.ai, their  new API to automatically enhance and optimize user-generated content at scale for digital marketplaces, they needed to process millions of images every month and manage sudden peaks in user demand.

However, building and deploying an AI-enabled service at scale for global use is a huge technical challenge that spans model building, training, inference serving and resource scaling. It demands an infrastructure that’s easy to manage and monitor, can deliver real-time performance to end customers wherever they are and can scale as user-demand peaks, all while optimizing costs. 

To support their growing user-base, Let’s Enhance chose to deploy their AI-powered platform to production on Google Cloud and NVIDIA . But before diving into their solution, let’s take a close look at the technical challenges they faced in meeting their business goals. 

Architecting a solution to fuel the next wave of growth and innovation

To deliver the high-quality enhanced images that end-customers see, Let’s Enhance products are powered by cutting-edge deep neural networks (DNNs) that are both compute-  and memory-intensive. While building and training these DNN models in itself is a complex, iterative process,   application performance — when processing new user-requests, for instance — is crucial to delivering a quality end user experience and reducing total deployment costs. 

A single inference or processing request i.e., from user-generated image, which can vary widely in size, at the input to the AI-enhanced output image, requires combining multiple DNN models within an end-to-end pipeline. The  key inference performance metrics to optimize for included latency (the time it takes from providing an input image to the enhanced image being available) and throughput ( the number of images that can be processed per second).. 

Together, Google Cloud and NVIDIA technologies provided all the elements that the Let’s Enhance team needed to set themselves up for growth and scale. There were three main components in the solution stack:

Compute resources: A2 VMs, powered by NVIDIA A100 Tensor Core GPUs

Infrastructure management: Google Kubernetes Engine (GKE)

Inference serving: NVIDIA Triton Inference Server

Improved throughput and lower costs with NVIDIA A100 Multi-Instance GPUs (MIG) on Google Cloud

To meet the computational requirements of the DNN models and deliver real-time inference performance to their end-users, Let’s Enhance chose Google Cloud A2 VMs powered by NVIDIA A100 Tensor Core GPUs as their compute infrastructure. A100 GPU’s Multi-Instance GPU (MIG) capability offered them the unique ability to partition a single GPU into two independent instances and simultaneously process two user-requests at a time, making it possible  to service a higher volume of user requests while reducing the total costs of deployment.

The A100 MIG instances delivered a 40% average throughput improvement compared to the NVIDIA V100 GPUs, with an increase of up to 80% for certain image enhancement pipelines using the same number of nodes for deployment. With improved performance from the same sized node pools, Let’s Enhance observed a 34% cost savings using MIG-enabled A100 GPUs.

Simplified infrastructure management with GKE

To offer a guaranteed quality-of-service (QoS) to their customers and manage user demand, Let’s Enhance needed to provision, manage and scale underlying compute resources while keeping utilization high and costs low. 

GKE offers industry-leading capabilities for training and inference such as support for 15,000 nodes per cluster, auto-provisioning, auto-scaling and various machine types (e.g. CPU, GPU and on-demand, spot), making it the perfect choice for Let’s Enhance.

With support for NVIDIA GPUs and NVIDIA GPU sharing capabilities, GKE can provision multiple A100 MIG instances to process user requests in parallel and maximize utilization. As the compute required for the deployed ML pipelines increases (e.g., a sudden surge in inference requests to service), GKE can automatically scale to additional node-pools with MIG partitions — offering finer granularity to provision the right-sized GPU acceleration for workloads of all sizes.

“Our total averaged throughput varied between 10 and 80 images / sec. Thanks to support for NVIDIA A100 MIG and auto scaling mechanisms in GKE we can now scale that up to 150 images/sec and more, based on user-demand and GPU availability,” said Vlad Pranskevičius, Co-founder and CTO, Let’s Enhance. 

In addition, GKE’s support for dynamic scheduling, automated maintenance and upgrades, high availability, job API, customizability and fault tolerance simplified managing a production deployment environment, allowing the Let’s Enhance team to focus on building advanced ML pipelines.

High-performance inference serving with NVIDIA Triton Inference Server

To optimize performance and simplify the deployment of their DNN models onto the GKE-managed node pools of NVIDIA A100 MIG instances, Let’s Enhance chose the open-source NVIDIA Triton Inference Server, which deploys, runs and scales AI models from any framework onto any GPU- or CPU-based infrastructure. 

NVIDIA Triton’s support for multiple frameworks enabled the Let’s Enhance team to serve models trained in both TensorFlow and PyTorch, eliminating the need to set up and maintain multiple serving solutions for different framework backends. In addition, Triton’s ensemble model and shared memory features helped maximize performance and minimize data transfer overhead for their end-to-end image processing pipeline, which consists of multiple models and large amounts of raw image data transferring between them.

“We saw over 20% performance improvement with Triton vs. custom inference serving code, and were also able to fix several errors during deployment due to Triton’s self-healing features”, said Vlad. “Triton is packed with cool features, and as I was watching its progress from the very beginning, the pace of product development is just amazing.”

To further enhance end-to-end inference performance, the team is adopting NVIDIA TensorRT, an SDK to optimize trained models for deployment with the highest throughput and lowest latency while preserving the accuracy of predictions.

“With the subset of our models which we converted from Tensorflow to NVIDIA TensorRT we observed a speedup of anywhere between 10% and 42%, depending on the model, which is impressive,” said Vlad. “For memory-intensive models like ours, we were also able to achieve predictable memory consumption, which is another great benefit of using TensorRT, helping prevent any unexpected memory-related issues during production deployment.”

Teamwork makes the dream work

By using Google Cloud and NVIDIA, Let’s Enhance addressed the real-time inference serving and infrastructure management challenges of taking their AI-enabled service to production at scale in a secure Google Cloud infrastructure.  

GKE, NVIDIA AI software and A2 VMs powered by NVIDIA A100 GPUs brought together all the elements they needed to deliver the desired user experience and build a solution that can dynamically scale their end-to-end pipelines based on user demand.

The close collaboration with the Google Cloud and NVIDIA team, every step of the way, also helped Let’s Enhance get their  solution to market fast. “The Google Cloud and NVIDIA teams are incredible to work with. They are highly professional, always responsive and genuinely care about our success,” said Vlad. “We were able to get technical advice and recommendations directly from Google Cloud product and engineering teams, and also have a channel to share our feedback about Google Cloud products, which was really important to us.”

With a solid foundation and solution architecture that can meet their growing business needs, Let’s Enhance is marching towards their goal of bringing AI-enhanced digital images to everyone. “Our vision is to help businesses effectively manage user-generated content and increase conversion rates with next-gen AI tools,” said Sofi. ”Our next step towards that is to expand our offerings to completely replace manual work for photo preparation, including as well as cover pre-stages as image quality assessment and moderation.”

To learn more about the Let’s Enhance products and services, check out their blog here.

Editor’s note:Today’s post is by David Anderson, CIO for American Medical Facilities Management (AMFM), a group of 19 skilled nursing and rehabilitation centers in West Virginia. AMFM has adopted Chrome Enterprise Upgrade and ChromeOS devices to equip its frontline healthcare workers with seamless access to their caregiving apps and data.

AMFM’s skilled nursing centers scale seamlessly with ChromeOS

Whenever a new app came along at AMFM’s senior care centers, we used to spend months planning the technical rollout, assessing hardware choices, and hoping that caregivers could quickly ramp up on adoption. Today, we only ask two questions: “Can we make this work with ChromeOS devices?” and “Is it web-based?” If the answer to both questions is “yes,” and it usually is, the hardware purchases, employee training, and other stressful jobs disappear. 

How did we make the rollout of new caregiving and productivity tools so easy? ChromeOS devices, plus a passion for delivering seamless, consistent workflows for our frontline healthcare employees. We want their experiences online to be reliable and frustration-free. Our mission is to adopt company infrastructure that’s completely scalable, with as little friction as possible.

The challenges of sharing devices

The Windows desktop and laptop computers we had in place at AMFM were not meeting our mission. Scalability was important because we were growing quickly. We’ve been acquiring or starting up new care facilities at the rate of about one a year, and those employees needed to be equipped with devices and app access. But the Windows computers had high management overhead, and they were expensive to buy. 

Sharing Windows devices in an environment where caregivers would trade laptops and computers was very difficult. We had to enforce compliance rules about patient data privacy, while worrying what to do if laptops with access to the data were misplaced. The touch-screen monitors used by nurses and CNAs (certified nursing assistants) were bulky and hard to use.

A couple of years ago, when just about all of our software vendors had launched web versions—including PointClickCare, our EHR (electronic health record) solution—we realized our caregivers could do just about all their work on ChromeOS devices and Chrome browser. With about 300 devices—including Acer 315 Chromebooks and Acer Chromebooks—shared among 1,000 users, and replacing the Windows desktop and laptop computers, we’ve moved definitively away from the old days of unscalable, unchangeable, and high-maintenance hardware and systems.

How life has changed at AMFM

We have better compliance and security with managed guest sessions. Most of our ChromeOS devices run with managed guest sessions, allowing caregivers to log into and out of any Chromebook or Chromebase that’s in their work areas. This has been a game-changer and is perfect for our environment, where nurses and CNAs are constantly moving to different patients and buildings. 

Once a caregiver leaves a device behind for a few minutes, they’re logged out of ChromeOS automatically. The next user just has to log in to see the AMFM welcome Chrome browser screen, which links to all the daily tools like PointClickCare.

When devices enrolled in Chrome Enterprise Upgrade run in managed guest session mode, we’re able to isolate them on their own virtual network within our environment. We don’t have to worry about the devices commingling with other devices, and we can have a very specific security policy for them that’s designed for the multi-user environment. That’s huge from a compliance standpoint.

Since adopting ChromeOS, our hands-on support is minimal. This speaks volumes about our ChromeOS environment. Our IT team is the same size it was in 2012, when the company was only about half the size it is today. The time that it took to support the Windows desktop environment is now spent on expanding and managing other systems beyond what the caregivers use. Our trouble ticket volume has also stayed flat even as we’ve expanded.

The change in provisioning has also been huge. It used to take about a week minimum to replace a lost laptop, or repair broken hardware. Now, once we get new Chromebooks or Chromebases from our partner CDW, we enroll them in the Google Admin console using Chrome Enterprise Upgrade in a few minutes.

We halved our hardware cost when it comes to mobile devices like laptops. For stationary devices, we used to pay about $2,100 each for desktop computers and screens. Today, Chromebases cost about $700, just one-third the cost. In total, we saved about $109,000 on hardware purchases. 

We know the difference ChromeOS has made to our company, but we’re proud that McKnight’s Excellence in Technology Awards recognized it also, giving us the Bronze Award in its “Keep It Super Simple/Skilled Care” category. We met our mission to build out infrastructure that’s scalable, with little friction, and offering an easy and consistent experience for caregivers.

Many enterprises need data backups with long-term retention – for months or years – to help support requirements for compliance, audit, or disaster recovery. For example, a workload that generates receipt data might need to keep snapshots of this data for several years, even if the data might never be retrieved or retrieved only rarely. Persistent Disk snapshots have long provided reliable, indefinite data storage, but may not always be cost efficient for supporting these long-term scenarios.

Today we are introducing Persistent Disk archive snapshots, a low-cost option for protecting Persistent Disks. Archive snapshots offer lower storage prices than standard snapshots and have a 90 day billing period and charges for retrievals – making it a great option for long-term storage that is rarely accessed.

Archive snapshots are designed for simple compatibility with existing snapshot workflows. The choice of a standard snapshot or archive snapshot is set on creation, i.e. when calling the snapshots insert API, you simply pass the flag snapshotType and set it to ARCHIVE or STANDARD. Each snapshot type is stored in separate incremental snapshot chains, and archive snapshots are listed separately in the Google Cloud console. You can restore and use archive snapshots with the same snapshot and disk APIs you use today.

Archive snapshots are priced at $0.019 / GB-month for storage and $0.019 / GB for snapshot retrievals for regional snapshots, and $0.024 / GB-mo for storage and $0.024 / GB for snapshot retrievals for multi-regional snapshots.  (Prices are for the us-central1 region and the us multi-region – prices vary in other locations and networking charges also apply.)  Archive snapshots are billed for a minimum of 90 days of storage.

Persistent Disk Archive Snapshots Provide Flexibility and Efficiency

Archive snapshots provide the same benefits you expect from standard snapshots, including incremental storage chains, compression, encryption, durability, cross-project storage, and disk compatibility. Let’s highlight a few of these benefits.

First, Persistent Disk operates a global block storage system backed by the operational excellence required to deliver a highly resilient and durable storage infrastructure for business critical applications.  Taking snapshots of your disks – including archive snapshots – provides increased durability and reliability by replicating data to the location of your choice.

In large organizations running many workloads, a centralized backup administrator is often responsible for managing snapshots. Snapshots can be stored in a destination project that is different from the project housing the source disk – for example a dedicated backup project – with separate IAM permissions only granted to select personnel. This option is available for standard snapshots and archive snapshots. There is no need to create data copies to store snapshots in the location and with the permissions you require.

Archive snapshots are also incremental, meaning that once the first archive snapshot is created for a disk, subsequent archive snapshots only store the incremental data that has changed on that disk since the last archive snapshot. Additionally, built-in compression ensures that data is stored efficiently and with minimal footprint.

Archive snapshots are now generally available to all customers in all regions. For additional information, read more about updates to choices in Google Cloud infrastructure pricing and capabilities, and detailed pricing for Persistent Disk snapshots.

Cloud Bigtable is a popular and widely used key-value database available on Google Cloud. The service provides scale elasticity, cost efficiency, excellent performance characteristics, and 99.999% availability SLA. This has led to massive adoption with thousands of customers trusting Bigtable to run a variety of their mission-critical workloads.

Bigtable has been in continuous production usage at Google for more than 15 years now. It processes more than 5 billion requests per second at peak and has more than 10 exabytes of data under management. It’s one of the largest semi-structured data storage services at Google. 

One of the key use cases for Bigtable at Google is ad personalization. This post describes the central role that Bigtable plays within ad personalization.

Ad personalization

Ad personalization aims to improve user experience by presenting topical and relevant ad content. For example, I often watch bread-making videos on YouTube. If ads personalization is enabled in my ad settings, my viewing history could indicate to YouTube that I’m interested in baking as a topic and would potentially be interested in ad content related to baking products

Ad personalization requires large-scale data processing in near real-time for timely personalization with strict controls for user data handling and retention. System availability needs to be high, and serving latencies need to be low due to the narrow window within which decisions need to be made on what ad content to retrieve and serve. Sub-optimal serving decisions (e.g. falling back to generic ad content) could potentially impact user experience. Ad economics requires infrastructure costs to be kept as low as possible.

Google’s ad personalization platform provides frameworks to develop and deploy machine learning models for relevance and ranking of ad content. The platform supports both real-time and batch personalization. The platform is built using Bigtable, allowing Google products to access data sources for ads personalization in a secure manner that is both privacy and policy compliant, all while honoring users’ decisions about what data they want to provide to Google

The output from personalization pipelines, such as advertising profiles are stored back in Bigtable for further consumption. The ad serving stack retrieves these advertising profiles to drive the next set of ad serving decisions.

Some of the storage requirements of the personalization platform include:

Very high throughput access for batch and near real-time personalization

Low latency (<20 ms at p99) lookup for reads on the critical path for ad serving

Fast (i.e. in the order of seconds) incremental update of advertising models in order to reduce personalization delay

Bigtable 

Bigtable’s versatility in supporting both low-cost, high-throughput access to data for offline personalization as well as consistent low-latency access for online data serving makes it an excellent fit for the ads workloads. 

Personalization at Google-scale requires a very large storage footprint. Bigtable’s scalability, performance consistency and low cost required to meet a given performance curve are key differentiators for these workloads. 

Data model
The personalization platform stores objects in Bigtable as serialized protobufs keyed by Object ids. Typical data sizes are less than 1 MB and serving latency is less than 20 ms at p99. 

Data is organized as corpora, which correspond to distinct categories of data. A corpus maps to a replicated Bigtable.

Within a corpus, data is organized as DataTypes, logical groupings of data. Features, embeddings, and different flavors of advertising profiles are stored as DataTypes, which map to Bigtable column families. DataTypes are defined in schemas which describe the proto structure of the data and additional metadata indicating ownership and provenance. SubTypes map to Bigtable columns and are free-form. 

Each row of data is uniquely identified by a RowID, which is based on the Object ID. The personalization API identifies individual values by RowID (row key), DataType (column family), SubType (column part), and Timestamp.

Consistency
The default consistency mode for operations is eventual. In this mode, data from the Bigtable replica nearest to the user is retrieved, providing the lowest median and tail latency.

Reads and writes to a single Bigtable replica are consistent. If there are multiple replicas of Bigtable in a region, traffic spillover across regions is more likely. To improve the likelihood of read-after-write consistency, the personalization platform uses a notion of row affinity. If there are multiple replicas in a region, one replica is preferentially selected for any given row, based on a hash of the Row ID. 

For lookups with stricter consistency requirements, the platform first attempts to read from the nearest replica and requests that Bigtable return the current low watermark (LWM) for each replica. If the nearest replica happens to be the replica where the writes originated, or if the LWMs indicate that replication has caught up to the necessary timestamp, then the service returns a consistent response. If replication has not caught up, then the service issues a second lookup—this one targeted at the Bigtable replica where writes originated. That replica could be distant and the request could be slow. While waiting for a response, the platform may issue failover lookups to other replicas in case replication has caught up at those replicas.

Bigtable replication
The Ads personalization workloads use a Bigtable replication topology with more than 20 replicas, spread across four continents. 

Replication helps address the high availability needs for ad serving. Bigtable’s zonal monthly uptime percentage is in excess of 99.9%, and replication coupled with a multi-cluster routing policy allows for availability in excess of 99.999%.

A globe-spanning topology allows for data placement that is close to users, minimizing serving latencies. However, it also comes with challenges such as variability in network link costs and throughputs. Bigtable uses Minimum Spanning Tree-based routing algorithms and bandwidth-conserving proxy replicas to help reduce network costs. 

For ads personalization, reducing Bigtable replication delay is key to lowering the personalization delay (the time between a user’ action and when that action has been incorporated into advertising models to show more relevant ads to the user). Faster replication is preferred but we also need to balance serving traffic against replication traffic and make sure low-latency user-data serving is not disrupted due to incoming or outgoing replication traffic flows. Under the hood, Bigtable implements complex flow control and priority boost mechanisms to manage global traffic flows and to balance serving and replication traffic priorities. 

Workload Isolation
Ad personalization batch workloads are isolated from serving workloads by pinning a given set of workloads onto certain replicas; some Bigtable replicas exclusively drive personalization pipelines while others drive user-data serving. This model allows for a continuous and near real-time feedback loop between serving systems and offline personalization pipelines, while protecting the two workloads from contending with each other.

For Cloud Bigtable users, AppProfiles and cluster-routing policies provide a way to confine and pin workloads to specific replicas to achieve coarse-grained isolation. 

Data residency
By default, data is replicated to every replica—often spread out globally—which is wasteful for data that is only accessed regionally. Regionalization saves on storage and replication costs by confining data to the region where it is most likely to be accessed. Compliance with regulations mandating that data pertaining to certain subjects are physically stored within a given geographical area is also vital.

The location of data can be either implicitly determined by the access location of requests or through location metadata and other product signals. Once the location for a user is determined, it is stored in a location metadata table which points to the Bigtable replicas that read requests should be routed to. Migration of data based on row-placement policies happens in the background, without downtime or serving performance regressions.

Conclusion

In this blog post, we looked at how Bigtable is used within Google to support an important use case—modeling user intent for ad personalization. 

Over the past decade, Bigtable has scaled as Google’s personalization needs have scaled by orders of magnitude. For large-scale personalization workloads, Bigtable offers low cost storage with excellent performance characteristics. It seamlessly handles global traffic flows with simple user configurations. Its ease at handling both low-latency serving and high-throughput batch computations make it an excellent option for lambda-style data processing pipelines.

We continue to drive high levels of investment to further lower costs, improve performance, and bring new features to make Bigtable an even better choice for personalization workloads.

Learn more

To get started with Bigtable, try it out with a Qwiklab and learn more about the product here.

Acknowledgements
We’d like to thank Ashish Awasthi, Ashish Chopra, Jay Wylie, Phaneendhar Vemuru, Bora Beran, Elijah Lawal, Sean Rhee and other Googlers for their valuable feedback and suggestions.