Cloud

Customers want to leverage Google Cloud Storage for its simplicity, scalability, and security. But migrating 100s of TB of data from a self-managed object storage is complicated. Writing and maintaining scripts that use resources effectively during transfer without compromising on security can take months. 

To accelerate and simplify this migration, Storage Transfer Service recently announced Preview support for moving data from S3-compatible storage to Cloud Storage. This feature builds on Cloud Storage recent launches, namely support forMultipart upload andList Object V2, which makes Cloud Storage suitable for running applications written for the S3 API. 

With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage for migration, archiving cold data, replicating data for business continuity, or creating data pipelines. For customers moving data from AWS S3 to Cloud Storage, it gives an option to control network routes to Google Cloud, resulting in considerably lower egress charges.

Storage Transfer Service is a managed service that enables customers to quickly and securely transfer data to, from, and between object and file storage systems, including Google Cloud Storage, Amazon S3, Azure Storage, on-premises data, and more. It offers scheduling, encryption, data integrity checks, and scale-out performance out of the box.

Cloud Storage advantage

Cloud Storage is a planet-scale object storage designed for at least 11 9’s annual durability and offers multiple levers including Archive storage class and Lifecycle management to manage and deliver storage at ultra-low cost. It provides a single namespace that is strongly consistent and can span across the continent.  

With your data in Cloud Storage, you can take advantage of Google Cloud’s cutting edge capabilities in content serving with Cloud CDN, computation with Compute Engine and Google Kubernetes Engine, analytics with BigQuery, Dataproc etc. 

How does transfer from S3 Compatible storage work?

Storage Transfer Service consists of two components –  control plane and agents. 

Control plane coordinates the transfer of data including distributing work to agents, scheduling, maintaining state, scaling resources etc. Agents are a small piece of software, self hosted on a VM close to the data source. These agents run in a Docker container and belong to an “agent pool”. 

To orchestrate transfer, you create a transfer job, which contains all the information necessary to move data including source, destination, schedule, filters to include and exclude objects, option to control the lifecycle of source and destination objects.  

To use this feature, your object storage must be compatible with the following Amazon S3 API operations: GetObject, ListObjectV2 or ListObjectV1, HeadObject, and DeleteObject. It must also support AWS Signature Version 4 or Version 2 for authenticating requests. 

Here is a quick CLI tutorial to help you get started. 

How to copy data from S3 Compatible storage to Cloud Storage? 

To transfer data from a self-managed object storage to Cloud Storage, you do the following:

Step 1: Configure access to the source

Step 2: Deploy Storage Transfer Service agents

Step 3: Create Transfer Job

Step 1: Configure access to the source:

You need to gather configuration details and credentials for STS to access the source data. 

To configure agents to access the source, generate and note down the access and secret keys for the source. Agents need following permissions:

List the bucket.

Read the objects in the source bucket

Delete the objects in the source bucket

In addition, note down the following information for your source:

Bucket Name: test-3

Endpoint: s3.source.com 

Region: us-west-1

Step 2: Deploy Storage Transfer Service agents:

You need to deploy Storage Transfer Service agents near your storage system with appropriate permission to access source and Google Cloud resources.

To assign permissions for agent to access Google Cloud resources:

Create service account for the agent by navigating to the Cloud Console

Assign following roles to the agent service account:

Storage Transfer Agent (role/storagetransfer.transferAgent)

Storage Object Admin (roles/storage.objectAdmin)

Pub/Sub Editor (roles/pubsub.editor)

Generate the credentials file: 

Replace the following values:

PROJECT_ID: The project id.

SERVICE_ACCOUNT_ID: The service account id.

service_account.json: The name of file to store credentials

To deploy agents on the host machine, first create an agent pool and then install agents.

Step 3: Create Transfer Job

STS uses transferJob to coordinate the movement data from a source to destination. Steps create a transferJob

Assign permissions to STS service account: STS uses a Google Managed service account to manage transfer. The service account’s format is project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com. 

For a new project, provision service accounts by making googleServiceAccounts.get API call. Assign the following roles or equivalent permissions to this service account:

Storage Object Creator (roles/storage.objectCreator)

Storage Object Viewer (roles/storage.objectViewer)

Pub/Sub Editor (roles/pubsub.editor)

Storage Legacy Bucket Reader (roles/storage.legacyBucketReader)

You need to roles/storagetransfer.admin to create a transfer job. You can assign permission to this service account by navigating to “IAM and Admin” in the side nav bar and then to “IAM”.

You can monitor the created transfer job either via CLI or cloud console. 

Best practices

For large migrations, bandwidth is often the bottleneck. Use Cloud Interconnect for a more consistent throughput for large data transfers. To avoid impacting production workload, you can limit the amount of bandwidth consumed by the Storage Transfer Service through an agent pool parameter.

We recommend deploying agents close to the source to minimize network latency between the agent and your storage system. Run at least 3 agents for fault tolerance and allocate at least 4 vCPU and 8 RAM per agent.

When transferring large numbers of small objects, listing objects at the source can be a bottleneck. In this scenario, create multiple transfer jobs, each dedicated to a particular set of prefixes, to scale transfer performance. To run transfer on a particular set of prefixes, useinclude/exclude prefixes in the transfer job config. 

To replicate data for business continuity, you can use a combination of deleteObjectsUniqueInSink and overwriteWhen. With these settings you always always overwrite the destination object with the source object and delete files deleted at the source, ensuring that Cloud Storage bucket destination is an exact copy of your source. 

For guidance on migrating users sending requests to self-managed object storage using an API, refer to fully migrate from Amazon S3 to Cloud Storage.

In this blog, we’ve demonstrated how you can use Storage Transfer Service to quickly and securely transfer data from a self-managed object storage to Cloud Storage.  

For more details on data transfer, refer to the Storage Transfer Service documentation.

“Cloud Wisdom Weekly: for tech companies and startups” is a new blog series we’re running this fall to answer common questions our tech and startup customers ask us about how to build apps faster, smarter, and cheaper. In this installment, we explore how to leverage artificial intelligence (AI) and machine learning (ML) for faster innovation and efficient operational growth. 

Whether they’re trying to extract insights from data, create faster and more efficient workflows via intelligent automation, or build innovative customer experiences, leaders at today’s tech companies and startups know that proficiency in AI and ML is more important than ever.

AI and ML technologies are often expensive and time-consuming to develop, and the demand for AI and ML experts still largely outpaces the existing talent pool. These factors put pressure on tech companies and startups to allocate resources carefully when considering bringing AI/ML into their business strategy. In this article, we’ll explore four tips to help tech companies and startups accelerate innovation and reduce costs with AI and ML. 

4 tips to accelerate innovation and reduces costs with AI and ML

Many of today’s most innovative companies are creating services or products that couldn’t exist without AI—but that doesn’t mean they’re building their AI and ML infrastructure and pipelines from scratch. Even for startups whose businesses don’t directly revolve around AI, injecting AI into operational processes can help manage costs as the company grows. By relying on a cloud provider for AI services, organizations can unlock opportunities to energize development, automate processes, and reduce costs. 

1. Leverage pre-trained ML APIs to jumpstart product development 

Tech companies and startups want their technical talent focused on proprietary projects that will make a difference to the business. This often involves the development of new applications for an AI technology, but not necessarily the development of the AI technology itself. In such scenarios, pre-trained APIs help organizations quickly and cost-effectively establish a foundation on which higher-value, more differentiated work can be layered. 

For example, many companies building conversational AI into their products and services leverage Google Cloud APIs such as Speech-to-Text and Natural Language. With these APIs, developers can easily integrate capabilities like transcription, sentiment analysis, content classification, profanity filtering, speaker diarization, and more. These powerful technologies help organizations focus on creating products rather than having to build the base technologies. 

See this article for examples of why tech companies and startups have chosen Google Cloud’s Speech APIs for use cases that range from deriving customer insights to giving robots empathetic personalities. For an even deeper dive, see

our AI product page to explore other APIs, including Translation, Vision, and more;

and the Google Cloud Skills Boost for ML APIs.

2. Use managed services to scale ML development and accelerate deployment of models to production

Pre-trained models are extremely useful, but in many cases, tech companies and startups need to create custom models to either derive insights from their own data or to apply new use cases to public data. Regardless of whether they’re building data-driven products or generating forecasting models from customer data, companies need ways to accelerate the building and deployment of models into their production environments.  

A data scientist typically starts a new ML project in a notebook, experimenting with data stored on the local machine. Moving these efforts into a production environment requires additional tooling and resources, including more complicated infrastructure management. This is one reason many organizations struggle to bring models into production and burn through time and resources without moving the revenue needle. 

Managed cloud platforms can help organizations transition from projects to automated experimentation at scale or the routine deployment and retraining of production models. Strong platforms offer flexible frameworks, fewer lines of code required for model training, unified environments across tools and datasets, and user-friendly infrastructure management and deployment pipelines. 

At Google Cloud, we’ve seen customers with these needs embrace Vertex AI, our platform for accelerating ML development, in increasing numbers since it launched last year. Accelerating time to production by up to 80% compared to competing approaches, Vertex AI provides advanced end-to-end ML Ops capabilities so that data scientists, ML engineers, and developers can contribute to ML acceleration. It includes low-code features, like AutoML, that make it possible to train high performing models without ML expertise. 

Over the first half of 2022, our performance tests found that the number of customers utilizing AI Workbench increased by 25x. It’s exciting to see the impact and value customers are gaining with Vertex AI Workbench, including seeing it help companies speed up large model training jobs by 10x and helping data science teams improve modeling precision from the 70-80% range to 98%.

If you are new to Vertex AI, check out this video series to learn how to take models from prototype to production. For deeper dives, see

this article about Vertex AI’s role in an ambitious project to measure climate change with AI; 

BigQuery has built-in Machine Learning (ML) and Analytics that you can use to create no-code predictions using just SQL queries.

this blog about how Vertex AI and BigQuery work together to make data analysis easier and more powerful; 

and this blog about Example-based explanations, one of our most recent updates to make model iteration more intuitive and efficient. 

3. Harness the cloud to match hardware to use cases while minimizing costs and management overhead 

ML infrastructure is generally expensive to build, and depending on the use case, specific hardware requirements and software integrations can make projects costly and complicated at scale. To solve for this, many tech companies and startups look to cloud services for compute and storage needs, attracted by the ability to pay only for resources they use while scaling up and down according to changing business needs. 

At Google Cloud, customers share that they need the ability to optimize around a variety of infrastructure approaches for diverse ML workloads. Some use Central Processing Units (CPUs) for flexible prototyping. Others leverage our support for NVIDIA Graphics Processing Units (GPUs) for image-oriented projects and larger models, especially those with custom TensorFlow operations that must run partially on CPUs. Some choose to run on the same custom ML processors that power Google applications—Tensor Processing Units (TPUs). And many use different combinations of all of the preceding. 

Beyond matching use cases to the right hardware and benefiting from the scale and operational simplicity of a managed service, tech companies and startups should explore configuration features that help further control costs. For example, Google Cloud features like time-sharing and multi-instance capabilities for GPUs — as well as features like Vertex AI Training Reduction Server — are built to optimize GPU costs and usage. 

Vertex AI Workbench also integrates with the NVIDIA NGC catalog for deploying frameworks, software development kits and Jupyter Notebooks with a single click—another feature that, like Reduction Server, speaks to the ways organizations can make AI more efficient and less costly via managed services.

4. Implement AI for operations

Besides using pre-trained APIs and ML model development to develop and deliver products, startup and tech companies can improve operational efficiency, especially as they scale, by leveraging AI solutions built for specific business and operational needs, like contract processing or customer service. 

Google Cloud’s DocumentAI products, for instance, apply ML to text for use cases ranging from contract lifecycle management to mortgage processing. For businesses whose customer support needs are growing, there’s Contact Center AI, which helps organizations build intelligent virtual agents, facilitate handoffs as appropriate between virtual agents and human agents, and generate insights from call center interactions. By leveraging AI to help manage operational processes, startups and tech companies can allocate more resources to innovation and growth. 

Next steps toward an intelligent future

The tips in this article can help any tech company or startup find ways to save money and boost efficiency with AI and ML. You can learn more about these topics by registering for Google Cloud Next, kicking off October 11, where you’ll hear Google Cloud’s latest AI news, discussions, and perspectives—in the meantime, you can also dive into our Vertex AI quickstarts and BigQuery ML tutorials. And for the latest on our work with tech companies and startups, be sure to visit our Startups page.

Rising shipping costs are preventing retailers from delivering basic items people need at an affordable price. Although the retail industry has spent decades trying to bring new efficiencies to shipping, higher fuel costs and supply chain disruptions caused by the COVID-19 pandemic are worsening the challenge.  

AtPaccurate, we believe smarter packing is key to creating a more efficient, sustainable, and greener supply chain. That’s why we built a comprehensive shipping and fulfillment optimization platform. With Paccurate, retailers analyze their shipping history to find ideal box sizes, control how they’re packed so they can minimize costs and SCOPE-3 emissions. Retailers also leverage Paccurate to avoid costly supply chain issues by anticipating how many of each carton they will need in the future.

Our customers are seeing impressive results as they cut air in shipments by 22%, lower shipping costs 16%, and decrease the use of packaging materials by over 13%. As Paccurate continues to grow, we’ll introduce new solutions and services to further optimize the packing process, lower distribution center operating costs, and minimizeSCOPE-3 shipping emissions well beyond our current track record of 14%.

Analyzing and optimizing packing instructions in less than two milliseconds

Although Paccurate was initially built on a legacy technology stack, we quickly realized we needed a scalable cloud-based solution to reliably support tens of millions of high-speed, low latency queries. We also knew it would be difficult for our small team to develop new capabilities and features without reducing IT costs and shifting more resources to R&D.

We looked at the options and identifiedGoogle Cloud, including theGoogle for Startups Cloud Program, as the best choice to help us cost-effectively scale Paccurate to meet increased customer demand. Thesecure-by-design infrastructure of Google Cloud now enables retailers to efficiently pack and deliver the items people need at more affordable prices.

Specifically, we rely onCloud Run to rapidly analyze shipments and return optimized packing instructions in less than two milliseconds. We also leverageCloud DNS for end-to-end domain management,App Engine to develop and host web applications,Cloud Functions to run code across multiple environments, andCloud SQL to automate database provisioning.

With the help of Paccurate and Google Cloud solutions, our customers save an average of one square foot of cardboard per carton while lowering overall fulfillment costs by over 20%.

Building a comprehensive shipping and fulfillment optimization platform on Google Cloud

The Google for Startups Cloud Program has been instrumental in helping us expand and scale our shipping and fulfillment optimization platform. We especially want to highlight the Google Cloud credits we relied on to cost-effectively trial and deploy the solutions Paccurate now uses to analyze and optimize millions of shipments every month.  

In the future, we plan on leveraging Google CloudAI and machine learning products to further refine on-demand analyses that simulate how variables such as SKU dimensions, warehouse location, and packing material vendors affect packing and shipping costs. We also want to help retailers program smarter warehouse robots and machines to more efficiently create the right boxes based on carrier, zone, and service type. 

We can’t wait to see what we accomplish next as we grow our team and introduce new smart packing services and solutions that reduce cubic volume, wasted material, and shipping costs to benefit retailers and customers alike. 

If you want to learn more about how Google Cloud can help your startup, visit our pagehere to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Pub/Sub offers a rich set of metrics for resource and usage monitoring. Previously, these metrics were like buried treasure: they were useful to understand Pub/Sub usage, but you had to dig around to find them. Today, we are announcing out-of-the-box Pub/Sub metrics dashboards that are accessible with one click from the Topics and Subscriptions pages in the Google Cloud Console. These dashboards provide more observability in context and help you build better solutions with Pub/Sub.

Check out our new one-click monitoring dashboards

We added metrics dashboards to monitor the health of all your topics and subscriptions in one place, including dashboards for individual topics and subscriptions. Follow these steps to access the new monitoring dashboards:

To view the monitoring dashboard for all the topics in your project, open the Pub/Sub Topics page and click the Metrics tab. This dashboard has two sections: Overview and Quota.

To view the monitoring dashboard for a single topic, in the Pub/Sub Topics page, click any topic to display the topic detail page, and then click the Metrics tab. This dashboard has up to three sections: Overview, Subscriptions, and Retention (if topic retention is enabled).

To view the monitoring dashboard for all the subscriptions in your project, open the Pub/Sub Subscriptions page and click the Metrics tab. This dashboard has two sections: Overview and Quotas.

To view the monitoring dashboard for a single subscription, in the Pub/Sub Subscriptions page, click any subscription to display the subscription detail page, and then click the Metrics tab. This dashboard has up to four sections: Overview, Health, Retention (if acknowledged message retention is enabled), and either Pull or Push depending on the delivery type of your subscription.

A few highlights

When exploring the new Pub/Sub metrics dashboard, here are a few examples of things you can do. Please note that these dashboards are a work in progress, and we hope to update them based on your feedback. To learn about recent changes, please refer to the Pub/Sub monitoring documentation.

As you can see, the metrics available in the monitoring dashboard for a single topic are closely related to one another. Roughly speaking, you can obtain Publish throughput in bytes by multiplying Published message count and Average message size. Because a publish request is made up of a batch of messages, dividing Published messages by Publish request count gets you Average number of messages per batch. Expect a higher number of published messages than publish requests if your publisher client has batching enabled. 

Some interesting questions you can answer by looking at the monitoring dashboard for a single topic are: 

Did my message sizes change over time?

Is there a spike in publish requests?

Is my publish throughput in line with my expectations?

Is my batch size appropriate for the latency I want to achieve, given that larger batch sizes increase publish latency?

You can find a few powerful composite metrics in the monitoring dashboard for a single subscription. These metrics are Delivery metrics, Publish to ack delta, and Pull to ack delta. All three aim to give you a sense of how well your subscribers are keeping up with incoming messages. Delivery metrics display your publish, pull, and acknowledge (ack) rate next to each other. Pull to ack delta and Publish to ack delta offer you the opportunity to drill down to any specific bottlenecks. For example, if your subscribers are pulling messages a lot faster than they are acknowledging them, expect the values reported in Pull to ack delta to be mostly above zero. In this scenario, also expect both your Unacked messages by region and your Backlog bytes to grow. To remedy this situation, you can increase your message processing power or setting up subscriber flow control.

Another powerful composite metric available in the monitoring dashboard for a single subscription is the Delivery latency health score in the Health section. You may treat this metric as a one-stop shop to examine the health of your subscription. This metric tracks a total of five properties; each can take a value of zero or one. If your subscribers are not keeping up, zero scores for “ack_latency” and/or “expired_ack_deadlines” effectively tell you that those properties are the reason why. We prescribe how to fix these failing scores in our documentation. If your subscription is run by a managed service like Dataflow, do not be alarmed by a “utilization” score of zero. With Dataflow, the number of streams open to receive messages is optimized, so the recommendation to open more streams does not apply. 

Some questions you can answer by looking at your monitoring dashboard for a single subscription are: 

What is the 99th percentile of my ack latencies? Is the majority of my messages getting acknowledged in under a second, allowing my application to run in near real-time? 

How well are my subscribers keeping up with my publishers? 

Which region has a growing backlog? 

How frequently are my subscribers allowing a message’s ack deadline to expire?

Customize your monitoring experience

Hopefully the existing dashboards are enough to diagnose a problem. But maybe you need something slightly different. If that’s the case, from the dropdown menu, click Save as Custom Dashboard to save an entire dashboard in your list of monitoring dashboards, or click Add to Custom Dashboard in a specific chart to save the chart to a custom dashboard. Then, in the custom dashboard, you can edit any chart configuration or MQL query. 

For example, by default, the Top 5 subscriptions by ack message count chart in the Subscriptions section of the monitoring dashboard for a single topic shows the top five attached subscriptions with the highest rate of acked messages. You can modify the dashboard to show the top ten subscriptions. To make the change, export the chart, click on the chart, and edit the line of MQL “| top 5, .max()” to “| top 10, .max()”. To know more about editing in MQL, see Using the Query Editor | Cloud Monitoring.

For a slightly more complex example, you can build a chart that compares current data to past data. For example, consider the Byte Cost chart in the Overview section of the monitoring dashboard for all topics. You can view the chart in Metrics Explorer. In the MQL tab, add the following lines at the end of the provided code snippet:

The preceding lines turn the original chart to a comparison chart that compares data at the same time on the previous day. For example, if your Pub/Sub topic consists of application events like requests for cab rides, data from the previous day can be a nice baseline for current data and can help you set the right expectations for your business or application for the current day. If you’d prefer, update the chart type to a Line chart for easier comparison. 

Set alerts

Quota limits can creep up on you when you least expect it. To prevent this, you can set up alerts that will notify you once you hit certain thresholds. The Pub/Sub dashboards have a built-in function to help you set up these alerts. First, access the Quota section in one of the monitoring dashboards for topics or subscriptions. Then, click Create Alert inside the Set Quota Alert card at the top of the dashboard. This will take you to the alert creation form with an MQL query that triggers for any quota metric exceeding 80% capacity (the threshold can be modified).

In fact, all the provided charts support setting alerting policies. You can set up your alerting policies by first exporting a chart to a custom dashboard and then selecting Convert a provided chart to an alert chart. using the dropdown menu.

For example, you might want to trigger an alert if the pull to ack delta is positive more than 90% of the time during a 12-hour period. This would indicate that your subscription is frequently pulling messages faster than it is acknowledging them. First, export the Pull to Ack Delta chart to a custom dashboard, convert it to an alert chart, and add the following line of code at the end of the provided MQL query:

| condition gt(val(), 0)

Then, click Configure trigger. Set the Alert trigger to Percent of time series violates, the Minimum percent of time series in violation to 90%, and Trigger when condition is met for this amount of time to 12 hours. If the alert is created successfully, you should see a new chart with a red horizontal line representing the threshold with a text bubble that tells you if there have been any open incidents violating the condition. 

You can also add an alert for the Oldest unacked message metric. Pub/Sub lets you set a message retention period on your subscriptions. Aim to keep your oldest unacked messages within the configured subscription retention period, and fire an alert when messages are taking longer than expected to be processed. 

Making metrics dashboards that are easy to use and serve your needs is important for us. We welcome your feedback and suggestions for any of the provided dashboards and charts. You can reach us by clicking on the question icon on the top right corner in Cloud Console and choosing Send feedback. If you really like a chart, please let us know too! We will be delighted to hear from you.

Encryption and data protection is a major requirement for customers moving their workloads to the cloud. To meet this requirement, organizations often invest a great deal of time in protecting sensitive data in cloud-based databases. This is driven mostly by government regulations, compliance, and organizations’ security requirements to have data protected at rest. As Customer Engineers on the Security and Compliance technology team in Google Cloud, we engage both executive and technical stakeholders to help customers build secure deployments that enable their digital transformation on our Cloud platform.

As Google Cloud continues its efforts to be the industry’s most trusted cloud, we’re taking steps to help customers better understand encryption options available to protect workloads on our platform. In this post, we provide a guide on how to accelerate your design considerations and decision making when securely migrating or building databases with the various encryption options supported on Google Cloud platform. 

Managing data at rest with encryption on Google Cloud

When you move data to Google Cloud, you have options to choose from databases that are simple to use and operate without cumbersome maintenance tasks and operational overhead. Google Cloud keeps the databases highly available and updated, while your IT team can focus on delivering innovations and your end users enjoy reliable services. 

Additionally, you inherit security controls like encryption of data at-rest by default that can help simplify your security implementations. For most organizations, encryption is one piece of a broader security strategy. Encryption adds a layer of defense in depth for protecting data and provides an important mechanism for how Google helps ensure the privacy of data. Encryption ensures that if the data accidentally falls into an attacker’s hands, they cannot access the data without also having access to the encryption keys. Our platform offers data-at-rest encryption by default, ensuring that all data stored within the cloud is encrypted by Google-managed keys.

Management options for encryption keys

Google Managed Keys: All data stored within Google Cloud is encrypted at rest using the same hardened key management systems that we use for our own encrypted data. These key-management systems provide strict access controls and auditing, and encrypt user data at rest using AES-256 encryption standards. No setup, configuration, or management is required. Google managed keys is an appropriate choice if you don’t have specific requirements related to compliance or locality of cryptographic materials. 

Customer Managed Keys: Customer managed encryption keys (CMEK) offer the ability to protect your databases with encryption keys you control and manage. Using CMEK gives you control over more aspects of the lifecycle and management of your keys, such as key rotation, defining access control policies, auditing and logging, and enforcing data locality or residency requirements. CMEKs are supported on Cloud Key Management Service, Cloud Hardware Security Module, and Cloud External Key Manager.

Encryption options for Google Cloud databases 

In addition to default security controls inherited on Google Cloud, we believe customers should have options to choose the level of protection over data stored in the cloud. We’ve developed database products integrated with our encryption capabilities that enable you to control your data and provide expanded granularity into when and how data is accessed.

Google’s default encryption: Customers’ content stored on our platform is encrypted at rest, without any action from customers using multiple encryption mechanisms. Data for storage is split into chunks, and each chunk is encrypted with a unique data encryption key. The data encrypted keys are protected with key encryption keys (KEK) and stored centrally in Google’s KMS, a repository built specifically for storing keys.

Cloud Key Management Service (Cloud KMS) provides you with the capability to manage cryptographic keys in a central cloud service for either direct use or use by other cloud resources such as databases and datastores. Cloud KMS combines secure, highly available infrastructure with capabilities not only to provide the mechanisms to create keys of various types and strengths, but also an option for the keys to remain exclusively within the Google Cloud region with which the data is associated.

Cloud Hardware Security Module (Cloud HSM) enables you to generate encryption keys and perform cryptographic operations in FIPS 140-2 Level 3 certified HSMs. The service is fully managed, so you can protect your most sensitive workloads without worrying about the operational overhead of managing an HSM cluster. Google manages the HSM hardware, automatically scales based on your use,  spares you the complexity of managing and using HSM-backed keys in production. For example, you can encrypt data in Cloud SQL tables using a Cloud HSM key that you manage and control the life cycle of.

Cloud External Key Manager (EKM) gives you ultimate control over the keys and encrypted data-at-rest within Google Cloud resources such as CloudSQL, Cloud Spanner, etc. Google EKM enables you to use keys managed in a supported key management system external to Google to protect data within Google Cloud. It’s important to note that for this option, externally managed keys are never cached or stored within Google Cloud. Whenever Google Cloud needs to decrypt data, it communicates directly with the external key manager. In addition to Cloud EKM, customers may leverageKey Access Justifications to understand why their externally-hosted keys are being requested to decrypt data.

Here’s a look at the encryption options for database services that Google Cloud offers

Database Platform

Google Cloud Database service

Encryption Options Supported

Microsoft SQL Server

Cloud SQL for SQL Server

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

MySQL

Cloud SQL for MySQL

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

PostgreSQL

Cloud SQL for PostgreSQL

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

MongoDB

MongoDB Atlas

Google default encryption

Cloud KMS

CloudHSM

Apache HBase

Cloud BigTable

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

PostgreSQL

CloudSpanner for PostgreSQL

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

Google Standard SQL

CloudSpanner Google Standard SQL

Google default encryption

Cloud KMS

CloudHSM

CloudEKM

Redis Memory Store

Memorystore for Redis.

Google default encryption

Cloud KMS

CloudHSM

Firestore

Firestore

Google default encryption

Oracle Database

Bare Metal Solution for Oracle

Customer owned key management system

For more information on Key Management on GCP read our KMS Deep Dive Whitepaper.

Even before 2020, the education market was rapidly leaning into digitalization. That trend exploded when the pandemic forced education online, requiring every part of the education process to be digitized. Interaction between students and teachers shifted from  physical classrooms to entirely digital ones.

Institutions had to work quickly to support this shift. Most either expanded their e-learning platform’s computational capacity to support the influx of users or quickly created new platforms to adhere to “new” digital demand. These e-learning platforms, also known as Learning Management Systems (or LMS’s), already played an important role in educational institutions. However, the pandemic changed their role from “beneficial” to “essential”. While the moves institutions took to support their e-learning platforms helped bridge the gap in a moment of crisis, they were never intended to be a permanent solution.

The digitization of education isn’t slowing down. The LMS market is projected to grow from $16.19 billion in 2022 to $40.95 billion by 2029, at a CAGR of 14.2%, according to Fortune Business Insights. To support this continued growth and create a more stable foundation for e-learning platforms, we need to take a closer look at the environments hosting these platforms. By revising and optimizing these environments, we can offer a solid, supportive infrastructure that improves the experience for students and teachers alike.

Modernizing Moodle with Google Cloud 

Introduced in 2002, Moodle is one of the most used e-learning platforms in the world. Moodle is a trusted tool for institutions around the world, but the platform wasn’t designed for the public cloud. This has made scaling difficult. To ensure users can rely on Moodle to continue delivering connected experiences, Google Cloud’s Customer Engineers have developed an open-source, cloud-native approach for hosting Moodle.  This brings together the built-in, reliable, and highly scalable services of Google Cloud  and Moodle to create a more modern, robust, easy to manage, and cost-effective implementation.

From a technical perspective, this solution incorporates an official Moodle instance into a custom Docker image. This gets deployed into an enterprise version of Google Kubernetes Engine (GKE), along with underlying services to support Moodle’s operational and specialized tasks. See Figure 1 below for the complete architecture.

Incoming traffic first hits GCP’s Load Balancer. Among other functions, it plays the role of ingress controller for GKE, where Moodle’s pods run.

Load Balancer gets connected with two separate services: Cloud CDN (to deliver static content from pops closer to the requestor, reducing latency) and Cloud Armor, which is a Web Application Firewall (WAF) layer 7, and validates requests against OWASP top ten risks.

If the request is valid from a WAF perspective, it then hits reCAPTCHA enterprise, which will go over an additional access validation. This step is optional, but highly recommended.

Moodle’s web application pods run on top of highly scalable GKE. Moodle’s data files (aka moodledata) sit on a private version of Filestore service in Google Cloud and get accessed dynamically by working those pods in a scalable way. The storage is also shared among all pods.

It also utilizes Memorystore for Redis in Google Cloud for persisting users’ data in memory. This lets Moodle become stateless, allowing it to scale up and down easily.

Last but not least, Moodle’s pods communicate with the instance’s database that lives in a high-scalable version of MySQL (part of Cloud SQL service).

Benefits of modernizing Moodle with Google Cloud

Quick go-to-market. Everything you need to deploy this solution has been scripted and made available in this GitHub repository. That means institutions can simply follow the steps described in the repository’s documentation to get things running quickly. Google Cloud customized every step of the deployment to give you optimal resources and cost savings with the initial setup.

Cloud-native. This implementation of Moodle is 100% cloud-native. That means that all the services utilized by the solution are built-in Google Cloud services—meaning no more Virtual Machine management—and you get all the default benefits enabled within Google Cloud.

Highly scalable. The solution makes Moodle stateless, allowing it to scale up and down with no damage to users’ in-memory data. With GKE, you can also horizontally scale Moodle’s pods and cluster nodes, allowing web applications to always be on. All underlying services can follow GKE’s auto-scalability.

Open-source. The implementation of this solution is publicly available on GitHub. Institutions can collaborate with our engineers, or take it as a starting point for building something tailor-made.

Continuous Integration (CI) and Continuous Deployment (CD) ready. The solution was designed to work smoothly with DevOps automation practices. For that, Google Cloud recommends leveraging Google Cloud’s Artifact Registry and Cloud Build services.

Multi-sized environment options. We’re starting the project with an enterprise version for Moodle using GKE as the base for running Moodle’s instances. This is most suitable for bigger environments (5,000+ users). We will soon bring Moodle to a serverless model as well, allowing smaller environments and institutions to run at a lower cost.

Security embedded. On top of the web security layer provided by Cloud Armor and reCAPTCHA, modern Moodle deploys in private mode, meaning that all the underlying services supporting Moodle instance get enclosed to its  network boundaries. Only internal resources (from the same Virtual Private Network) can access the services.

Bringing a modern Moodle to your institution

It was important to us to keep modern Moodle on Google Cloud an open source solution. Every institution has its own needs based on individual priorities, goals, course needs, and communications preferences, and we wanted to bring together the speed of ready-to-use deployment with the customization needed to make modern Moodle a perfect fit for differing needs. Keeping it open source allows institutions to build their own environments. Simply clone the repository’s content, customize it, and deploy it directly into your Google Cloud accounts.

In addition, Google Cloud-certified partners can help institutions make sure their support needs are covered, including service level agreements, regulatory measurements for compliance and additional layers of security. 

Visit GitHub to learn more about modern Moodle with GCP and how it can support your e-learning platform use today.

When Google set out to achieve 24/7 carbon-free energy (CFE) in our operations by 2030, we committed to share insights and lessons that could help others move in the same direction. In 2021 we published a detailed methodology for 24/7 CFE, and earlier this year we released a roadmap for policymakers to enable 24/7 CFE for all and accelerate grid-level decarbonization. Today, we’re releasing a white paper that details our experience with a new energy transaction model that we call a Carbon-free Energy (CFE) Manager. We hope the insights in this white paper will support organizations’ move to source round-the-clock clean power. 

Google was a pioneer of corporate Power Purchase Agreements (PPAs), which have since become the most prominent approach for corporations to procure clean energy. But PPAs are complex instruments that come with several forms of risk that can be hard for buyers to manage. PPAs are also fundamentally a work-around; they are a way to access clean energy where it is not otherwise directly available from an electricity supplier. 

As we considered how to move toward 24/7 carbon-free energy, we reached out to our energy service providers to discuss solutions. We learned that there weren’t any round-the-clock carbon-free energy products being offered, so we worked with our energy providers to create them. 

A CFE Manager is an energy service provider that an energy buyer tasks with assembling a portfolio of CFE projects. Compared to an energy user like Google signing its own PPAs, the CFE Manager model reduces buyers’ transaction costs, shifting PPA risks to the entity best suited to manage them (the energy service provider), and can lead to greater decarbonization of hourly electricity consumption — all at a competitive price. The model is also scalable: contracts can be designed so that the portfolio of clean energy projects keeps pace with a buyer’s growing electricity demand. 

Our paper shares three case studies where Google recently signed CFE Manager agreements — in Virginia, Germany, and California— and discusses some contractual terms and issues that buyers pursuing these agreements should consider. In each of these cases, we worked with our energy service provider to create a ready-made solution for carbon-free electricity supply that can be adopted by other clean energy buyers. These agreements represent a new energy service model that can support a customer’s journey toward full decarbonization of their electricity demand while accelerating the decarbonization of electricity grids.

A CFE Manager model for energy purchasing enables buyers to sign a single supply agreement for multiple facilities, avoid some risks associated with PPAs, and secure guaranteed carbon-free energy matching.

As Google progresses toward our goal of operating on 24/7 CFE by 2030, we will continue to support others on their own clean energy journeys, which is why we’ve released this new paper today. It’s also why we’re advocating for policies to accelerate the decarbonization of grids around the world and why we and our partners recently launched the global 24/7 Carbon-free Energy Compact — a set of commitments and actions from energy buyers, energy suppliers, governments, system operators, solutions providers, investors, and other organizations to advance 24/7 CFE. 

Direct PPA contracting by end users will continue to play an important role in the transition to a carbon-free grid. But for end users for whom PPAs are not practical or desirable, we hope the CFE Manager model will be a helpful tool to support and expand CFE procurement. We stand ready to work with energy suppliers, utilities, and other clean energy buyers to scale this new energy supply model and help advance 24/7 CFE for all.

GitOps takes DevOps best practices used for application development (such as version control and CI/CD) and applies them to infrastructure automation. In GitOps, the Git repository serves as the source of truth and the CD pipeline is responsible for building, testing, and deploying the application code and the underlying infrastructure.

Nowadays, an application is not just code running on infrastructure that you own and operate. It is usually a set of first-party and third-party microservices working together in an event-driven architecture or with a central service orchestrator such as Workflows.

Service orchestrations, which have their own definition files and deployment cycles, can benefit from a GitOps approach. This blog post describes how to set up a simple Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build. 

Architecture

Let’s take a look at the overall approach. 

In this approach, you have a staging branch where you make changes to a workflow. This triggers a Cloud Build configuration that deploys a test staging workflow and runs some staging tests against it. If all tests pass, Cloud Build deploys the staging workflow. After more manual testing of the staging workflow, you merge changes from the staging branch to the main branch. This triggers the same Cloud Build configuration to deploy a test production workflow, run more production tests, and if all tests pass, deploy the production workflow.

This approach enables you to have an automated and staged rollout of workflow changes with tests along the way to minimize risk. 

Configuration

Setting up such an automated workflow deployment pipeline is straightforward. 

First, you need a workflow definition file that can benefit from such automation. You can use one of your workflow definition files or workflow.yaml, which simply returns Hello World. 

Next, define a Cloud Build configuration file (see cloudbuild.yaml) with all the stages. In this configuration, Cloud Build deploys a test workflow with the branch name and commit hash, runs the workflow and captures the output, deletes the test workflow, and tests the workflow with the supplied test script. If all the tests pass, it deploys the final workflow in the branch.

Tests for the branch are defined in appropriate test-{branchname}.sh files. For example, test-staging.sh runs against the workflows deployed in the staging branch and only checks the workflow execution state. On the other hand, test-main.sh runs against the main branch, checks the workflow execution state, and also checks the output of the execution. You can add more tests as you see fit.

Connect your repository to Cloud Build

Now that you have the basic configuration in place, you connect your (or workflows-demos) repository to Cloud Build before creating triggers. Follow the instructions here.

Create a Cloud Build trigger

You now create a Cloud Build trigger to watch for commits to the main and staging branches. General instructions are here.

Go to the Create Trigger section of Cloud Build in the console and create a trigger with the following properties:

Name: workflows-trigger

Event: Push to a branch

Repository: GoogleCloudPlatform/workflows-demos

Branch: ^main$|^staging$

Included files filter: gitops/workflow.yaml

Configuration type: Cloud build configuration file

Cloud build configuration file location: gitops/cloudbuild.yaml

Add a Substitution variable with key/value: _WORKFLOW_NAME and workflows-gitops

Test the staging workflow

You’re now ready to test the build pipeline with the staging branch. 

Switch to the staging branch:

git checkout staging

Change Hello World in workflow.yaml to Bye World:

Commit and push the change to the staging branch:

You should see the trigger running:

After a few seconds, the build (including all its stages) is successful:

And a staging workflow has been deployed:

Test the production workflow

Once you’re ready to deploy the staging workflow to production, simply merge the staging branch to the main branch.

In this case, however, the build fails:

This is because the test script for the production workflow in test-main.sh is expecting to see Hello World as output of the workflow.

You need to go back to the staging branch, and change Bye World in workflow.yaml back to Hello World:

Check in your changes to staging, see the build succeed, and merge to main. Finally, you should also see the build succeed and see that a production workflow has been deployed alongside staging:

Next steps

This post covered how to set up a Git-driven development, testing, and deployment pipeline for Workflows using Cloud Build. All the details and sample configuration files are in our workflows-demos/gitops repository. 

Of course, Cloud Build is not the only way to set up such a pipeline. GitHub Actions is another useful tool that can help to set up similar service orchestration pipelines. Feel free to contribute to our repository with GitHub Actions based pipelines and reach out to me on Twitter @meteatamel for any questions or feedback.

Are you ready to take your cloud skills to new heights? We’re excited to announce the Google Cloud Fly Cup Challenge, created in partnership with The Drone Racing League (DRL) and taking place at Next ‘22 to usher in the new era of tech-driven sports. Using DRL race data and Google Cloud analytics tools, developers of any skill level will be able to predict race outcomes and provide tips to DRL pilots to help enhance their season performance. Participants will compete for a chance to win an all-expenses-paid trip to the season finale of the DRL World Championship Race and be crowned the champion on stage.  

How it works: 

Register for Next 2022 and navigate to the Developer Zone challenges to unlock the game

Complete each stage of the challenge to advance and climb the leaderboard

Win prizes, boost skills and have fun!

There will be three stages of the competition, and each will increase in level of difficulty. The first stage kicks off on September 15th, where developers will prepare data and become more familiar with the tools for data-driven analysis and predictions with Google ML Tools. There are over 500 prizes up for grabs, and all participants will receive an exclusive custom digital badge, and an opportunity to be celebrated for their achievements alongside DRL Pilots. There will be one leaderboard that will cumulate scores throughout the competition and prizes will be awarded as each stage is released. 

Stage 1: 

DRL Recruit: Starting on September 15th, start your journey here to get an understanding of DRL data by loading and querying race statistics. You will build simple reports to find top participants and fastest race times. Once you pass this lab you will be officially crowned a DRL recruit and progress for a chance to build on your machine learning skills and work with two more challenge labs involving predictive ML models. 

Prize: The top 25 on the leaderboard will win custom co-branded DRL + Google Cloud merchandise.

Stage 2: 

DRL Pilot: Opening in conjunction with the first day of Next 2022 on October 11, in this next stage you will develop a model which can predict a winner in a head to head competition and a score for each participant, based on a pilots profile and flight history. Build a “pilot profile card” that analyzes the number of crashes and lap times and compares it to other pilots. Fill out their strengths and weaknesses and compare them to real life performances, and predict the winner of the DRL Race in the Cloud at Next 2022, and be crowned top developer for this stage.

Prize: The first 500 participants to complete stage 2 of the contest will receive codes to download DRL’s Simulator on Steam.

Stage 3: 

DRL Champion: Continue this journey throughout the DRL championship season. Using the model developed in Stage 2. Use data from past races to score participants and predict outcomes. Provide pilots with real life tips and tricks to help improve their performance. The developer at the top of the leaderboard at the end of December 2022 will win an expenses-paid VIP trip to DRL’s final race in early 2023. 

Prize: Finish in the top 3 for an opportunity to virtually present your tips and tricks to professional DRL Pilots before the end of the 2022-2023 race season

Top the leaderboard as the Grand Champion and win an expenses paid VIP experience to travel to a DRL Championship Race in early 2023 and be celebrated on stage. For more information on prizes and terms please visit the DRL and Google Cloud website.  

Ready to Fly? 

The Google Cloud Fly Cup Challenge opens today and will remain available on the Next ‘22 portal through December 31, 2022 when the winner will be announced. We are looking forward to seeing how you innovate and build together for the next era of tech-driven sports. Let’s fly!