Google Cloud

We have never seen a proliferation of open source products sparked by a tool like we have seen with Kubernetes. These building blocks are what make Kubernetes the de facto container orchestrator to securely and reliably run services. Since organizations don’t run Kubernetes on its own, Kubernetes is more often than not deployed alongside other tools to solve real business challenges that Kuberntes doesn’t natively solve. Google offers a managed collection of these products under the Anthos umbrella.

In this article, we will demonstrate how organizations can leverage Anthos to centralize the management of internet traffic using Multi Cluster Ingress (MCI) and Anthos Service Mesh (ASM).

Problem statement

For large organizations, the separation of duty is a major concern. It’s this concern that drives the design of the cloud foundation. Projects are the boundaries to create least privileges strategy for most of the resources Google Cloud Platform provides. Hence, it’s fair to assume that separate teams would have separate projects to run their workloads.

This separation of duty applies also to the management of the networking. Shared VPC is one of the tools teams have to centrally manage networking. This allows multiple projects to share the same VPC. Having the same VPC is also required for some features Anthos provides.

Applications running in Kubernetes sometimes need to be exposed to the internet. Exposing the application can be achieved  using  a load balancer. The management of the lifecycle and configuration of the load balancer is done through annotations and custom resource definitions (CRD). MCI is a managed controller that manages the lifecycle of Google Cloud Global Load Balancer (GLBC) in order to expose services running on Kubernetes to the internet. Unfortunately, GLBC doesn’t support cross-project backends. Since projects are the foundation to implementing least privilege strategy, one solution is to deploy a GLBC per project. The management overhead that this would create for the network and security teams is substantial. The second solution is to deploy one GLBC using MCI and use ASM to route the traffic cross-projects. This can also enable security teams to achieve the required separation of duty with a seamless separation of responsibilities while minimizing confusion and accidental configuration errors. 

Now that we have talked about the problem we are solving and what tools we are going to use, let’s deep dive into how we are going to do it. 

Architecture Diagram

This architecture shows the suggested solution to support MCI in a multi project setup. Both the fleet project and the service project share the same VPC. The shared VPC host project is not depicted in the picture but it’s a separate project. The fleet project refers to the project where the Fleet API is enabled. All the clusters that are part of the same fleet are also part of the same mesh. One of the fleet GKE clusters hosts the MCI configuration, but the configurations are deployed in both Fleet GKE clusters for high availability. Deploying the same configuration in both clusters makes it both possible and easy to swap the configuration cluster. 

The service projects on the other hand host workloads. Development teams would have access to a namespace to deploy services they manage.

How to Implement this Solution

To implement this solution, a deep understanding of how the traffic flows is key. The following steps describe the process to make sure everything is working. Before starting, take the following actions:

Make sure MCI is already set up.

Make sure managed ASM is enabled.

Create a namespace for ingress gateway and other resources.

Conclusion

In this blog post, we have seen that even though MCI doesn’t support backends in different projects we can overcome this limitation using ASM. The combination of both products makes it easy to centralize traffic management and reuse the same GLBC for workloads running in different projects. Make sure to check the GKE Network Recipes repository for more GKE ingress examples.

With a global, always-on workforce, the U.S. Navy requires secure collaboration between teams across countries and time zones. This is especially relevant for the 50,000 U.S. Navy sailors deployed aboard approximately 100 ships at any given time, who need to connect with personnel at regional shipyards for everything from routine maintenance, to more serious ship repairs. 

Google Public Sector directly assists Naval Sea Systems Command (NAVSEA), the largest of the U.S. Navy’s five “systems commands,” by providing support to the U.S. Naval Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC) in Yokosuka and Sasebo, Japan. NAVSEA is using Google Workspace, which harnesses Google’s threat protection and zero trust capabilities, to enable effective, secure, and compliant collaboration between the SRF-JRMC and Navy stakeholders around the world.  

The workforce in the SRF-JRMC facility had two unique challenges. The first was around ineffective collaboration and communication channels. Before deploying Google Workspace, a Navy officer in Japan would need to be on-base to communicate over a secure connection, and calls were typically late at night given the time difference. This challenge was even further magnified when the COVID-19 pandemic hit, further restricting staff mobility. 

The second challenge was a language barrier. With more than 3,000 Japanese Master Labor Contract (MLC) employees providing critical support to SRF-JRMC’s shipyard, Navy leadership needed an easier way to communicate with their Japanese-speaking counterparts. Breaking down this barrier would ensure work could be completed faster and more effectively, facilitating a more collaborative environment.

Enabling secure collaboration

The Navy partnered with Google Public Sector to enable Google Workspace for SRF-JRMC. Today, Google Workspace assists the Navy’s shipyards by enabling Google Voice for secure voice over Internet Protocol (VoIP) calling options, so personnel can join calls and sessions on-site or at-home to communicate securely and reliably across continents. In addition, early access to solutions like English to Japanese translated captions in Google Meet help break down the language barrier by providing instant translation during meetings. The platform also saves the U.S. Navy thousands of dollars a month in phone bills by providing country-specific dial-in numbers for interviews. 

Moreover, Google Workspace tools like Google Drive and Docs also help to simplify human resource workflows by streamlining the hiring for onboarding local Japanese employees for the shipyard. Having a shared Google Drive eliminates the need to send multiple files back and forth among the NAVSEA team, allowing them to minimize on-premises storage space. In addition, Google Docs enables Navy employees to communicate and collaborate with each other and with potential candidates securely, and across any device.  

“As the largest overseas ship repair facility of the U.S. Navy, operational readiness and continuity of operations is our top priority,” said Peter Guo, chief information officer at SRF-JRMC. “Cloud collaboration capabilities provide us seamless and secure connectivity across continents and break down language barriers with our colleagues across the globe. We’ve improved our ability to operate anytime, and anywhere and have increased our ability to securely communicate and coordinate especially during network outages and natural disasters.”

Providing pandemic assistance

Google for Government’s Workspace solutions also became useful to the SRF-JRMC during the pandemic, providing valuable communication and collaboration tools during a time of uncertainty. In addition to deploying Google Workspace, SRF-JRMC’s IT department created a COVID-19 Pandemic Dashboard, a Google-based site that consolidated Japanese and international open-source data on COVID-19 outbreaks and provided updated guidance. The Dashboard was built in less than two hours, using Looker Studio, and it leveraged an automated data collection process to track local hospitalization numbers. Before building the site, it took the NAVSEA team hours to compile this information; now, the team can access this information in real-time. 

“This Dashboard drastically reduced redundant weekly meetings centered on COVID-19 updates, and saved more than 10 hours per week manually gathering data and presenting it via slide decks,” said Guo. “Additionally, this capability enabled our leadership to make real-time, data-driven decisions and put necessary risk mitigations in place. It empowered supervisors across the shipyard to reference this website at any time and put additional health measures in place to minimize the transmission of COVID-19. Every minute counts at our two shipyards in Japan, so this made a tremendous impact on our operational efficiency and ensured the safety of our sailors.”

Delivering the best possible tools means making life better and work more fulfilling for millions of people, inside and outside of government. To help government agencies maintain access to communications and collaboration tools that they need during and after an incident to keep work going, we are also offering workshops for federal, state and local governments. Learn more about Google for Government solutions for the Department of Defense, and Google Workspace for Government.

Editor’s note: This post is part of an ongoing series on IT predictions from Google Cloud experts. Check out the full list of our predictions on how IT will change in the coming years.

Prediction: By 2025, 90% of data will be actionable in real-time using ML

A recent survey uncovered that only one-third of all companies are able to realize tangible value from their data. As a result, organizations are saddled with the operational burden of managing data infrastructure, moving and duplicating data, and making it available to the right users in the right tools. 

At Google, data is in our DNA, and we want to make the same solutions that help us innovate available to our customers. For example, we helped Vodafone unify all their data so that thousands of their employees can innovate across 700 different use-cases and 5,000 different data feeds. They now run AI development 80% faster, more cost-effectively, and all without compromising governance and reliability.

In our own experience building data infrastructure, we’ve found the following principles to be helpful for overcoming barriers to value and innovation:  

You have to be able to see and trust your data. First off, spend less time looking for your data. Then, leverage automation and intelligence to catalog your data so you can be sure that you can trust it. Using automatic cataloging tools like Dataplex allows you to discover, manage, monitor, and govern your data from one place, no matter where it’s stored. Instead of spending days searching for the right data, you can find it right when you need it and spend more time actually working with it. Plus, built-in data quality and lineage capabilities help  automate data quality and troubleshoot data issues.  

You have to be able to work with data. Adopt the best proprietary and open source tools that allow your teams to work across all of your data, from structured to semi-structured to unstructured. The key is finding ways to leverage the best of open source, like Apache Spark, while integrating enterprise solutions, so you can deliver reliability and performance at scale. Imagine what’s possible when you can leverage the power of Google Cloud infrastructure without forking the open source code?    

You need to act on today’s data today — not tomorrow. Apply streaming analytics so you can work with data as it’s collected. Building unified batch and real-time pipelines allows you to process real-time events to achieve in-context experiences. For example, a streaming service like Dataflow lets you use Apache Beam to develop unified pipelines once and deploy in batch and real time. 

When you can see the data, trust the data, and work with data as it’s collected, we can see how 90% of data will become actionable in real-time using ML and the incredible innovation that it will unlock.

Early in the year, we shared the new capabilities that made BigQuery SQL more user-friendly than ever in our Valentine’s Day update. In this year-end edition, we are pleased to walk you through new SQL capabilities that we launched in the categories of Enterprise Class, Data Quality and Schema Operations.

Enterprise Class

For our enterprise data analysts, we are pleased to share new capabilities in BigQuery SQL that allow them to manage their workloads using the same familiarity that they may have used in legacy databases and data warehouses.

Sessions (GA)

A database session (or session) represents a connection between a user or a program and a database or data warehouse that stores and queries data. A data analyst through a tool or a program may execute multiple commands which are tracked under this session. BigQuery at its heart is a stateless query engine which means it doesn’t maintain a persistent network connection, as traditional database connections do. This allows BigQuery to scale compute and storage independently without limits.

For those enterprise data administrators looking to manage BigQuery activity through sessions, they can optionally enable session support in BigQuery using the UI, API and CLI or can by setting the EnableSession parameter in the JDBC or ODBC driver.  This allows enterprises to accrue all the benefits of sessions without the overhead of a persistent connection.

For data analysts, sessions provides the benefits of:

Session variables: which allows the creation of a session-specific variable that is persisted for all commands that get executed in the session.

System variables: Pre-build variables for timezone, dataset identifiers, projects, session identifiers, which if specified, allow the data analysts to override the organization or project level default values with session-specific settings.

Session-duration temporary tables and temporary functions: When temporary tables are used in a session to stage results or a session-specific temporary function is used to make query operations efficient, they are persisted for the entire duration of the session and are accessible to all jobs in that session.

For data administrators, the benefits are:

Session labels: By assigning labels for job tracking, data administrators can find all activities associated with that session using that label in the audit log.

Session management: data administrators can manually terminate sessions by calling the BQ.ABORT_SESSION system procedure with the session ID as input.

BigQuery sessions also support multi-statement queries and transactions. Documentation

Case-insensitive dataset and table names (Preview)

BigQuery tables names and dataset names are case sensitive, by default, reflecting its origins in big data analysis where dataset file names have maintained case sensitivity. To become more user-friendly to the tools and programs brought over by data analysts from their legacy data warehouses, BigQuery now supports case-insensitive table names and schema (dataset) names so that a dataset.Table ≡ DaTaSeT.TABLE ≡ Dataset.table ≡ DATASET.tAbLe in BigQuery. You can configure this in the definition (DDL) for dataset or table using the is_case_insensitive option. Documentation

Data Quality

As data analytics platforms, such as BigQuery, bring data from different sources to help with decision making, data engineers need to maintain the quality of critical data elements. Data sources may be imperfect and have incomplete information. Therefore, data engineers need to configure intelligent defaulting logic when column data is missing to ensure that the right data gets populated. Similarly, data pipelines need to provide flexible logic for numeric data to reduce bias and ensure accurate outcomes. Data analysts need to match text information independent of case to ensure accurate reporting.

Default column value (Preview)

When new rows are inserted into a table, some columns may not have any data. Default value expression allows the specification of a default value using a literal value or a function that computes the default value when the associated column data is missing.  The default value specification can be set when creating a new table or by altering the column properties of an existing table. In addition, DML statements such as INSERT or CREATE TABLE AS SELECT or MERGE allow the specification of the defaulting logic in place of an actual inserted value which uses the defaulting specification of the column being inserted into. Documentation

Case-insensitive string collation (Preview)

A collation specification when associated with a column operation, e.g. join, comparison or ORDER BY or GROUP BY clause, determines the logic used to compare or order string values. For example, the default collation specific in BigQuery is ‘binary’ which uses the code point order to specify the ordering sequence in Unicode by which all uppercase letters [A-Z] precede lower case letters [a-z]. We are pleased to offer case insensitive collation, specified using ‘und:ci’  by which [A,a] will be treated as equivalent characters and will precede [B. b] for string value operations. This allows data analysts to find matching string values independent of the case by treating Maclean and MacLean as equivalent.  Documentation

Banker’s rounding (GA)

BigQuery has been natively rounding all values that overflow maximum precision using the most common rounding logic: “round half away from zero”. Using “Round half away from zero” exclusively however, can introduce a rounding  bias in the data by causing the aggregate operation e.g. sum or average to drift away from the actual value. Other rounding techniques, such as  Gaussian rounding, commonly known as banker’s rounding, provide an alternate rounding scheme which does not suffer from negative or positive bias as much as the round half away from zero method over aggregations over most reasonable distributions. With this, BigQuery is pleased to introduce support for multiple rounding modes for columns in table definitions and in the explicit ROUND function: the existing mode of rounding, called “round_half_away_from_zero”, and a new mode, Bankers Rounding, otherwise known as “round_half_even”. The “round_half_even” mode rounds towards the nearest “neighbor” unless both neighbors are equidistant, in which case, round towards the even neighbor, e.g., 3.1 & 3.2 will round to 3 and 3.5 & 4.5 will round to 4. Documentation 

Schema operations

Continuing the expansion of SQL syntax for schema operations, BigQuery is pleased to introduce two new SQL capabilities: LOAD DATA to load data into tables and RENAME column to allow existing table columns to be renamed

LOAD DATA (Preview)
This statement provides data engineers and analysts with a SQL interface to load one or more files into tables. It provides the ability to load into a named table, create a new table , or truncate an existing table as a part of the load from all the file types supported by BigQuery including CSV, AVRO, Parquet, JSON, etc. In addition to specifying table options, such as table metadata or table expiration, the LOAD DATA command also allows users to specify partition and clustering schemes as a part of the table specification or from a hive partitioning specification for external files.   

LOAD DATA uses the same resource pool as the BigQuery load api to load data. If flat slots customers have configured a PIPELINE reservation, then the LOAD DATA will use the dedicated compute slots from the PIPELINE reservation assignment. Documentation

RENAME column (Preview)

RENAME COLUMN allows users to change a column name in an existing table; this is a metadata only change. Previously, renaming columns required a workaround which required column data to be rewritten. Now, when a column has to be renamed because the current naming is obsolete or was named incorrectly in error, data owners can run a zero cost metadata only command, RENAME column, to fix and correct the column name. Documentation

As we close out this year, we are excited about bringing new capabilities in the new year to you, our BigQuery data aficionados, to make BigQuery SQL more user-friendly than ever.

If you’re looking to free yourself from legacy databases and their overpriced, restrictive licensing, you’ve probably looked at cloud-based alternatives. Google Cloud’s database portfolio includes AlloyDB for PostgreSQL as a key modernization option. AlloyDB is a fully-managed, PostgreSQL-compatible database service for the most demanding enterprise database workloads, combining the best of Google with one of the most popular open-source database engines, PostgreSQL, for superior performance, scale, and availability.

No database is an island, of course. Modern applications require databases to work together with a fleet of services and solutions to deliver business outcomes. Oftentimes, when building or modernizing a complex application, finding the right services and solutions, and testing whether they work well with the database can be time consuming and expensive. How can you ensure that your entire application ecosystem provides you with openness and flexibility? 

Along with the general availability of AlloyDB, which was announced today, we’re excited to announce an expanded ecosystem of more than 50 technology solutions and service partners to support our customers’ deployments.

AlloyDB integrates with leading technology solutions to simplify building and modernizing your applications on AlloyDB. Today, 33 partner solutions across 30 technology vendors are ready to support your needs in business intelligence, advanced analytics, data integration, governance, observability, and migration assessment. Google Cloud has closely collaborated with these partners to ensure the best experience and support when you use their solutions with AlloyDB.

Confluent expanded their managed Kafka Connect offering with a dedicated sink connector to stream data into AlloyDB from various upstream sources. The connector is now available for use.

“There is unprecedented growth in the volumes of data being generated, and it’s essential for businesses to be able to contextualize and harness the value of their data in real time”, said Paul Mac Farland, Vice President Customer Solutions and Innovation Design, Confluent. “Today’s organizations need scalable, cloud-native databases like AlloyDB for improved agility, elasticity, and cost-efficiency. We’re excited to help our customers build and deliver real time applications with the joint power of Confluent and AlloyDB.” 

Collibra, a leader in data intelligence and governance, released an AlloyDB driver in their marketplace which can be used in the Collibra Catalog by AlloyDB customers. Leveraging this driver, Collibra Catalog will be able to register AlloyDB database information and extract the structure of the source into its schemas, tables and columns.

“Collibra is proud to support AlloyDB. AlloyDB compatibility with PostgreSQL has made it easy for tools like Collibra to support integration from an enterprise data governance and cataloging perspective”, said  Koen Van Duyse, Vice President Partner Success – Cloud Partners at Collibra. “Customers will benefit both from the performance increases that AlloyDB has to offer while ensuring the solution aligns well with our Enterprise Data Intelligence best practices.”

Many organizations partner with systems integrators and consultants to build new applications and to migrate and modernize existing ones. We’ve cultivated an ecosystem of 21 regional and global service partners, who have tested AlloyDB extensively and developed the necessary expertise to help you with implementations. They can migrate your PostgreSQL databases from on premises or other clouds, and can also execute heterogeneous migrations from legacy databases, including all necessary schema and code conversion.

Our ecosystem of  global system integrators provide consultation, development, deployment, maintenance and support services for global organizations. For example, Infosys is a global leader in next-generation digital services and consulting that enables customers in more than 50 countries to accelerate cloud-led transformation with Infosys Cobalt, a powerful set of services, solutions and platforms. 

“Our clients are increasingly moving their databases to managed services in the cloud, and rely on our deep expertise to modernize their IT landscape with zero disruption. AlloyDB provides a powerful new option for modernizing legacy databases with its full PostgreSQL compatibility, high performance and high availability. Combined with our Infosys Cobalt Modernization Suite, AlloyDB will help our clients accelerate their database modernization journey,” said Gautam Khanna, Vice President and Global Head, Modernization Practice, Infosys.

Regional partners such as Pythian provide more choice and deeper expertise to Google Cloud customers. Pythian has been dedicated to designing, implementing, and supporting data solutions for 25 years. Organizations across industries choose and value Pythian to reduce the risk and cost of operating data platforms and accelerate their time and value to insights.

“AlloyDB brings impressive new capabilities to PostgreSQL, particularly in availability and performance,” said Paul Lewis, Chief Technology Officer at Pythian. “We’ve seen firsthand how AlloyDB’s Columnar Engine delivered outstanding results – at least 26x faster in one of our benchmarks, with potentially much more. It really transforms PostgreSQL into a hybrid transactional/analytical processing (HTAP) database engine. We’re excited about bringing AlloyDB to our customers to run their most demanding workloads.”

Get started with AlloyDB

A modern database strategy plays a critical role in developing great applications and delivering new experiences faster. With a vibrant partner ecosystem supporting AlloyDB, you can confidently define your application strategy and ecosystem solutions with a wide selection of compatible, well-supported 3rd-party products and services.

To learn more about the technology innovations behind AlloyDB, check out the deep dive into its intelligent storage system. Also visit cloud.google.com/alloydb to get started and create your first cluster. To see the list of partners who work closely with us, visit the partner section on AlloyDB’s product site.

In May 2022 at Google I/O, we announced the preview of AlloyDB for PostgreSQL, a fully-managed, PostgreSQL-compatible database service that provides a powerful option for modernizing your most demanding enterprise database workloads. We’re happy to announce that AlloyDB is now generally available.

AlloyDB is an excellent choice for organizations looking to break free from their legacy, proprietary databases and for existing PostgreSQL users looking to scale with no application changes. It combines full PostgreSQL compatibility with the best of Google: scale-out compute and storage, integrated analytics, and AI/ML-powered management. That means you get better performance, availability, and scalability with minimal management overhead. 

Based on our performance tests, AlloyDB is more than four times faster for transactional workloads, and up to 100 times faster for analytical queries than standard PostgreSQL. It’s also two times faster for transactional workloads than Amazon’s comparable PostgreSQL-compatible service. You can expect high out-of-the-box regional availability, backed by a 99.99% availability SLA inclusive of maintenance. AlloyDB automatically detects and recovers from most database failures within 60 seconds, independent of database size and load. And finally, autopilot systems for automatic provisioning of storage, adaptive auto-vacuuming, and more make it easier than ever to manage your database. Pricing is transparent and predictable, with no expensive, proprietary licensing and no hidden I/O charges.

Since our preview announcement in May, we’ve introduced a number of new product capabilities to address the needs of high-end enterprise applications. We’ve added critical security features like customer-managed encryption keys (CMEK) and VPC Service Controls, and announced the preview of cross-region replication. We expanded configuration options, including introducing a new 2 vCPU /16 GB RAM machine type, and adding support for additional PostgreSQL extensions like pgRouting, PLV8, and amcheck. We continued to make AlloyDB easier to manage, introducing the preview of an index advisor and new fleet-wide monitoring.

Earlier, in September , we announced the preview of PostgreSQL to AlloyDB migrations with our easy-to-use, secure, and serverless Database Migration Service. We also announced the preview of AlloyDB’s integration with Datastream for seamless change data capture and replication to destinations like BigQuery. And finally, we continued to make improvements in performance, availability, replication lag, and monitoring to better meet the needs of your workloads.

Customer and partner momentum

We’re seeing increased momentum from organizations looking to modernize their database estates with high performance, scale, and availability. At this opportunity we’d like to thank everyone who participated in the preview and sent us useful feedback.

B4A, a Brazilian beauty-tech startup, is one such customer. B4A was founded with the purpose of democratizing the beauty market, offering monthly beauty subscription services to more than 100,000 paying subscribers. “We opted for AlloyDB for performance reasons, and indeed, the results were amazing,” says Jan Riehle, CEO and Founder at B4A. “Combining AlloyDB with a GraphQL API decreased query times for our full catalog of products by up to 90 percent compared to our previous database solution. We also came to appreciate the ease of maintenance that a fully managed solution like AlloyDB provides – setting up and configuring the database was a very smooth process that went really quickly for us. And compared to our previous database, we’ve been able to cut costs, and are no longer stuck paying for traditional database licenses.”

London-based technology company Thought Machine is on a mission to create technology that can run the world’s banks according to the best designs and software practices of the modern age. “We were delighted to be in the preview programme with AlloyDB, and now to be a launch partner as it comes into general availability,” says Will Montgomery, CTO at Thought Machine. “We are confident that AlloyDB performs at the highest level for our global Tier 1 bank clients running on Google Cloud Platform, which have stringent demands on performance and availability.”

Today we’re also happy to announce the expansion of the AlloyDB partner ecosystem with several new partnerships, for a total of over 40 technology and service partners to support your AlloyDB-based deployments. Technology vendors are ready to support your needs in business intelligence, advanced analytics, data integration, governance, observability, and migration assessment, while regional and global service partners have the necessary expertise to help your database migrations and other implementation needs. Read more about the expanded AlloyDB partner ecosystem.

What else do I need to know?

Preview pricing will continue through December 31, 2023, after which normal billing will commence. For additional pricing information, see AlloyDB pricing. You can find more information about product functionality, regional availability, and our service-level agreement (SLA) in the AlloyDB for PostgreSQL documentation.

Learn more about AlloyDB

Check out the Introducing AlloyDB for PostgreSQL video for the full story on migrating, modernizing and building applications with AlloyDB. You can also learn about the technology underlying AlloyDB from our “Under the Hood” blog series. We cover the disaggregation of compute and storage in Intelligent, database-aware storage and explore AlloyDB’s vectorized columnar execution engine, which enables analytical acceleration with no impact on operational performance, in the Columnar engine blog post.

Also check our animated video series, “Introducing AlloyDB”. The first episode, What is AlloyDB?, discusses how AlloyDB provides full PostgreSQL compatibility together with accelerated database performance, high availability, and analytics integration. Be sure to subscribe for future episodes.

Getting started

You can get started with AlloyDB in just a few clicks by navigating to the AlloyDB console and creating your first cluster. 

Getting started is easy with three simple steps:

Create your AlloyDB cluster

Create a primary instance for that cluster

Connect to the cluster using a psql client

Check out our quickstart for a step-by-step guide on creating a new AlloyDB cluster. Or you can use Database Migration Service to easily migrate an existing PostgreSQL database to AlloyDB.

Global businesses across industries consistently trust us when they build geospatial products and solutions for their own customers. One of the key reasons businesses choose Google Maps Platform is the freshness of its points of interest (POI) data. For example, using Google Maps Platform, a travel site can help its customers discover open restaurants near a vacation rental. Or a grocery chain can ensure its customers know its updated weekend hours. Through the Places API, Google Maps Platform provides developers with access to business information, ratings, and reviews for over 200 million businesses and places around the world.

A question we often get from our customers and industry peers is how Google Maps Platform maintains the freshness and accuracy of its POI data. In this blog, we will explore two ways in which we provide our customers with access to up-to-date POI information. 

Street View and AI/ML capabilities drive faster and more accurate POI insights 

We continue to evolve our AI and machine learning capabilities to maintain POI information. Street View imagery lets you virtually explore the world, and helps us accurately reflect local information about places. We’ve traveled across countries to capture this imagery and bring new information online—from unmapped roads to new businesses. However, to identify information like a new business or street name from signs in an image, we use text recognition, which is challenging in the natural environment—especially at scale. One reason is that the average Street View photo has more in the image than just a business name or street sign. They have cluttered backgrounds with additional buildings and cars. They show the signs from only one angle from which the image was taken. And they might even be blurry or have lighting issues. After years of training machine learning models, our text recognition systems can tune out these distractions and detect business names and addresses even when they’re handwritten on the side of a building or abbreviated. These models can understand a variety of languages across various scripts, from Latin, Cyrillic and Thai to Mandarin, Japanese and Korean.

Applying artificial intelligence and machine learning to our more than 220 billion Street View images helps us create high-quality maps faster than we could before. For instance, applying machine learning models to Street View imagery has helped improve the accuracy of address data available to customers in Google Maps Platform, resulting in more reliable geospatial experiences for their end users. Additionally, we also use algorithmic detection of businesses, facilitated by extracting details about a business from its web page, to ensure up-to-date POI information.

Contributions from our user community, business owners, and government partners facilitate constant additions and updates to POI data 

In addition to using our own geospatial data libraries such as Street View and algorithmic detection of businesses, we also rely heavily on our users, businesses, and partners to provide us with constant updates to POI information. Thanks to our Local Guides community of over 150 million contributors around the world and partnerships covering more than 10,000 local governments, transit agencies and organizations around the world, we’re able to reflect the latest information on our maps. 

We also continue to bring the most relevant features of Google Maps to Google Maps Platform so developers can incorporate helpful information into their products and services. For instance, using the Places API, you can now add editorial summaries, special business hours, and show reviews sorted by most recent or relevant. And restaurants can now add key POI attributes, like takeout, delivery, dine-in, and curbside pickup. By sharing rich, descriptive content about your business, you can give customers the information they need before they visit. 

These are just a few of the ongoing efforts to keep our POI data up-to-date. Stay tuned for the next blog on how Google Maps Platform enabled millions of businesses and government organizations to provide critical information and services during the pandemic. Until then, you can learn more about how we use AI and imagery to build a self-updating map.

For more information on Google Maps Platform, visit our website.

Vodafone is currently the second largest telecommunication company in Hungary, and recently  acquired UPC Hungary to extend its previous mobile services with fix portfolio. Following the acquisition, Vodafone Hungary serves approximately 3.8 million residential and business subscribers. This story is about how Vodafone Hungary benefited from moving its data and analytics platform to Google Cloud. 

To support this acquisition, Vodafone Hungary went through a large business transformation that required changes in many IT systems to create a future-ready IT architecture. The goal of the transformation was to provide future-proof services for customers in all segments of the Hungarian mobile market. During this transformation, Vodafone’s core IT systems changed, which created the challenge of building a new data and analytics environment in a fast and effective way. During the project data had to be moved from the previous on-premises analytics service to the cloud. This was achieved by  migrating existing data and merging them with data coming from the new systems in a very short timeframe of  around six months.  During the project there were several changes in the source system data structure that needed to be adapted quickly on the analytics side to reach the Go Live date.

Data and  analytics in Google Cloud

To answer this challenge, Vodafone Hungary decided to partner with Google Cloud. The partnership was based on implementing a full metadata-driven analytics environment in a multi-vendor project using cutting edge Google Cloud solutions such as Data Fusion and BigQuery. The Vodafone Hungary Data Engineering team gained significant knowledge of the new Google Cloud solutions, which meant the team was able to support the company’s long-term initiatives.

Based on data loaded by this metadata-driven framework, Vodafone Hungary built up a sophisticated data and analytics service on Google Cloud that helped it become a data-driven company.

By analyzing data from throughout the company with the help of Google Cloud, Vodafone was able to gain insights that provided a clearer picture of the business. They now have a holistic view of customers across all segments. 

Along with these core KPIs, the advanced analytics and Big Data models built on the top of this data and analytics services ensures that customers get more personalized offers than was previously possible.. It used to be the case that a business requestor needed to define a project to send new data to the data warehouse. The new metadata-driven framework allows the internal data engineering team to onboard new systems and new data in a very short time (within days), thus speeding up the BI development and decision-making process.

Technical solution

The solution uses several technical innovations to meet the requirements of the business. The local data extraction solution is built on the top of the CDAP and Hadoop technologies written in CDAP pipelines, PySpark jobs, and Unix shell script. In this layer, the system gets data from several sources in several formats including database extracts and different file types. The system needs to manage around 1,900 loads on a daily basis, and most data arriving in a five-hour time frame. Therefore, the framework needs to be a highly scalable system that can handle the high loading peaks without generating unexpected cost during the low peaks.

Once collected, the data from the extraction layer goes to the cloud in an encrypted and anonymized format. In the cloud, the extracted data lands in a Google Cloud Storage bucket. By arriving at the file, it triggers the Data Fusion pipelines in an event-based way by using the Log Sink, Pub/Sub, Cloud Function, and REST API. After triggering the data load, Cloud Composer controls the execution of the metadata-driven, template-based, auto-generated DAGs. Data Fusion ephemeral clusters were chosen as they adapt to the size of each data pipeline while also controlling costs during low peaks. 

The principle of limited liability is important. Each component has a relatively limited range of responsibilities, which means that Cloud Function, DAGs, and Pipelines contain the minimum responsibilities and logic that is necessary to finish their own tasks.

After loading this data into a raw layer, several tasks are triggered in Data Fusion to build up an historical aggregated layer. The Vodafone Hungary data team can use this to create their own reports in a Qlik environment (which also runs on the Google Cloud environment) and build up Big Data and advanced analytical models using the Vodafone standard Big Data framework. 

The most critical point of the architecture is the custom triggering function, which handles scheduling and execution of processes. The process triggers more than 1,900 DAGs per day, while also moving and processing around 1 TB of anonymized data per day.

The way forward

After the stabilization, the optimization of the processes started taking into account cost and efficiency levels. The architecture was upgraded to use Airflow 2 and Composer 2 as these systems became available. Moving the architecture to these versions increased performance and manageability. Going forward, Vodafone Hungary will continue searching for even more ways to improve processes with the help of the Google Support team. 

To support fast and effective processing, Vodafone Hungary recently decided to move the control tables to Google Cloud Spanner and keep only the business data in BigQuery. This delivered a great improvement in  processing.

In the analytics area, Vodafone Hungary plans to move to more advanced and cutting-edge technologies, which will allow the Big Data team to improve their performance by using Google Cloud native machine learning tools such as Auto ML and Vertex AI. These will further improve the effectiveness of the targeted campaigns and offer the benefit of advanced data analysis.

To get started, we recommend you check out BigQuery’s free trial and BigQuery’s Migration Assessment.

Everyone wants affordable, quality healthcare but not everyone has it. A 2021 report by the Commonwealth Fund ranked the U.S. in last place among 11 high-income countries in healthcare access.1 Carbon Health is working to change that. We are doing so by combining the best of virtual care, in-person visits, and technology to support patients with their everyday physical and mental health needs.

Rethinking how data and analytics are accessed at Carbon Health 

Delivering premium healthcare for the masses that’s accessible and affordable is an ambitious undertaking. It requires a commitment to operating the business in an efficient and disciplined way. To meet our goals, our teams across the company require detailed, daily insights into operating results.

In the last year, we realized our existing BI platform was inaccessible to most of our employees outside of R&D. Creating the analytics, dashboards, and reports needed by our clinic leaders and executives required direct help from our data scientists. 

However, this has all changed since deploying Looker as our new BI platform. We initially used Looker to build tables, charts, and graphs that improved how people could access and analyze data about our operating efficiency. As we continued to evaluate how our data and analytics should be experienced by our in-clinic staff, we learned about Connected Sheets for Looker, which has unlocked an entirely new way of sharing insights across the company.

A new way to deliver performance reporting and drive results

Connected Sheets for Looker gives Carbon Health employees who work in Google Sheets—practically everyone—a familiar tool for working with Looker data. For instance, one of our first outputs using the Connected Sheets integration has been a daily and weekly performance push-report for the clinic’s operating leaders, including providers. 

Essentially a scorecard, the report tracks the most important KPIs for measuring clinics’ successes, including appointment volume, patient satisfaction such as net promoter score (NPS), reviews, phone call answer rates, and even metrics about billing and collections. To provide easy access, we built a workflow through Google App Script that takes our daily performance report and automatically emails a PDF to key clinic leaders each morning. 

Within the first 30 days of the report’s creation, clinic leaders were able to drive noticeable improvements in operating results. For instance, actively tracking clinic volume has enabled us to manage our schedules more effectively, which in turn drives more visits and enables us to better communicate expectations with our patients. Other clinics have dramatically improved their call answer rates by tracking inbound call volume, which has also led to better patient satisfaction. 

Greater accountability, greater collaboration

As you can imagine, a report that holds people accountable for outcomes in such a visible way can create some anxiety. We’ve eased those concerns by using the information constructively, with the goal to use reporting as a positive feedback mechanism to bolster open collaboration and identify operational processes that need improvement. For example, data about our call answer rates initiated an investigation that led to an operational redesign of how phones are deployed and managed at more than 120 clinics across the U.S.

Looker as a scalable solution with endless applications

We’re now rolling out Connected Sheets for Looker to deliver performance push-reporting across all teams at Carbon Health. Additionally, we continue to find new ways to leverage Connected Sheets for Looker to meet other needs of the business. 

For instance, we’ve recently been able to better understand our software costs by analyzing vendor spend from our accounting systems directly in Google Sheets. Going forward, this will allow us to build a basic workflow to monitor subscription spend and employee application usage, which will lead to us saving money on unnecessary licenses and underutilized software. 

We’ve come a long way in the last year. Between Looker and its integration with Google Sheets, we can meet the data needs of all our stakeholders at Carbon Health. Connected Sheets for Looker has been an impactful solution that’s going to help us drive measurable results in how we deliver premium healthcare to the masses.

1. Mirror, Mirror 2021: Reflecting Poorly
2.  HEALTHCARE EDITORS’ PICK Meet The Immigrant Entrepreneurs Who Raised $350 Million To Rethink U.S. Primary Care

In a startup, you need to get to the MVP fast, gather feedback from early adopters, and iterate in a quick cycle. Anything that takes time away from developing and iterating on features delays the launch and that’s a serious problem when time-to-market is crucial. 

Google Cloud offers products that can help you to build and run your backend services on a fully managed serverless platform, saving time and freeing you from the burden of provisioning and managing infrastructure needed to run those services. Less time on infrastructure means more time implementing and iterating on your services. Additionally, most serverless products have a pay-as-you-go pricing model with little upfront cost at the beginning.  Pricing increases in line with the scale of your services and your startup, so you only pay for what you need, when you need it.

Let’s take a look at some of these products and see how they can help you to iterate fast. 

Where to run code?

First, you need to decide where to deploy and run your code. Cloud Functions is the easiest way to run code in Google Cloud. With Cloud Functions, you write a simple function in a supported language, decide how that function should be triggered (via HTTP or events), and let Cloud Functions handle all the details of how to run, trigger, and autoscale that function. 

Cloud Functions is designed primarily for single-purpose use cases and not for entire apps, so it’s important to be aware of its limitations. For example, you can use a limited set of languages and your functions need to respond back within a certain amount of time. 

When you run into a Cloud Functions limitation, you might be able to use Cloud Run for more flexibility. With a Cloud Run service, you run serverless HTTP containers instead of functions. This enables you to define your own runtime, so you can use any language you want and you can implement more sophisticated logic (rather than a simple function), as long as everything can run in a container. 

If all you need is a piece of code that runs to completion (rather than a service that handles requests continuously), you can use Cloud Run jobs, where you can execute long-running containers in parallel on a fully managed serverless environment. 

In both Cloud Functions and Cloud Run, you focus on writing your function or container and Google Cloud takes care of the infrastructure needed to run and scale it. This enables you to iterate fast on the features of your service.

Cloud Code for serverless development

Whether you use Cloud Functions or Cloud Run to run your services, you can rely on Cloud Code to quickly create and deploy your service to Google Cloud from your favorite IDE. 

In Cloud Code, there are templates to quickly create Cloud Functions or Cloud Run applications:

You can then deploy those services and see the list of deployed Cloud Functions and Cloud Run services all from the IDE:

How to coordinate services?

Once you have a service running, you need to decide how to trigger it beyond simple HTTP. You can use Cloud Scheduler for time-based executions. For more control, you can front any HTTP service with a Cloud Tasks queue, which gives you sophisticated features out of the box such as retries, rate limiting, deduplication, and more.

With multiple services, you also need to decide how your services will communicate and coordinate. The options are:

Direct service-to-service communication

Indirect event-driven communication (also known as choreography)

A central orchestrator directing the communication

Direct service-to-service communication is easy to implement. However, it creates tight coupling between services, so it’s a good idea to avoid it.

For event-driven communication, Google Cloud has Pub/Sub and Eventarc. Pub/Sub is low-level service with a high degree of event-delivery customization whereas Eventarc is higher level service with easier semantics to connect services. The important point is that both are fully managed, global services that help you to connect event sources to event consumers without you having to worry about the underlying infrastructure.

For service orchestration, you can use Workflows. You define how services should be called in what order in a service orchestration definition. Workflows takes care of making those calls in the way you defined while capturing the outputs (and handling failures) along the way.

All of these coordination services run in a serverless way without you having to worry about the underlying infrastructure, giving you more time to work on your own services. 

What if you need a server?

Sometimes you need a server due to some limitation of the serverless platform. For example, all serverless platforms have time limits on how long a workload can run and what CPU and memory resources it can have.  Even if you can’t run your code on a serverless service such as Cloud Functions or Cloud Run, you can still use Batch to run and manage batch jobs on virtual machines (VM) in Compute Engine. Similarly, you can use Workflows and its Compute Engine connector to run your code on a VM and manage the VM lifecycle with Workflows. In both cases, the actual workload runs on servers (giving you the full flexibility that comes with that) but the management of those servers is done in a serverless way with services like Batch and Workflows. 

Conclusion

Whenever you need to run some piece of code, ask yourself: Can I run this code on a serverless platform? Even if the answer is no, you might be able to manage its lifecycle on a serverless platform. The more infrastructure management you can delegate to the cloud, the more precious time you’ll have for developing and iterating your own services.

Editor’s note: OVO Energy was founded in 2009 as an energy retailer that redesigned the energy experience to be fair, simple and green for consumers. Inspired by Google’s carbon-aware load shifting to data centers with the cleanest energy, OVO Energy implemented its own solution using Cloud Composer and Cloud Run Jobs. Here Jack Lockyer-Stevens, Software Engineer at OVO Energy, describes the solution which he open-sourced as VertFlow.

Our world needs a greener grid

The transition to cleaner sources of electricity, such as wind and solar power, is critical to mitigating climate change and reducing humanity’s dependence on fossil fuels.

It’s tempting to think this can be easily achieved by building more wind and solar farms. But what happens when the wind doesn’t blow and the sun doesn’t shine? The variable output of renewable energy sources means we have to supplement them with power generated from fossil fuels when demand is high. 

One critical tool for addressing this is demand response, which incentivizes  consumers to shift their power consumption to different times when more renewable energy is available on the grid. OVO Energy offers this service to owners of electric vehicles, allowing them to charge at greener, off-peak times and return surplus power back to the grid when clean power supply is low.

The grass is greener in the other Google Cloud region

If OVO Energy can shift electricity demand from hardware, then why not for software?

Workloads running in a Google Cloud region draw from the local electricity grid, and some grids have a lower carbon intensity than others.

Regions like Paris (europe-west9) have an abundance of carbon-free power. Other locations like London (europe-west2), whose carbon intensity over three days is pictured above, are more variable as the availability of renewable power waxes and wanes with the wind and sun.

Cloud Run Jobs gives a container the green light in seconds

Cloud Run Jobs is a new serverless platform for running containers to completion on Google Cloud. It’s great for batch processes like sending invoices or running extract, transform, load (ETL) jobs. Rather than managing and paying for a Kubernetes cluster 24/7, it allows you to kick off an arbitrary container on fully managed Google infrastructure, in any region, in seconds.

Cloud Run Jobs pairs well with Cloud Composer, a fully managed workflow orchestration service built on Apache Airflow. It acts as mission control for scheduling, triggering and monitoring containerised jobs.

At OVO Energy we developed VertFlow, a free open source task operator for Airflow that can reduce the carbon emissions associated with a workload by automatically deploying your container into the greenest Google Cloud region at runtime. It does this using real-time carbon intensity data from CO2 signal. 

It’s a breeze to deploy greener Cloud Run Jobs on Composer with VertFlow

Let’s walk through the following steps to run your first container with VertFlow:

Setup VertFlow: Install, connect, authorize.

Create a DAG using the VertFlowOperator to deploy your Cloud Run Job.

Kick off the process and see which Google Cloud region is selected.

You need an Airflow environment to get started. Cloud Composer is a fully managed service based on Airflow and can be provisioned in minutes using the Cloud Console, gcloud or Terraform.

1.Setup VertFlow: Install, connect, authorize

Start by installing VertFlow on your Composer environment:

Authorize your environment to manage Cloud Run Jobs on your behalf, by updating permissions on the service account:

You will need to get an API key from Electricity Maps to enable VertFlow to access real-time carbon intensity data for the electricity grid in each Cloud Run region. You can get a free one for non-commercial use. Commercial users need to sign up here for an API Key. 

Navigate to the Airflow UI and save the key in a new Airflow Variable called VERTFLOW_API_KEY.

Ensure your environment has outbound access to the internet.

2.Create a workflow using theVertFlowOperatorto deploy your Cloud Run Job.

Create a Python file locally to define your workflow (known as a DAG in Airflow).

We use the VertFlowOperator  to define a job specification, including the address of the container image to run, command to execute and the allowed list of regions in which to deploy the job.

Update the vertflow_example.py file in the dags directory with the service account you want your Cloud Run container to use.

Upload the workflow to your Cloud Composer environment:

3. Kick off the process and see which Google Cloud region is selected.

Trigger the DAG:

VertFlow will run your container in the allowed_region with the lowest carbon intensity. You can then monitor its progress in the Airflow UI:

You should see Hello World in the logs of the Cloud Run Jobs console:

In this example, Belgium was a clear winner. We used nearly a third less carbon than if we’d run the job elsewhere.

VertFlow puts wind in the sails of OVO Energy’s sustainability journey

At OVO Energy we host many of our customer journeys on Google Cloud, from onboarding customers to processing payments. Many of our data and analytics pipelines comprise containerized batch jobs often orchestrated using the KubernetesPodOperator on Cloud Composer V1. This setup is rife for optimization by migrating to Cloud Composer V2 to take advantage of the more efficient resource consumption of Google Kubernetes Engine (GKE) Autopilot. VertFlow was developed as the last piece of the puzzle to allow us to go fully serverless and practice what we preach on sustainability.

VertFlow is perfect for batch workloads that are:

Containerized and stored in Artifact Registry

Relatively short, satisfying Cloud Run’s one hour runtime limit

Are I/O light but compute intensive, such as our energy forecasts. This ensures that the carbon saving of moving the compute is not outweighed by the carbon cost of transmitting the data over a larger distance.

Do not need access to resources on a VPC, such as our data pipelines that read from/write to BigQuery only

Are legally allowed to move from one region to another. We set the allowed_regions parameter for jobs with data residency constraints

Find out more about how OVO Energy is powering progress towards net zero carbon living with Plan Zero and learn more about VertFlow.

Incumbent financial services institutions (FSIs) are currently in the midst of a generational shift in enterprise technology toward cloud. These institutions are making a bet that cloud can help accelerate their digital transformation journey and give them the speed, agility, and scalability needed to compete with fintech. A foundational piece of this transition is around modernizing legacy core systems (mostly mainframes), however FSIs have been slow to act given  the time, cost, and risk associated with transitioning from decades-old, mission-critical legacy systems to modern cloud-native solutions. And while the legacy systems generally perform well in terms of resiliency and availability, they are increasingly expensive to maintain and difficult to support (e.g., talent pools for COBOL developers are ever shrinking). They also struggle to meet growing market demands around delivering data-driven, personalized engagement in near real time and to be agile enough to support open banking and embedded finance ecosystems that hinge on API-enabled microservices (e.g., BNPL financing for your favorite consumer product).

As these limitations become more apparent and acute, momentum is building across the industry to make progress on core modernization and FSIs are exploring a number of different approaches to minimize risk and maximize time to value. These approaches range from “ripping and replacing” by doing a wholesale replacement of core systems to a more phased option that focuses on “hollowing out” the existing core by progressively thinning it. This latter approach starts with prioritizing a set of customer journeys and rebuilding them as cloud native microservices while slowly deprecating legacy services in parallel. This approach can also now take advantage of new technologies like Google Cloud’s Dual Run that enables customers to simultaneously run workloads on legacy and cloud based services, allowing them to perform real-time testing and quickly gather data on performance and stability with no disruption to their businesses. 

Another common approach focuses on launching a greenfield bank in a targeted area that’s powered by a new, cloud-native core banking provider. This approach allows FSIs to get immediate value from the modern tech, albeit at a small scale out of the gate as it is typically only focused on new customers. 

One increasingly popular approach is to combine the “hollowing out” of your legacy core along with launching a greenfield bank on a cloud native core at the same time. FSIs who choose this path, are able  to test and learn with new core technology in a contained manner, while also continuing the heavy lifting of gradually replacing the legacy core. 

Whichever path to modernization an institution chooses, it needs a robust ecosystem of technology partners to deliver an end-to-end banking solution that covers both traditional transactional functions and newer ways of reaching customers and managing regulatory requirements, security, and risk. 

How Thought Machine, Mambu, and Finxact deliver customer value with Google Cloud

To that end, Google Cloud has been investing in and building such an ecosystem of fintech partners and our own services that provide open-cloud, best-in-class security, and data capabilities to banks. Our goal is to support the financial services industry as it evolves through this period of disruption and accelerated digital transformation.

“For too long, banks have been consigned to their slow, inflexible core systems,” says Paul Taylor, CEO and founder ofThought Machine, which offers Vault Core, a cloud-native core banking system that allows banks to run their core technology the cloud, as well as build and deliver highly personalized banking products. “Pairing Thought Machine’s next-generation core-banking capabilities with the scale, security, and open cloud ethos of Google Cloud has been nothing short of transformational for our customers.” 

Cloud-native core banking systems like Thought Machine, Mambu, and Finxact leverage Google Cloud technologies like Apigee (API management), Google Kubernetes Engine (GKE), and Security Command Center to provide highly secure, scalable, interoperable, and extensible services to their customers.

Mambu, for example, moved the core of its workloads to GKE to make it easier for customers to integrate Mambu’s services with their legacy systems and multiple clouds for flexible interoperability. GKE also offers high availability and resiliency out of the box, helping Mambu meet customers’ stringent SLAs across 65 countries. Compute Engine, with its wide range of VM hardware, allows Mambu to provide the speed and latency customers need.

“Security is critical,” says Fernando Zandona, Mambu’s Chief Product & Technology Officer. “With Google Cloud, we can ensure all our data is encrypted [and has] access restrictions that are managed from a service to a network level. By building on Google Cloud’s global, secure infrastructure, we can also provide local data residency for countries that require it.”

Frank Sanchez, founder and CEO of Finxact (and now vice chairman of Fiserv, which acquired Finxact in April 2022), says the scalability and data capabilities of Google Cloud provide profound value. “Banks don’t have to look into a crystal ball and build for a worst-case volume scenario,” he says. “They can launch products and markets with more fluidity and not have to outlay huge capital expenditures.”

Finxact provides an advanced System of Record for financial institutions that require high velocity transaction processing against multi-position accounts. “We’re a reliable, scalable, and secure engine for banking as a platform, which integrates with the rest of an ecosystem to deliver an end-to-end solution,” says Sanchez. “We feel cloud providers have a lot to say about what that ecosystem looks like. Google Cloud is making the biggest commitment in the vertical. Their AI/ML and data services integrated with our APIs provide a robust solution for our customers.”

Modern core banking solutions on the cleanest cloud in the industry

In addition to partnering with next-generation core banking platforms like these, Google Cloud has developed its own frameworks and partnership-driven solutions for advancing more secure, innovative, and data-driven banking practices. Some of these include:

An Apigee-based industry solution for powering embedded finance/open banking   

Anti-money laundering (AML) services that are powered by Google Cloud AI/ML 

A BigQuery-based regulatory reporting solution that streamlines and automates how financial and risk data are managed and configured to support reporting needs

The comprehensive Google Cloud financial ecosystem provides a one-stop shop for banks needing to modernize their technologies, but the benefits extend far beyond the seamless ease of procurement, management, and support on Google Cloud. (Though avoiding red tape when testing a new solution and streamlining how you manage your tech stack are no small things.) 

Storing data in Google Cloud means the assurance of highly resilient, globally scalable infrastructure. With hybrid and multi-cloud flexibility, your cloud strategy won’t be locked down. And because of Google Cloud’s commitment to innovation, your institution will keep pace in today’s open playing field with AI/ML helping to drive customer experience personalization, new product development, and operational excellence. Lastly, deploying on the cleanest cloud in the industry helps you to drive your sustainable finance and other ESG objectives.

Learn more about Google Cloud’s ecosystem of financial services partners and retail banking solutions.

Editor’s note: B4A, a Brazilian company that runs a beauty e-commerce platform, recently migrated their database to AlloyDB and saw performance accelerate by 90 percent across their queries. Now the company is looking for further gains in management and ease of use by migrating to other fully managed database solutions from Google.

B4A is a Brazilian beauty-tech startup founded in 2017 with the purpose of democratizing the

beauty market. Today, we offer monthly beauty subscription services to more than 120,000 paying subscribers, as well as, a completely customized online shopping experience for our loyalty club members that’s tailored to each customer’s unique profile. Our core technology platform, called B4A Connect is where we house our full catalog of products—roughly 10,000 different SKUs—and foster connections between our users, which include beauty brand representatives, consumers, and digital influencers. As adoption of B4A Connect grew, we struggled with load times for queries that returned several hundred SKUs at a time. At the time, we were using Microsoft SQL Server, and this made us start looking at alternative databases to run our platform and speed up performance. In 2022, we migrated the backend of B4A Connect to AlloyDB for PostgreSQL and saw catalog query performance accelerate by 90 percent.

Making a mark on the beauty industry

Just like most other industries, the beauty industry is undergoing its own digital transformation. We recognized this trajectory and founded B4A as a beauty tech company, born with digital DNA and meant to capitalize on digital channels to create lasting connections between brands and consumers. We offer female and male monthly subscription plans where B4A sends beauty products on a regular basis. Beyond subscriptions, our customers can also shop on our e-commerce platform whenever they wish. 

We knew the beauty e-commerce market was already very competitive, so we designed our shopping experience with integrated gamification strategies, opportunities for user-generated content creation, machine learning algorithms to offer product matches, and social selling strategies via influencers. As B4A’s popularity continues to grow, we’re constantly iterating on our platform to ensure we deliver the best possible customer experience. We’re also maturing as a startup, so finding ways to reduce costs is always on our minds. 

Migrating to AlloyDB for faster queries and lower costs

We were looking for an extremely performant database with a high service level to power B4A Connect. Not only does our B4A platform contain our full product catalog with more than 10,000 beauty products, descriptions, and reviews, it also customizes results for each user. With our previous database, we had some collections of products that would take up to 12 seconds to load. We knew we needed a database that could scale efficiently and would be easy to maintain. In addition, because our frontend applications are built on Angular, having a database solution from Google would be an easier integration. 

We opted for AlloyDB for performance reasons. And indeed, the results were amazing. We combined AlloyDB with a GraphQL API and had query times for our full catalog of products decrease by up to 90 percent compared to our previous database solution. In absolute numbers we went from 12 seconds to 1.2 seconds. And in production our load time is in the 0.25~0.4 second range. Because AlloyDB disaggregates compute and storage at every layer of the stack, it was able to scale seamlessly and offer predictable performance while running simultaneous queries.

But beyond performance, we also came to appreciate the ease of maintenance that a fully managed solution like AlloyDB provides. Setting up and configuring the database was a very smooth process that went really quickly for us. We also were thankful we didn’t have to worry about infrastructure, which enabled us to focus on our service right away. 

Compared to our previous database, we’ve been able to cut costs, too. We’re no longer stuck paying for traditional database licenses. With AlloyDB, pricing is transparent and predictable, and we’re only charged for what we use. 

Going all in on Google databases for easier management

Today, having everything as a fully managed solution on Google Cloud allows us to devote more time to product development and user experience, rather than managing infrastructure. In addition to using AlloyDB, we also run Firestore for various other use cases. One example is our monetization platform for micro-influencers, which runs on Firestore. This solution gives our micro-influencers the ability to sell our beauty products and subscriptions, including participating in advertising campaigns for beauty brands. As a fully managed, scalable, and serverless database, Firestore enables us to work with real-time change requests, which has been very beneficial for us. We love that we can go serverless and combine Firestore triggers with Cloud Functions. 

In a startup like ours resources are limited, so the ease of use and integration plus the lack of infrastructure requirements, allow for a much better overall result. Now, B4A developers can handle almost everything without support from the operations team.

Eventually, we plan to migrate all our databases to fully managed Google Cloud solutions, whether that’s AlloyDB, Cloud SQL for PostgreSQL, or Firestore. We want to break our monolithic application into small microservices using Cloud SQL and the corresponding Google databases that makes more sense for each service. All performance-critical queries with a direct user context will migrate to AlloyDB. We also plan to combine AlloyDB with our machine learning services to further improve the levels of customized search results for our users.

Learn more about the B4A and try out AlloyDB for PostgreSQL today.

The Places API provides you with business information, ratings, reviews and more for over 200 million businesses and places around the world. Starting today, we’re starting to roll out more attributes that reflect some of the most highly-requested information about places. These Place details and attributes play an important role in keeping existing customers informed, but also in helping convert customers from conducting a simple search to making an in-person visit. 

Display wheelchair accessible entrances

The Places API now includes wheelchair accessibility data, so your apps and websites can help users know if a destination has stair-free access and entrances that are at least three feet (one meter) wide. That’s helpful for the nearly 1-in-6 people with mobility challenges, and helps anyone with a stroller, luggage, delivery cart, or other need that might require a ramp.  

Wheelchair accessible entrance is a field available in the Basic category of Places API that is included in the base cost of Places requests at no additional charge.

Display special and secondary opening hours

Businesses sometimes adjust their opening hours for holidays or other reasons. Or they may have specific hours for different services at the same location–such as a drive-through that is open longer than the dining room, or a pharmacy that closes earlier than the retail store. Now, in addition to displaying regular open hours, you can account for schedule variations with these new parameters:

opening_hours provide regular hours of operation

current_opening_hours provide the place’s schedule for the next 7 days, including a sub-field “special days” to let you know if that day’s schedule is different from the place’s regular schedule. This makes it clear to customers that the special hours are accurate.

secondary_opening_hours provides the place’s secondary schedule for the next 7 days, and is structured similarly to the “current opening hours” field.

Place opening hours, including special hours and secondary opening hours are fields available in the Contact category in Places API. 

Display a short business description

The editorial summary is a one-line description that Google created to describe a place on Google Maps. Based on your feedback, we are making it available through the Places API. Now, you can display the same short summary about places in your apps and in your preferred language.

Editorial summary is available in the Atmosphere category in Places API.

Share relevant attributes about dining and shopping services

Helping your users choose a restaurant requires sharing quite a bit of information. Does a restaurant or shop offer take-out? Does it take reservations? Does it serve brunch? And so on. With the new available data fields for dining and shopping services, you’ll be able to provide more of the details your users need to make a decision:

curbside_pickup

delivery

dine_in

takeout

reservable

serves_breakfast

serves_lunch

serves_dinner 

serves_beer

serves_wine 

serves_brunch

serves_vegetarian_food

Restaurant and shopping attributes are available in the Atmosphere category in Places API.

Sort reviews in your preferred language

The Places API can provide user-written reviews about a place. Previously, developers couldn’t choose which reviews would get returned. Now, you can request the most relevant or most recent reviews using the reviews_sort parameter.  The Places API can also now return translated reviews in the request’s preferred language by default. You can opt out of receiving translations with the reviews_no_translations parameter. 

Review sorting and review translations are available in the Atmosphere category in Places API.

All these new fields are available in the Places API, and are being released in phases across our Places Library, Maps JavaScript API, Places SDK for Android, and Places SDK for iOS. We plan on introducing more attributes to the Places Details API so you can continue giving customers the information they need. And we want to hear from you. Let us know which features you want to see by requesting it here.

For more information on Google Maps Platform, visit our website.

Spanner is a fully managed, strongly consistent and highly available database providing up to 99.999% availability. It is also very easy to create your Spanner instance and point your application to it. But what if you want to migrate your schema and data from another database to Cloud Spanner? The common challenges with database migrations are ensuring high throughput of data transfer, and high availability of your application with minimal downtime,  and all this needs to be enabled with a user-friendly migrations solution. 

Today, we are excited to announce the launch of HarbourBridge 2.0 (Preview) – an easy to use open source migration tool, now with enhanced capabilities for schema and data migrations with minimal downtime.

This blog intends to demonstrate migration of schema and data for an application from MySQL to Spanner using HarbourBridge.

About HarbourBridge

HarbourBridge is an easy to use open source tool, which gives you highly detailed schema assessments and recommendations and allows you to perform migrations with minimal downtime. It just lets you point, click and trigger your schema and data migrations. It provides a unified interface for the migration wherein it gives users the flexibility to modify the generated spanner schema and run end to end migration from a single interface. It provides the capabilities of editing table details like columns, primary key, foreign key, indexes, etc and provides insights on the schema conversion performance along with highlighting important issues and suggestions.

What’s new in HarbourBridge 2.0?

With this recent launch, you can now do the following:

Perform end to end minimal downtime terabyte scale data migrations 

Get improved schema assessment and recommendations

Experience ease of access with gCloud Integration 

We’ll experience the power of some of these cool new add-ons as we walk through the various application migration scenarios in this blog.

Types of Migration

Data migration with HarbourBridge is of 2 types:

Minimal Downtime 

Migration with downtime

Minimal Downtime is for real time transactions and incremental updates in business critical applications to ensure there is business continuity and very  minimal interruption.Migration with downtime is recommended only for POC’s/ test environment setups or applications which can take a few hours of downtime.

Connecting HarbourBridge to source

There are three ways to connect HarbourBridge to your source database:

Direct connection to Database – for minimal downtime and continuous data migration for a certain time period

Data dump –  for a one time migration of the source database dump into Spanner 

Session file – to load from a previous HarbourBridge session

Migration components of HarbourBridge

With HarbourBridge you can choose to migrate:

Schema-only 

Data-only 

Both Schema and Data 

The below image shows how at a high level, the various components involved behind the scenes for data migration:

To manage a low-downtime migration, HarbourBridge orchestrates the following processes for you. You only have to set up connection profiles from the HarbourBridge UI on the migration page, everything else is handled by Harbour Bridge under the hood:

HarbourBridge sets up a Cloud Storage bucket to store incoming change events on the source database while the snapshot migration progresses

HarbourBridge sets up a datastream job to bulk load a snapshot of the data and stream incremental writes. 

HarbourBridge sets up the Dataflow job to migrate the change events into Spanner, which empties the Cloud Storage bucket over time

Validate that most of the data has been copied over to Spanner, and then stop writing to the source database so that the remaining change events can be applied. This results in a short downtime while Spanner catches up to the source database. Afterward, the application can be cut over to use Spanner as the main database.

The application

The use case we have created to discuss to demonstrate this migration is an application that streams in live (near real-time) T20 cricket match data ball-by-ball and calculates the Duckworth Lewis Target Score (also known as the Par Score) for Team 2, second innings, in case the match is disrupted mid-innings due to rain or other circumstances. This is calculated using the famous Duckworth Lewis Stern (DLS) algorithm and gets updated for every ball in the second innings; that way we will always know what the winning target is, in case the match gets interrupted and is not continued thereafter. There are several scenarios in Cricket that use the DLS algorithm for determining the target or winning score. 

MySQL Database

In this use case, we are using Cloud SQL for MySQL to house the ball by ball data being streamed-in. The DLS Target client application streams data into MySQL database tables, which will be migrated to Spanner. 

Application Migration Architecture

In this migration, our source data is being sent in bulk and in streaming modes to the MySQL table which is the source of the Migration. Cloud Functions Java function simulates the ball by ball streaming and calculates the Duckworth Lewis Target Score, updates it to the baseline table. HarbourBridge reads from MySQL and writes (Schema and Data) into Cloud Spanner. 

The below diagram represents the high level architectural overview of the migration process:

Note: In our case the streaming process is simulated with the data coming from a CSV into a landing table in MySQL which then streams match data by pushing row by row data to the baseline MySQL table. This is the table used for further updates and DLS Target calculations.

Migrating MySQL to Spanner with HarbourBridge

Set up HarbourBridge 

Run the following 2 gCloud commands on Google Cloud Console Cloud Shell:

Install the HarbourBridge component of gCloud by running:
gcloud components install HarbourBridge

Start the HarbourBridge UI by running:
gcloud alpha spanner migration web

Your HarbourBridge application should be up and running:

Note: 

Before proceeding with the migration, remember to enable the DataStream and Dataflow API from Google Cloud Console

Ensure you have Cloud SQL for MySQL or your own MySQL server created for the source and Spanner instance created for the target

Ensure all source database instance objects are created. For access to the DB DDLs, DMLs and the data CSV file refer to this git repo folder

For data validation (post-migration step) SELECT queries for both source and Spanner, refer to this git repo folder

Ensure Cloud Functions is created and deployed (for Streaming simulation and DLS Target score calculation). For the source code, refer to the git repo folder. You can learn how to deploy a Java function to Cloud Functions here

Also note that your proxy is set up and running when trying to connect to the source from HarbourBridge. If you are using Cloud SQL for MySQL, you can ensure that proxy is running by executing the following command in Cloud Shell:
./cloud_sql_proxy -instances=<<Project-id:Region:instance-name>>=tcp:<<3306>>

Connect to the source

Of the 3 modes of connecting to source, we will use the “Connect to database” method to get the connection established with source:

Provide the connection credentials and hit connect:

You are now connected to the source and HarbourBridge will land you on the next step of migration.

Schema Assessment and Configuration

At this point, you get to see both the source (MySQL) version of the schema and the target draft version of the “Configure Schema” page. The Target draft version is the workspace for all edits you can perform on the schema on  your destination database, that is, Cloud Spanner.

HarbourBridge provides you with comprehensive assessment results and recommendations for improving the schema structure and performance. 

As you can see in this image above, the icons to the left of table represent the complexity of table conversion changes as part of the schema migration

In this case, the STD_DLS_RESOURCE table requires high complexity conversion changes whereas the other ones require minimal complexity changes

The recommendation on the right provides information about the storage requirement of specific columns and there other warnings indicated with the columns list as well

You have the ability to make changes to the column types at this point 

Primary Key, Foreign Key, Interleaving tables, indexes and other dependencies related changes and suggestions are also available

Once changes are made to the schema, HarbourBridge gives you the ability to review the DDL and confirm changes

Once you confirm the schema changes are in effect before triggering the migration

Schema changes are saved successfully.

Prepare Migration

Click the “Prepare Migration” button on the top right corner of the HarbourBridge page.

1. Select Migration Mode as “Schema and Data”
2. Migration Type as “Minimal Downtime Migration”
3. Set up Target Cloud Spanner Instance

NOTE: HarbourBridge UI supports only Google SQL dialect as a Spanner destination today. Support for PostgreSQL dialect will be added soon.

4. Set up Source Connection profile

This is your connection to the MySQL data source. Ensure, you have the IP Addresses displayed on the screen allow-listed by your source.

5. Set up Target Connection profile

This is the connection to your Datastream job destination which is the Cloud Storage. Please select the instance and make sure you have allow-listed the necessary access.

Once done, hit Migrate at the bottom of the page and wait for the migration to start. HarbourBridge takes care of everything else, including setting up the Datastream and Dataflow jobs and executing them under the hood. You have the option to set this up on your own. But that is not necessary now with the latest launch of HarbourBridge.

Wait until you see the message “Schema migration completed successfully” on the same page. Once you see that, head over to your Spanner database to validate the newly created (migrated) schema.

Validate Schema and Initial Data

Connect to the Spanner instance, and head over to the database “cricket_db”. You should see the tables and rest of schema migrated over to the Spanner database:

Set up Streaming Data

As part of the setup, after the initial data is migrated, trigger the Cloud Functions job to kickstart data streaming into My SQL.

Validate Streaming Data

Check if the streaming data is eventually migrating into Spanner as the streaming happens.

The Cloud Functions (Java Function) can be triggered by hitting the HTTPS URL in the Trigger section of the function’s detail page. Once the streaming starts, you should see data flowing into MySQL and the Target DLS score for Innings 2 getting updated in the DLS table.

In the above image, you can see the record count go from 1705 to 1805 with the streaming. Also, the DLS Target field has a calculated value of 112 for the most recent ball.

Now let’s check if the Spanner database table got the updates in migration. Go to the Spanner table and query:

As you can see, Spanner has records increasing as part of migration as well. 

Also note the change in Target score field value ball after ball:

Wait until you see all the changes migrated over.

For data validation, you can use DVT (Data Validation Tool), which is a  standardized data validation method built by Google, and can be incorporated into existing GCP tools and technologies. In our use case, I validated the migration of the initial set of records from MySQL source to Spanner table using Cloud Spanner queries. 

End the Migration

When you complete all these validation steps, click End Migration. Follow the below steps to update your application to point to Spanner database:

Stop writes to the source database – This will initiate a period of downtime

Wait for any other incremental writes to Spanner to catch up with the source

Once you are sure source and Spanner are in sync, update the application to point to Spanner

Start your application with Spanner as the database

Perform smoke tests to ensure all scenarios are working

Cutover the traffic to your application with Spanner as the database

This marks the end of the downtime period

Clean Up 

Finally hit the “Clean Up” button on the End Migration popup screen. This will remove the migration jobs and dependencies that were created in the process.

Watch the migration in action

Next Steps

As you walked through this migration with us, you would have noticed how easy it is to point to your database, assess and modify your schema based on recommendations, and migrate your schema, your data, or both to Spanner with minimal downtime.

You can learn more about HarbourBridge on the README, and learn to install gCloud here. 

Get started today

Spanner’s unique architecture allows it to scale horizontally without compromising on the consistency guarantees that developers rely on in modern relational databases. Try out Spanner today for free for 90 days or for as low as $65 USD per month.

Back when Cloud Run was still in its public preview stage in 2019, I wrote about a dozen reasons why Cloud Run complies with the Twelve-Factor App methodology. That article described how Cloud Run covered most of the twelve factors of application development. However the twelfth factor — Admin Processes — was noted as “outside the scope of Cloud Run”. This gave Cloud Run a “near perfect score” of 11 out of 12. 

With Cloud Run jobs, we can now check off that last factor, as Cloud Run now has a way to run admin processes! You can use this pattern for things such as applying one-off commands, or more complex tasks like running database schema migrations for your deployments that rely on web frameworks (such as in Django, Rails, or Laravel). This article will focus on the database schema migration use case. 

While there are existing patterns to apply database migrations, they come with considerations and are possibly limited by your organization’s security policies. Using Cloud SQL Auth Proxy means you can apply migrations against your production database, but you’d be running those commands on your local machine. You can run migration commands as part of your Cloud Build processes, but there are going to be situations where you don’t want to automatically apply any database schema migrations when you deploy.

One job at a time

Cloud Run jobs allows you to run commands within the scope of Cloud Run itself. For Cloud Run services, your container has to start and have a process that is constantly running (like a web server).  With Cloud Run jobs you can invoke a command that will exit when complete. 

Cloud Run jobs have the same configuration as Cloud Run services, so you can configure secrets, databases, and service accounts for your job. This also means you can use the same container, and make use of existing IAM configurations from your service, for example, without having to reconfigure things for a new product.

How you approach this pattern will depend on if you have Dockerfile-based containers or Cloud Buildpack-based containers. 

Dockerfile-based database migrations

As a simple example, take an existing Cloud Run deployment command for a container built with a Dockerfile:

Usually, this service will run as is, with the default entrypoint of the container running your web server of choice. However, you can run something else, overriding the default entrypoint with –command. The only limitation with a Cloud Run service is that whatever you deploy has to have a process that listens for requests, 

With Cloud Run jobs, you don’t have this limitation, and so you can set whatever custom command you want, using the same command line arguments to ensure connectivity with your database and secrets. 

To adapt the previous example to a job, change deploy to jobs create, and specify your –command:

In this example, we’re using the –execute-now flag to create the job and immediately execute it. Any time we need to execute that command again, we can just re-run the job:

The power of Procfile

When using Dockerfiles, you have to specify the full command you want to run. However, if you’re using Google Cloud’s buildpacks, you can add entries in your Procfile, and then call these entrypoints in your –command.

For example, take the following Procfile:

By default, Google Cloud’s buildpacks run the web entrypoint. When using Cloud Run jobs with this configuration, you can use –command migrate to run migrations, or –command test to run tests. 

In the previous examples we’ve used general commands, but you can create a Procfile with entries based on your application. For example, depending on your web framework of choice:

You can still run any command without having to add definitions in the Procfile, as buildpacks come with a “launcher” entrypoint, the entrypoint for user-provided shell processes with Buildpacks. Use this entrypoint to run any command you like:

Migrations in practice

So how would we go about adapting this pattern to an existing deployment? 

Say I have a Cloud Run service that’s running a Django application. In Django, I use python3 manage.py migrate to apply database migrations.

So, I can add that command to my Procfile in my application code:

Then, I redeploy my Cloud Run service with this new code change using source deployments, which will build a new container and update my service:

From there, I can create my job. In this case, I want to use gcloud to get the configurations from my service, specifically the image name and Cloud SQL instance name. I could also copy these from the Cloud Console, but I like scripts.

Then, I can execute the job:

Using –wait, I can confirm the job runs successfully, then check the logs in the URL provided. 

Since I configured the job to run against the latest version of my container, if I want to re-run migrations, after I deploy my service, the new container will be available so I just execute the job again:

I don’t just have to run this from the command line, either. I could add this as a step in my Cloud Build deployment script after the deployment step!

The only time I would need to update this job is if something like my Cloud SQL instance changed:

This does presume, however, that I’m building my service in a way that I can deploy my service and then apply database migrations. At the moment, I’m building my container and deploying my service in one action (using source deployments), then running my job. If I need to make sure my database migrations run first, I’d need to build my container, run my job, then deploy my service. Read more about Cloud Schema Migrations.

A note on Procfiles

If you’re already using Google Cloud’s buildpacks and a language other than Python, Procfiles may be unfamiliar to you. In JavaScript, for example, there’s “one way” to start a service (npm run start, as defined in package.json.). When Google Cloud’s buildpacks detects a JavaScript application, it runs this by default if you don’t have a Procfile. 

In Python, though, there’s no “one command” to start things going, so Python users have to work with Procfiles. 

If you start using Procfiles, you’ll have to make sure you work out what your web command would be, and add that to your Procfile. (At the time of writing, if you include Procfile, you can’t rely on the default entrypoint functionality). You can still run ad-hoc commands, Procfile or no, just make sure you use –command launcher and use –args for your command. 

What else can you automate?

With this pattern, think about what other things you could do. 

Are you working with Firestore? Do you have backups? You can automate this by setting up a job to run gcloud firestore export $BUCKET. 

Do you have reports you want to run? You could set those up as well. You can also set up a schedule so they are run daily, or weekly. 

You can also create jobs to do maintenance tasks, like cleaning up records, or for admin tasks, like clearing caches.

Now a perfect scope as a matter of fact(or)

With the introduction of Cloud Run jobs, Cloud Run now scores a perfect 12/12 for all the twelve factors of application development, which means it is now the perfect choice for your next serverless deployment. 

To learn more about Cloud Run Jobs: 

Quickstart: Create and execute a job in Cloud Run 

Codelab: Getting started with Cloud Run jobs 

To learn more about buildpacks: 

Docs: Google Cloud’s buildpacks 

Editor’s note: This post is part of an ongoing series on IT predictions from Google Cloud experts. Check out the full list of our predictions on how IT will change in the coming years.

Prediction: By 2025, over half of cloud infrastructure decisions will be automated by AI and ML

Google’s infrastructure is designed with scale-out capabilities to support billions of people, powering services like Search, YouTube, and Gmail every day. To do that, we’ve had to pioneer global-scale computing and storage systems and shorten network latency and distance limitations with new innovations. Along the way, we’ve come to see cloud infrastructure as more than a simple commodity — it’s a source of inspiration and new capabilities.

But even as the demand on the industry’s cloud infrastructure continues to increase, there are simultaneously plateaus in the efficiency available from the underlying hardware. In the past, we saw annual performance gains of 30-40%, levels that often enabled a single infrastructure configuration to meet the needs of the vast majority of workloads. As these improvements have slowed and new workloads such as AI/ML and analytics have emerged, we have seen a corresponding explosion in the variety and capability of infrastructure. While empowering, the burden of picking the right combination of infrastructure components for a given workload still falls on an organization’s cloud architects. 

But by 2025, we predict that the burden and complexity of infrastructure decision making will disappear through the power of AI and ML automation, which will automatically combine purpose-built infrastructure, prescriptive architectures, and an ecosystem to deliver a workload-optimized, ultra-reliable infrastructure. The focus for cloud architects will therefore be on enabling business logic and innovation, rather than how that logic maps to underlying infrastructure.

Already, we are making investments to turn this vision into reality, building custom silicon like the Infrastructure Processing Unit (IPU) for our new C3 VMs or a liquid-cooled board for the new tensor processing unit. The latter, the TPU v4 platform, is likely the world’s fastest, largest, and most efficient machine learning supercomputer. It can train large-scale workloads up to 80% faster and 50% cheaper than alternatives. Put another way, TPU v4 will nearly double the performance of critical ML and AI services at half the cost, unlocking new possibilities for what organizations can achieve when leveraging large-scale learning and inference for business services.

These same IPUs and TPUs represent the foundation that will make it possible to automate cloud infrastructure decisions. They’ll be able to support the telemetry data and ML-based analytics for proactive infrastructure recommendations that will increase the performance and reliability of workloads. 

Instead of determining hardware specifications and building the right infrastructure, you’ll only need to specify a workload. AI and ML will take over the burden and recommend, configure, and identify the best options based on your budgetary, performance, and scaling requirements. What is most exciting for us is how this will enable a much more rapid pace of service innovation, which is the primary end goal of great cloud infrastructure.

Global life science supply chains are lengthy and complex with many moving parts. One small disruption can create serious delays and affect your ability to deliver therapeutics for patients. 

Supply chain disruptors

Over the last few years, healthcare organizations have encountered a range of obstacles, from both internal and external factors, that have resulted in supply networks failing to get drugs and medical devices to where they need to be on time. These obstacles include:

Labor and supply shortages 

Rising material costs 

Raw material constraints

Geo-political events

Unpredictable weather 

How do you overcome supply chain disruptors that are out of your control?

The intelligent healthcare supply chain

While many organizations have already implemented data-driven supply chains, organizations are still faced with the challenges of static, siloed, and different functional supply chain applications; limited data exchange with key trading partners across upstream and downstream operations; and the inability to effectively leverage relevant external data. 

At Google Cloud, we believe the key to meaningful and effective change is a data-driven supply chain that allows you to achieve visibility, flexibility, and innovation.

Our solutions help you prepare for the unpredictable and enhance the value of your data. By unlocking AI-driven insights, you can strengthen distribution networks and optimize your workflows and supply chains to become more reliable, intelligent, and sustainable. Some of the business challenges we address include:

Predicting demand with Vertex AI Forecast

Visual inspection for quality and predictive maintenance with pre-built ML models

Automating and optimizing pickup and delivery operations with Cloud Fleet Routing API

Real time and holistic inventory visibility with Supply Chain Twin

Make sure you’re prepared for the unpredictable with real-time visibility over your distribution networks. Learn how you can harness the power of AI and analytics and gain actionable insights that enhance your supply chain.

Editor’s Note: A native of Recife, Brazil, Patricia Florissi is a Technical Director in OCTO, Google Cloud’s Office of the Chief Technology Officer. OCTO is a global team of senior technologists with experience at large corporations, institutions, and successful startups, who work alongside customers to identify solutions for tough, but transformational challenges. 

How has your personal story led you to what you are doing today?

It’s not immediately evident how my personal story led me here, but I am very passionate about what I do at OCTO. My specialty is large computing systems, in particular, federated computations, a superset of federated learning and federated analytics. For me, federated computations are about orchestrating computing across a set of locations, computing on data in place, while preserving the privacy of each participating entity, and sharing only aggregated results with the orchestrator

How does that relate to growing up in Recife in the 1970’s? 

I believe that life creates the paths you follow. I also believe that good technology always reflects human experience. My parents were older when they had me, so they were worried they wouldn’t always be around to take care of me. Because of that, I was raised to be independent, mindful, curious and resourceful. I’m driven by my passion for family and learning, and am proud of the role I play in uniting generations. 

Nothing about my education was simple. The university I attended in Recife didn’t have a lot of resources, and only a few of us were able to study computer programming. In fact, I had to share a computing textbook with two other students—one of whom became my husband. We learned to program on one of the country’s only personal computers.

Maybe my passion for federation comes from a passion for combining the power of the individual with societal benefit. When people come together, they can achieve bigger, more complex, and more ambitious goals. I’ve seen how the contributions of few, can create something that couldn’t have been achieved otherwise.  

Why did you ultimately decide to emigrate to the U.S.?

It was kind of an accident actually. Growing up, everybody in Brazil who wanted to do graduate work in computer science went abroad—so when funding opened up, my husband and I enrolled at Columbia University. After graduating, we went back to Brazil where our twin daughters were born. But a year later, my husband got sick and sought treatment in the States. At that point, I started working for computer companies in the U.S. and joined Google in 2020.

What drew you to Google Cloud?

I love working at Google because the work OCTO does is very complex—complex technology, complex implications for businesses, and complex to execute. We have a great culture for challenging ourselves, learning, and creating by thinking outside the box. As soon as we understand something, we use that as the basis and starting point for learning something else. At OCTO, there is a healthy combination of curiosity and humility.

How is that reflected in your work with customers?

At OCTO, we work with customers to identify their problems and help tackle them. We don’t show up with the answers; we ask questions to understand how they operate, their vision, their challenges and the technology trends that matter most to them. This helps us identify solutions. There’s a lot of joy in the “a-ha!” moment, but it also takes a lot of time and commitment to get there. There are many ups and downs, and it demands a tremendous amount of resilience. But the work is extremely impactful.

How does your specialty—federated computations—impact the world?

Federated computation is a way to process data that is scattered all over the world, without moving that data to a central location. The data stays in place and is computed locally. 

A few examples come to mind, such as:

Big retailers who have data in computers across their stores generating images of their shelf that could be collectively used for training ML models  to predict “out of the shelf” events before they occur. Federated computations enable the collective analysis without moving data from the store, enhancing privacy while utilizing the compute already dispersed in the stores.

Healthcare providers who collect patient data that could be collectively analyzed but need to be protected because it’s personal information. By analyzing the data in place, federated computations enhance data protection; and 

Vehicles, which have become like mobile devices with wheels, collect telemetry data that could be collectively analyzed and used for training predictive maintenance models. Federated computations allow the analysis of the data in-vehicle, enhancing data privacy.

Federated computation is also a positive trend in regards to technology and societal benefit. The world may be headed to increased computing capacity and more data available for analysis. At the same time, the growing concerns with data privacy and data residency, among others, have been leading towards increasing data isolation. It’s an opposition that I’d love to unite and Google gives me the opportunity to work with companies to figure out how to address this. Every day, I wake up and think, “What haven’t I tried yet?,” then I get started.

We’re pleased to announce that Confidential Space, our new solution that allows you to control access to your sensitive data and securely collaborate in ways not previously possible, is now available in public Preview. First announced at Google Cloud Next, Confidential Space can offer many benefits to securely manage data from financial institutions, healthcare and pharmaceutical companies, and Web3 assets. Today, we will explore some security properties of the Confidential Space system that makes these solutions possible. 

Confidential Space uses a trusted execution environment (TEE), which allows data contributors to have control over how their data is used and which workloads are authorized to act on the data. An attestation process and hardened operating system image helps to protect the workload and the data that the workload processes from an untrusted operator.

The Confidential Space system has three core components:

The workload is a containerized image with a hardened OS that runs in a cloud-based TEE. You can use Confidential Computing as the TEE that offers hardware isolation and remote attestation capabilities.

The attestation service, which is an OpenID Connect (OIDC) token provider. This service verifies the attestation quotes for TEE and releases authentication tokens. The tokens contain identification attributes for the workload.

A managed cloud protected resource, such as a Cloud Key Management Service key or Cloud Storage bucket. The resource is protected by an allow policy that grants access to authorized federated identity tokens. 

The system can help ensure that access to protected resources is granted only to authorized workloads. Confidential Space also can help protect the workload from inspection and tampering, before and after attestation.

In our published Confidential Space Security Overview research paper, we explore several potential attack vectors against a Confidential Space system and how it can mitigate those threats. Notably, the research notes how Confidential Space can protect against malicious workload operators and administrators, and malicious outside adversaries, who are attempting to create rogue workload attestations.

Through these protections, Confidential Space establishes confidence that only the agreed upon workloads will be able to access sensitive data. The research also highlights some of the extensive security reviews and tests executed to identify potential weak points in the system, including domain expert reviews, meticulous security audits, and functional and fuzz testing.

We asked the NCC Group for an independent security assessment of Confidential Space to analyze its architecture and implementation. NCC Group leveraged their experience reviewing other Google Cloud products to dig deep into Confidential Space. 

The NCC Group’s  extensive review, which included penetration testing and automated security scanning, found zero security vulnerabilities. In their report, the architecture review highlights how the security properties are achieved through the coordination of measured boot with vTPM attestation, reduced attack surface with constricted administrator controls and access, workload measurement and enforced launch policy, and resource protection policy based on attested workload runtime properties.

The combination of these attributes creates powerful security properties, gating release of data on runtime measurements of the actual workload code and environment instead of just user and service account credentials. Confidential Space provides a platform that includes:

A dependable workload attestation, including workload code measurement, arguments and environment, and operating environment claims

A fully-managed attestation verification service that validates expected environmental attestation claims

A policy engine allowing for arbitrarily complex (or extremely simple) policy to be created around those claims

A mechanism to attach those policies to Google Cloud resources

Together, the platform provides a mechanism where one can ensure that their data is only ever released into trusted workloads that will not abuse that data.

Take a look at our documentation and codelab and take it for a spin. We hope that Confidential Space can inspire organizations to solve their use cases around multi-party collaboration with sensitive data; please contact your Google Cloud sales representative if you have any questions.