Today we hear from Freestar, whose ad monetization platform helps digital media publishers and app developers maximize ad revenue with scalable, efficient targeting opportunities. Founded in 2015 and reaching over 70% of U.S. internet users monthly, the company is now among the top 15 in U.S. internet reach. Vijaymohanan Pudukkudi, Vice President of Engineering – Architecture at Freestar, guides the company’s Ad Tech platform architecture. In this blog post, Vijay discusses Freestar’s decision to adapt to a multi-regional cluster configuration for its Memorystore deployment and leverage Envoy proxy to facilitate replication.

As Freestar has continued to grow, we support a global customer base that demands the lowest possible latencies, both from a customer experience and revenue standpoint. Specifically, we aim for ad-configuration delivery with a client-side latency under 80 ms at the 90th percentile, with server-side round trip time for requests under 20 ms at the 95th percentile.

Since Freestar expanded globally, we’ve faced challenges with latency. Originally, we served ad configurations from a single region in Google Cloud’s US-Central region. This resulted in high latency for customers accessing the configuration from outside the US. One solution was to use Google Cloud’s Content Delivery Network (CDN), but it was deemed unsuitable because of the dynamic nature of configurations. To meet our ambitious goals, we had to evolve our single-region architecture (see IMG A.1) to be multi-regional.

The problem

As our global footprint grew, our risk analysis pinpointed a significant technical concern related to our critical service delivery infrastructure. Prior to transitioning to a multi-regional design, Freestar’s infrastructure was concentrated in the US-central region, spread across multiple zones. It left us exposed to potential regional disruptions in Google Kubernetes Engine, which could have led to a complete service downtime, affecting all our customers.

We also rely on Google Cloud’s Memorystore for rapid configuration lookups. When we transitioned to a multi-regional setup, additional network overhead was introduced when reading configurations from the Memorystore cluster located in the US-central region from other regions. When accessing the service from regions outside the US-Central, we observed an additional latency of 100 to 240 ms at the 95th percentile, and the total service latency would have increased from our benchmark of 80 ms to approximately 300 ms. Although this performance might be satisfactory in many scenarios, it didn’t meet the requirements for our specific use case. You can view the related data on inter-regional/continental network latency on Google Cloud in the publicly available Looker Studio Report. To address the issue, we decided to distribute the Memorystore cluster regionally.

The challenges

Currently, Memorystore does not support out-of-the-box replication across multiple regions within Google Cloud. One solution we considered was building application-level code to actively call individual regional API endpoints and trigger refreshes of Memorystore keys. However, this approach would have introduced multiple points of failure and lead to inconsistent configurations across regions. This inconsistency would then necessitate a process to ensure customer Ad configurations were kept consistent across all regions and refreshed as needed. Furthermore, this option would require a significant amount of development effort. Thus, we sought a different approach.

The solution

Ultimately, based on this solution blog, we chose to go with Envoy proxy to facilitate replication. This strategy was essential in enabling high-throughput applications and multi-regional access patterns on the Memorystore infrastructure. Motivated by this approach, we devised a solution to replicate and keep configurations consistent across multi-regional Memorystore clusters.

First, we configured the Envoy proxy to propagate all the write operations to every regional cluster. This procedure is triggered whenever an event occurs in the primary Memorystore cluster in the US-central region. The Envoy proxy is configured as a sidecar container, running alongside the microservice that initiates the Memorystore write operations.

This approach required minimal developer time because the only change necessary in the application code was updating the Memorystore configuration to utilize the Envoy proxy port and IP address. In this case, we used localhost, as the Envoy proxy, and the application ran within the same GKE Pod. Going forward, this solution will also enhance our ability to scale multi-regional clusters without requiring any application code updates.

Envoy proxy sidecar configuration

Remote cluster configuration

This cluster configuration (refer IMG A.2) defines remote regional memorystore IP address and port details. Envoy uses it to interact with the actual Memorystore cluster from the virtual configuration.

Local proxy cluster configuration

Following configuration (IMG A.3) is the us-central virtual / proxy configuration within the pod where the sidecar is running so that it can propagate or mirror the writes commands.

Envoy proxy listener configuration

The Envoy Listener (IMG A.4) initiates a process based on triggers from the application, in this instance, the Config Dispensary. The application interfaces with Memorystore using a local host and port 6379. Following this interaction, the system employs the ‘us_central1’ remote configuration, directing the command towards the actual cluster.

Examining the configuration closely, you’ll notice a specific block configuration set to mirror all commands into another local proxy cluster. Additionally, there’s an exclusion configuration designed to prevent all read operations from being mirrored, thereby restricting such actions.

The image below (IMG A.5) shows the actual configuration settings. It demonstrates how the application from the us-central region (primary location) interacts with Memorystore through Envoy proxy for write operations.

The architecture diagram (IMG A.6) below shows how the regional Memorystore cluster and the microservices communicate. Except in ‘us-central’, microservices function without a proxy. In ‘us-central’, the proxy oversees read/write activities within the region and distributes writes throughout other regions.

The fully configured envoy.yaml and Dockerfile configuration for the Envoy Proxy are available on GitHub.

The architecture diagram (IMG A.7) below provides a comprehensive view of our microservice infrastructure. We leverage services such as Cloud Spanner, Cloud SQL, and Datastore to store the primary copy of our ad configuration in the US-central region. Any changes to this primary configuration instigate a Memorystore refresh in the US-central region. Subsequently, these updates propagate to other regional Memorystore clusters with the assistance of the Envoy proxy.

What comes after ‘Go’: Post-deployment analysis

By switching to a multi-regional Memorystore Redis setup with Envoy proxy replication from a single-region Redis cluster, we’ve significantly reduced latency, matching the performance of the previous single-region deployment.

Though write replication takes around 120ms at the 95th percentile to synchronize data across regions, it’s inconsequential due to our system’s read-heavy characteristic, making the latency of write operations negligible.

Since implementing this setup two months ago, we’ve had no replication or Envoy proxy issues. The application uses the US-Central region as its main region and uses Envoy for all Memorystore operations. Memorystore handles about 40K requests per second at peak hours with 5ms latency at 95 percentile. At peak periods, our Ad Config Service efficiently handles more than 15K API requests per second. Achieving a consistent 80ms latency on the client side (web browser) at the 90th percentile, we’ve successfully reached our targets following the expansion of our critical services across multiple regions.

Collaborating with the Google Cloud team, we’ve turned complex challenges into a robust solution, and the Google Cloud team’s meticulous validation of our system made us confident that it complies with industry standards. We now have a reliable solution for replicating Memorystore data globally, and consider the collaboration with Google Cloud a great success.

Editor’s note: This is the third of a three part series from the BCW Group, a Web3 venture studio and enterprise consulting firm serving enterprise clients who want to integrate existing products or otherwise branch into the Web3 space. Today’s blog discusses the frameworks used for creating one-click, end-to-end blockchain deployments into Google Cloud. You can read the first post here and the second post here.

Google Cloud’s BigQuery public datasets offer a window into the world of data-driven insights and analysis. These publicly available datasets offer a diverse array of domains, including economics, genomics, blockchain, and more. Researchers, analysts, and curious minds can penetrate datasets with the massive processing power of BigQuery. Whether unraveling complex trends in global trade, unlocking the mysteries of the human genome, or visualizing geographical patterns, the BigQuery public datasets empower users to find the story in the data.

BCW Technologies is working to create a framework that will help organizations and individuals access existing extract-transform-load (ETL) codebases and Google Cloud public datasets platform (PDP) so they can build a one-click, end-to-end, deployment into Google Cloud. We will be working primarily on the blockchain side of the PDP. These datasets can support many data strategies including academic inquiry, supplying historical chain data to users, or building out a comprehensive Big Data as a Service (BDaaS) platform.

Blockchains such as Ethereum or Bitcoin already generate large amounts of data. However, when discussing high-throughput chains such as Solana, building an ETL is not enough: developers must be cognizant of the entire pipeline. Since all states assist in validating all subsequent states, a comprehensive set with no trimmings is needed. Developers must consider all aspects of the extract-transform-load (ETL) process while calibrating their own deployments to keep up with these networks while designing a system that can be deployed and integrated by a wide array of participants.

The Blockchain ETL platform is designed to be a modular framework. It extracts blockchain data from a data source (such as an RPC node), then packages it using the efficient Protocol Buffers format before sending it to a data loader. This data loader first transforms the data to match the data storage schema before performing the insertion. In this case, the storage is Google Cloud BigQuery, which is well suited for the many terabytes of blockchain data. Given how much data is moving through modern blockchains, BQ’s large insertion support as well as scalable queries make it more than ideal for warehousing this data. Consequently, this makes the public dataset useful for use cases such as data science, comparison with other blockchains, or indexing historical on-chain actions.

The Solana blockchain has one of the shortest block times, thereby creating more frequent new blocks and blockchain state changes compared to other chains. This presents unique challenges. In this case, and pertaining to any blockchain network built on a rapid consensus mechanism, the data that is generated can quickly become unmanageable on the decentralized ledger where it is recorded.

BCWT seeks to build out an end-to-end, one-click, solution to the rapidly mounting level of data that highly performant chains generate. Authorizing configurations & submodules into an infrastructure-as-code deployment is important for the mission of assisting in democratizing the PDP data set (public is in the name afterall). By defining configurations that are compatible with both BCWT and community authored ETL, we seek to build out a solution that leverages the existing codebases, alongside our performance-oriented architecture.

The Solana ETL system consists of three primary parts with accompanying modules: a non-voting validator for keeping the state up-to-date, the transformer, and a Google Cloud BigQuery instance. The validator connects to the Solana network and follows up to three days worth of full state data (account snapshots included). Next, the transformer acts as a middleware between the data source and storage, and is deployed as a separate Compute Engine instance. Finally, Google Cloud BigQuery accepts the structured data.

BigQuery offers a tremendous amount of power for both querying and inserting data, but there is also the opportunity to be thoughtful in design to help control cost. By defining efficient partitioning routines and queries, selecting an appropriate egress point, and defining tables that map to your sources, controlling costs is possible. We are currently testing and authoring a collection of Solana-flavored repositories which can be seen here.

A highly performant programming language that can handle the immense data churn and scale must be used to compliment the aforementioned requirement of large-scale data in creating the ETL. In our case, we chose the highly performant systems language, Rust. This is also the implementation language of Solana, and offers speed, memory, and stability benefits resulting from its unique memory management, compilation optimizations, andzero-cost abstractions. Additionally, Rust’s support for conditional compilation has enabled us to design our codebase with modular blockchain support, such that we can add future blockchains in a plugin design. With each of these future blockchains, we plan to support the open source community with publicly available Rust packages to avail other developers direct library access.

One of our goals with Rust is to keep our code within community expectations to encourage open source collaboration. For example, we use well-known Rust libraries, including clap for our command-line interface, reqwest for making HTTP requests, Serde for deserializing JSON RPC responses, and Tokio to parallelize our extraction process. We also attempt to use functional programming as much as possible to accommodate data engineers interested in the project.

Upon deployment and initialization, the Solana ETL will first begin extracting data from Solana Labs’ instance of Google Bigtable (BT). It will also spin up a Solana Node and then begin working up from the genesis block, filling a designated BQ instance. Three data requests are necessary to extract data for the pipeline: (1) blocks, (2) accounts, and (3) token mints. The block data response includes all of the necessary data to populate the transactions, instructions, blocks, block rewards, and token transfers tables. However, Solana Labs’ BT instance only provides the block data, so an RPC node is required for the account and token data. Also important to note is the order of the data requests: account and token data is only requested when a transaction in the block creates an account or token mint, so the block data must be parsed first before these final requests can be made.

Initially, there will be a testing period, where Google Cloud will host a test dataset for selected Solana ecosystem participants and companies to engage with. Next, the overall apparatus will be deployed into the PDP organization and the dataset will be made accessible to all ecosystem participants. Finally, all code and deployment repositories will be added to the Blockchain-ETLrepositories for access by developers. These repositories will also include accompanying helmcharts, IAC configurations, and playbooks for rapid deployment into a GCP account.

While cloud technologies are often considered a centralizing force in decentralized technologies, it would be unrealistic for many to access the full scope of data that is generated by Solana. Archive nodes that handle both the full scope of all Solana blocks, accounts, and transactions are hard to find and access while including Geyser-enabled state snapshots is a very heavy lift for any one team or individual. By opening up access to a public data set in a highly performant infrastructure, Google Cloud helps to solve these challenges and make Web3 more accessible to traditional Web2 developers, companies and businesses that are all already using GCP every day. Google Cloud’s commitment to opening data access fosters innovation and discovery, inviting users worldwide to embark on exciting journeys of exploration within the boundless realm of data science.

If it isn’t already clear, Google Cloud is invested in being FinOps-first. At Google Cloud Next 2023, we made some pretty exciting FinOps product announcements for Google Cloud Billing. Here are the top five:: 

1. Google Cloud continues to pioneer the FinOps movement: Through our strong roots in the FinOps Foundation, we announced our contributions to v 0.5 of the FinOps open billing standards FOCUS (FinOps Open Cost & Usage Specification) project. V 0.5 provides a common language and set of definitions for data dimensions, metrics, and attributes regarding billing data. Datasets conforming to FOCUS will enable FinOps practitioners to perform common FinOps capabilities, using a generic set of instructions, regardless of the dataset origin. 

Why do we want to participate in a project designed to level the cloud billing data playing field across all cloud service providers (CSPs)? First, being open is a part of Google’s DNA, having contributed to foundational open source internet technologies over the past 20 years. Second, we believe that customers who innovate faster actually save money faster. We don’t want you spending time trying to decipher your cloud bills! We want you to leverage your cloud billing data to make better business decisions that fuel innovation, drive industry changes, and leapfrog your competition. Open, high-quality cloud cost data is non-negotiable. 

In support of good quality cloud cost data, Google Cloud Billing announced support for App Engine, Spanner, Firestore, and BigQuery granular, resource-level cost data that comes out of the box with theBilling detailed big query data export. 

To further FinOps industry adoption, we also launched a spotlight on FinOps thought leadership series on the Transform with Google Cloud thought leadership blog. In this monthly series, learn how FinOps has been adopted through real-life illustrative examples at work today from industry leaders.  

2. DuetAI is trained to a sample Certified FinOps Level. As part of Duet AI in Google Cloud, FinOps practitioners can also take advantage of AI integrated assistance for Billing based on models trained on Google Cloud best practices. Tailored for FinOps practitioners, you can use Duet AI to increase your Google Cloud billing proficiency. To measure Duet AI’s FinOps IQ, we partnered with the FinOps Foundation to put Duet AI to the FinOps test — literally: Duet AI passed a similar exam to the FinOps Certified Practitioner with flying colors. You can try it for yourself here. We have a robust roadmap for Google Cloud Billing has to unlock the power of generative AI that prioritizes accuracy first, but this is just the start. Look out for more innovations in this area. 

3. New FinOps Hub helps you mature your FinOps practice with smarter cost optimizations, new FinOps score, and peer benchmark: We’re focused on removing the cognitive load for cloud Finops practitioners . But when it comes to optimizing cloud spend, one of the hardest parts is knowing where to start. That is where the FinOps hub comes in.

We announced theFinOps Hub public previewat Next ‘23 to provide one pane of glass to identify, adopt, and measure all cost optimizations across an entire Google Cloud landscape. WIth advanced machine learning capabilities, the FinOps hub tells users what their FinOps score is — a tangible measure of how they are using Google Cloud billing data and tools to achieve maximum cloud value, with suggestions on how to improve. Peer benchmarks (with an opt-out option) provide context with respect to how users are performing with their FinOps practice and overall rate optimizations through committed use discounts (CUDs). 

Additionally Google Cloud cost optimizations have also gotten a lot smarter — they’re no longer just about rate optimizations like purchasing a CUD or rightsizing a VM; rather, we recognize that cost optimizations should be orchestrated across your entire cloud landscape, doing both rate and usage cost optimizations together through personalized prescriptions, always with the ability to personalize optimizations according to business needs. The FinOps Hub includes rate and usage optimizations for a variety of Google Cloud services. Finally, the FinOps Hub provides a realized savings ledger so cloud FinOps teams can track their return on investment (ROI) on all their cost optimizations in one place, easily.  

4. Budgets gets extended to engineers: We also announced a great new tool for project users to keep track of their cloud costs through the extension of the rich Cloud Billing budgets tooling. Project owners, Project Editors and Project Viewers can now create budgets and Project owners get notifications about actual or forecasted budget alerts in the Google Cloud console or through the Cloud Billing Budget API.

5. Google Cloud Billing is building for FinOps 2.0: We’re building towards a future of FinOps automation, as seen through our latest API enhancements to get a head start including the new Pricing API in public preview. The new Pricing API allows you to not only get list prices on all the available services but also account-specific custom prices and discounts, with visibility into any internal SKUs that are available to your billing account as part of your contract or agreement. The Pricing API also provides additional metadata such as SKU to service mapping, a list of SKU groups, SKUs to SKU group mapping, and a geo and product taxonomy that allows you to further slice and dice the SKUs for improved analysis. This API, along with others such as the Commitments API, provide the basis for customers to advance their own advanced cost management applications. Learn how SAP leveraged Cloud Billing APIs and data in their journey to controlling costs and building financial resilience with automation through their Google NEXT session here. 

To stay up-to-date on all things cloud FinOps and Google Cloud Billing, sign up for our Billing Advisory cohort at billing-cohort@googlegroups.com, where you can gain access to early features, participate in user feedback sessions, and learn about our roadmap with a group of like-minded cloud FinOps professionals.

You do not have to burden yourself anymore with a long list of manual steps required to setup an application locally, test with an in-memory database, sift through discrepancies between environments and SQL dialects or not being able to test network failures typical for a distributed system.

When building Twelve-factor Google Cloud applications, developers are looking for a consistent way to develop, debug and test locally with the same containerized application dependencies as they would run in production in GCP, the environment parity from Dev to Prod – factor Ten on the list of twelve-factor apps.

Google’s Java Cloud Client Libraries and Spring Cloud GCP make migrating, or building a new cloud-native, production-ready Spring Boot application a breeze, with developers able to quickly spin up the application and focus on building out the business logic.

In this blog post we’ll focus on significantly improving (and speeding up) your development experience for Google Cloud Spring Boot Java Applications, with a consistent way to develop, debug and test locally, following the goal of environment parity from Development-to-Production in Google Cloud.

Google Cloud Emulators, Testcontainers and Spring Boot to the rescue

Testcontainers is an open source framework for providing throwaway, lightweight instances of databases, message brokers, web browsers, or any other custom Docker container.  There is no more need for mocks or complicated environment configs. Testcontainers allow you to define your test dependencies as code, then simply run your tests and containers will be created and then deleted at the end of the test lifecycle.

Google’s Cloud SDK provides libraries and tools for interacting with the products and services provided by the Google Cloud Platform. Testcontainers modules are preconfigured containerized implementations of various dependencies that make writing your tests very easy.

Currently, Testcontainers Google Cloud Module supports Bigtable, Datastore, Firestore, Spanner, and Pub/Sub GCP Emulators in Java and other languages.

Spring Boot 3.1 has added Testcontainers support, allowing you to configure all application dependencies (databases, messaging systems, caching mechanisms, dependent services) to start automatically when you run or test the application locally.

If you are looking to learn more about how to test your Spring Boot applications with Testcontainers, follow this Getting started guide.

Development tasks

Before we dive into details, let’s revisit the some of the tasks you would usually address as a developer:

Write Unit Tests for individual pieces of functionality, say classes/methods

Write (Domain) Service Integration Tests to test domain operations and business rules, including the Integration with dependent applications or cloud services

Write Application Tests to functionally test your application and its web controllers 

Write Network Failure Tests to assess that the source code will handle any such failure, part of chaos engineering best practices

Local Development with app dependencies

Blog Source Code

This blog post is supported by a full set of Spring Boot sample apps, with tests highlighting the usage of Cloud Firestore and open-source Postgres (for Cloud SQL) Testcontainers.

Clone the Git repository and follow along in the Audit (Firestore) and Quotes (Postgres) services to gain hands-on experience with the concepts exposed herein. The following material will continue on the Firestore usage path.

Test configuration 

First, add the spring-boot-testcontainers as a test dependency to your Maven pom.xml:

Next, add the Google Client Libraries Bill-of-Materials(BOM) for compatibility across more than 200 Google Java libraries:

Last, add the dependency on the respective application dependent service:

To test with the testcontainer for the Firestore emulator, configure the @Container and provide its dynamic properties in the test class. 

Ex: Firestore Service Test [test]

Note that Spring Boot 3.1  has added the concept of Service Connections, which automatically configures the necessary Spring Boot properties for the supporting containers. 

A Postgres configuration would be reduced to:

Writing (Domain) Service/Integration Tests

Testing your service against a containerized Firestore emulator provides the confidence that the code will run smoothly in Cloud Firestore: [test]

Writing Application Tests

Writing application tests allows you to test your app as you would run it in a cloud environment in Prod. 

For example, your service is running in a Serverless environment and is a component of an event-driven architecture. CloudEvents is a specification for describing event data in a common way and you can now test your HTTP call with CloudEvents in the web controller, while leveraging Firestore testcontainers: [test]

Writing Network Failure tests

Network failure conditions between services running in the cloud present one of the highest risks in distributed software, since they directly affect service delivery. Before running your services in cloud environments, you want to test the resiliency of your service client code in your codebase for any such random disruptions, which can cause applications to respond unpredictably and break under pressure.

These tests are part of Chaos Engineering best practices.

To simulate network failure conditions, you can place Shopify’s Toxiproxy container in between test code and a container, or in between containers. [test]

You can now simulate timeouts due to latency, retries or dropped network connections. 

For example, you can test with a simulated timeout as follows:

Local Development with application dependencies

Starting with Spring Boot 3.1, you can also use Testcontainers during your regular dev time for running and debugging the application locally with your respective dependent services! 

There is no need to do any extra work, just configure a TestAuditApplication and TestcontainersConfig class in your test classpath under /src/test/java and start your app.

Note that the app is started with the TestcontainersConfig class attached to the application launcher using .with(TestcontainersConfig):

 You can run the TestAuditApplication from your IDE or simply with:

Summary

By using Google Cloud Emulators, Testcontainers and Spring Boot 3.1, you can significantly improve (and speed-up) your development experience for Google Cloud Applications, with a consistent way to develop, debug and test locally with the same containerized application dependencies as you would run in production in Google Cloud — the environment parity from Dev to Prod.

Check out the codebase, if you want to get more hands-on experience with Spring Boot Applications using Google Cloud Emulators and Testcontainers. 

For questions or feedback, feel free to contact me on Twitter @ddobrin.

Are you expanding your business to a new region using Cloud SQL? Do you need to move your applications that are running on Cloud SQL to a region that has specific data compliance or geographic locality requirements? Maybe you have experienced a regional outage, and you need to move your Cloud SQL instance into another region. Or perhaps you are an enterprise customer looking to migrate to the newly announced Cloud SQL Enterprise Plus edition for the 99.99% availability, near-zero downtime planned maintenance and 3x improved performance. In all these cases, you would first move your data first to a new region and operationalize applications to quickly start serving customers. This article will focus on reducing migration complexity, application downtime, and simplifying the migration process for Cloud SQL. 

This blog post entails steps for planning a regional migration with Cloud SQL:

Assess migration requirements

Migration approach

Application enablement

Assess migration requirements

Whenever a migration plan is considered, you will need to define application-specific integration requirements. These applications interact with backend databases and help define regional migration decisions, for example:

What is the size of the database?
The size of the database determines how long it takes to migrate it. You can control the time to a degree by choosing between an online approach (e.g., replicas) or an offline approach (e.g., export/import or backups) approach. Online approaches are ideal for applications that cannot afford an extended downtime period, whereas offline approaches are simpler to plan and execute as long as you can afford the downtime. 

Is there downtime tolerance?
Applications having Change Request schedules in the production environment make it easy to decide migration complexity. In cases, where downtime is acceptable for change deployment, consider a simple one-time migration approach rather than continuous migration. 

How is the Database connectivity configured in the application?
Cloud SQL regional migration may require updates to the database connection configuration within your applications. If your applications are already using Cloud NAT gateway, Cloud DNS, or a Load balancer etc. for connecting to Cloud SQL, then updating their configuration will be easy without impacting applications itself.

What is the hourly size of updates to your database?
Regional migration approach decisions are highly impacted by frequency of changes or transactions hitting your database every hour. Choosing a migration window with the lowest transaction frequency can help you avoid the risk of losing transactional data, high number of user connection failures, application timeouts, high error rates, etc. This makes the transaction frequency during the hour of migration a critical factor to consider. Identifying the time when your instance handles the lowest amount of traffic to the database can help you plan your migration more easily.

Do you have client side database encryption?
Cloud SQL offers capability to encrypt data that can be decrypted by applications. Encryption keys used for data encryption and decryption are stored in Cloud key management service (KMS). The usage of regional or global KMS keys for data encryption impacts the migration decision for Cloud SQL. In case of a single region KMS key, data might need to be re-encrypted depending on the migration approach used. For more information, refer Cloud SQL interaction with encryption keys. 

Answering above questions helps you plan the best migration path possible and to migrate Client applications to the new instance of Cloud SQL. This ensures that the client applications do not experience a split-brain state.

Region migration approach

For cross-region migration of Cloud SQL databases, there are various approaches available. We will discuss most popular approaches in the following section:

Cross-region read replicas for region migration:

A read-only replica contains an entire copy of the database and can serve reads without requiring a round trip back to the leader region. In doing so, read-only replicas deliver low latency stale reads to nearby clients and help increase a node’s overall read scalability to clients everywhere. Cross-region replication makes it easy to create a fully managed read replica in a different region than that of the primary instance.

The cross-region read replica is set up to asynchronously replicate the primary instance. This means that the read replica can lag behind the primary instance. If the read replica is not in sync with the primary instance during migration, then new data in the primary instance will be lost. To ensure replica is not lagging behind primary instance Monitoring metrics must be defined and tracked on the replica instance during migration hour.

When the whole region needs to be migrated due to region expansion or business requirement, this cross-region read replica can be promoted to become a standalone instance. During replica promotion scenarios, the Cloud SQL instance IP address may stay the same or change. 

With zonal read replicas, the Cloud SQL replica instance IP address will be the same as the primary instance. 

However for regional read replicas, the Cloud SQL replica instance IP address will be different from the primary instance. 

So, when you promote a Cloud SQL regional read replica, its IP address becomes the IP Address of the newly promoted primary instance. Since applications are pointing to IP Address of Cloud SQL before promotion (which was primary instance IP), a config update will be required to point to the IP address of newly promoted Cloud SQL instance (which was regional read replica IP). 

The complexity of repointing applications to the new Cloud SQL instances depends on how your application configuration is defined. Refer Cloud SQL lab to understand how to use Cloud DNS for config update.

Data migration service for new regional Cloud SQL instance:

Database Migration Service (DMS) can be used to migrate databases from an existing CloudSQL instance to a new one with the same or different Cloud SQL edition in a different region, and then perform an ongoing replication from the source database into the target. DMS supports CDC-style migrations for all sources (both homogenous and heterogenous migrations) which continually move changes from the source to the destination. See the PostgreSQL DMS known limitations.

Backup restore for new regional Cloud SQL instance:

Automated or on-demand backups stored on multi-regional buckets can be used to migrate Cloud SQL to another region. Setting up Cloud SQL instances from backup consumes some time as it involves creating a new instance and then restoring from the backup. 

Export and import database in new region:

Database exports can also be used for database migration to a different region. Critical considerations while choosing export/import databases are mostly around efforts, time, and data loss tolerance levels for an organization. For regional migration, one time export and import can be used in multi-regional Cloud storage buckets which can be accessed in the target region. 

To plan disaster recovery using export/import, the process should be automated to export databases regularly based on business requirements. For example, what are your SLA requirements? Are there any compliance requirements such as GDPR to be considered while exporting data? What should be the frequency of data export?

Below is a quick comparison between backup/restore and export/import:

Application enablement

Applications consuming data from Cloud SQL have a dedicated configuration for the database. The complexity of moving away from one configuration to another depends how the configuration is defined. Database connection endpoints are defined in the application runtime configuration. 

It is recommended to use aliases for database endpoints instead of using database IP addresses in the connection strings. This allows for database migration without impacting applications, as there is no need to change the connection string. You will only need to update the alias details of the migrated Cloud SQL database. 

Enterprise customers mostly use a private Cloud SQL instance unless there is any specific need for a public instance. For example, Cloud DNS can be used to resolve DNS records (FQDN) to the public or private IP address of Cloud SQL. The corresponding Cloud DNS name can be used in application configuration for pointing to the SQL host record. After migration, it will only require an update to the Cloud DNS record for the migrated Cloud SQL instance.

Learn more

Cloud SQL migration from one region to another region involves the analysis of database components impacted, application enablement and reconfiguration, network configuration, impact on user interfacing applications, migration of database monitoring, and many more activities. We discussed different approaches for Cloud SQL regional database migration using backups, read replicas, and offline exports. Using cross-regional read replicas is the most convenient approach among all, however the decision should be made based on business, cost and latency requirements. We hope that this blog has helped you better understand the differences and choose the configuration that is right for you.

To get started, create a Cloud SQL instance in your GCP account or in a SQL regional migration qwiklab to try the regional migration experience.

Want to learn more? Check out the resources below to dive in:

Cross-region replica for Cloud SQL

Scheduling Cloud SQL exports 

Database Migration Program

To empower developers to identify sensitive content in a rapidly changing media environment, we are excited to announce Text Moderationpowered by PaLM 2, available through the Cloud Natural Language API. Built in collaboration with Jigsaw and Google Research, Text Moderation helps organizations scan for sensitive or harmful content. Here are some examples of how the Text Moderation service can be used:

Brand Safety: Protect against user-generated content and publisher content that are considered not “brand safe” for the advertiser  

User protection: Scan for potentially offensive or harmful content

Generative AI risk mitigation: Help safeguard against the generation of inappropriate content in outputs from generative models

Promote brand safety

Brand safety is a set of procedures that aim to protect the reputation and trustworthiness of a brand in the digital age. One of the biggest risks to brand safety is the content that ads are associated with; if an ad appears on a website that contains content that does not conform with the sponsoring brand’s values, it can reflect poorly on the brand and organization, so it’s important for companies to identify and remove content that isn’t aligned with brand guidelines or consistent with the brand. 

Text Moderation can be used by our customers to identify content that they determine is offensive or harmful, sensitive in context, or otherwise inappropriate for their brand. Once an organization has identified this content, teams can take steps to remove it from advertising campaigns or prevent it from being associated with the brand in the future, helping ensure that advertising campaigns are effective and that the brand is associated with positive and trustworthy content.

Protect users from harmful content

Digital media platforms, gaming publishers, and online marketplaces all have a vested interest in mitigating the risks of user-generated content. They want to provide a safe and welcoming environment for their users while also maintaining an open and free exchange of ideas. Text Moderation can help them achieve this goal, using artificial neural networks to detect and remove harmful content, such as harassment or abuse. These efforts can help reduce harm, improve customer experience, and increase customer retention.

Mitigate risks of generative models

Over the last year, progress in AI has enabled software to more reliably generate text, images, and video, leading to new products and services that use machine learning, including text generators, to create content. However, with any AI content generation, there is a risk of producing offensive material, even inadvertently. 

To address this risk, we have trained and evaluated the Text Moderation service on real prompts and responses from large generative models. Text Moderation is versatile and covers a broad range of content types, making it a powerful tool for protecting users from harmful content.

Getting started with Text Moderation using the Natural Language API

Text Moderation is powered by Google’s latest PaLM 2 foundation model to identify a wide range of harmful content, including hate speech, bullying, and sexual harassment. Easy to use and integrate with existing systems, the API can be accessed from almost any programming language to return confidence scores across 16 different “safety attributes.”

Visit the Natural Language AIwebsite to give it a try and refer to the “Text Moderation” page for details. You may also try out the Text Moderation codelab here.

With the global energy crisis driving a dramatic spike in the price of fossil fuels, countries around the world have urgently stepped up their transition to renewable power to reduce their reliance on imported, carbon-intense fuel sources. According to the International Energy Agency (IEA), renewables will account for over 90% of global electricity expansion over the next five years, overtaking coal to become the largest source of global electricity by early 2025. 

However, while investment in renewables improves countries’ long-term energy security, the rapid transition presents energy-security challenges of its own. As more renewable energy sources and Distributed Energy Resource (DER) assets (such as rooftop solar panels and microturbines) are added to the grid, this can lead to network congestion and imbalanced energy distribution, affecting the quality of supply. 

A new business model for a changing industry

An industry leader in energy management solutions for more than 125 years, Landis+Gyr helps utility companies and grid operators to manage their energy infrastructure more efficiently. For Landis+Gyr, the answer to the challenges presented by a rapid transition to renewables lies in modernizing the traditional smart metering business model by using artificial intelligence (AI) and machine learning (ML) to make the most of the huge amount of data that its utility customers are collecting every day. 

However, Landis+Gyr initially found it difficult to innovate at the pace required by the rapidly changing energy industry, as its legacy on-premises analytics infrastructure lacked AI and ML capabilities. 

“We wanted to build a new product that enabled advanced analytics, including AI and ML capabilities, which we could offer as a service to our customers,” says Antonio Hidalgo, VP Digital Solutions at Landis+Gyr. “We also needed a solution that was easy to set up, scalable, and which required very little management from our side. Those were the key criteria for us and Google Cloud met them all.”

Google Cloud partnered with Landis+Gyr, making it our mission to help the company to innovate faster with a unified and intelligent data platform

Keeping the lights on with real-time data insights

Thanks to Landis+Gyr’s smart-metering infrastructure, utility companies all over the globe retrieve granular interval data for millions of devices every single day, creating a huge volume of data that needs to be uploaded and processed. Now, with the help of Cloud Storage, BigQuery, and Looker, Landis+Gyr can provide its utility customers with a comprehensive data lake and a complete analytics solution by using a cloud data warehouse, enabling them to make full use of their data to gain powerful insights in real time.

“BigQuery was super fast to set up and it scales seamlessly up and down, which makes the pricing structure very scale-friendly,” says Hidalgo. “Flexibility was also key for us. Our utility customers want to be able to define their own dashboards on a continuous basis, based on the information that is relevant to them. With Looker, we are now able to build and release something in a matter of days, not months. With our on-premises solutions, that was impossible. But on the cloud, with a modern CI/CD pipeline and the right level of automation, we can do it.”

Landis+Gyr is now able to provide its customers with data insights to manage all aspects of the power supply and distribution more effectively, from determining the quality of energy supply, through forecasting energy demand, to helping utilities maintain a distribution network with a growing volume of DER assets.

“Near real-time monitoring of sensor data streams enables our utility customers to predict many network issues in advance,” says Hidalgo. “By using Pub/Sub, we can ingest events and stream them in real time to BigQuery with Dataflow. Utilities can then analyze the network data to identify the problem and make decisions accordingly.”

Mapping the power grid with machine learning

Landis+Gyr is also making use of Google Cloud ML tools to improve the way utility companies maintain an accurate network connectivity model. Understanding the power system’s network topology, or how the various components such as generators, transformers and transmission lines are connected, is crucial for maintaining reliable and efficient grid operations, such as network planning, outage management and fault detection. This topology is traditionally maintained using Geographic Information Systems (GIS), which Hidalgo says are prone to inaccuracy. 

“On average, GIS systems typically have 5-20% inaccuracy in terms of where the end consumers are located in relation to the network,” Hidalgo notes. “With the introduction of new ML capabilities with Vertex AI, we can develop something far more accurate without the need to go to the field to confirm anything. A prediction with 97-98% accuracy is far more valuable from an operational point of view than the current GIS systems.”

Secure data for energy security

As data becomes increasingly important to utility companies, Landis+Gyr understands how critical it is to secure that data, and has been working with Google Cloud to apply cloud security best practices as a cornerstone of its new analytics solution. 

“Before partnering with Google Cloud, we made a careful assessment of the data protection safeguards and security compliance of the different cloud providers,” says Hidalgo. “Google Cloud offers us state-of-the-art data protection and security standards, which means that our customers are the only ones who are accessing their data.”

Future-proofing the industry for a greener tomorrow

The past two years of this partnership have built the foundations for future innovations, not only from a technology perspective, but also from an organizational and operational perspective. And as the transition to renewables continues to transform the energy industry as a whole, Hidalgo is convinced that Landis+Gyr can use ML and AI to transform the way the industry meets the ongoing challenges of this transition. 

“With Google Cloud, we are engineering a scalable, flexible solution that enables our customers to release new features quickly, to meet the fast-changing conditions we are now facing in the energy sector,” Hidalgo says. “And with AI and ML, we have the opportunity to create completely new insights with much more accuracy in terms of predictions, providing an entirely new way for the utility companies to operate the grid. ”

We all know how important the transition to renewables is. That’s why we are proud to partner with Landis+Gyr to lead the data transformation in the utilities sector and ensure that utilities can manage the security of supply with a powerful analytics solution.