GCP – What’s new with Bigtable at Next ‘23?

It has been a busy few weeks for Cloud Bigtable, Google Cloud’s fully-managed, low-latency NoSQL database service. We had several big announcements leading to Google Cloud Next ’23 and even more during the event: from hybrid analytical and transactional processing (HTAP) and multi-cloud capabilities, to new cost savings and more ways to integrate with Google Cloud and the open-source ecosystem. Let’s go through some of the highlights:

Build event-driven applications with change streams

Now generally available, the new change streams feature allows you to track changes to your Bigtable data and easily integrate it with other systems. For example, you can replicate changes from Bigtable to BigQuery for analytics, to ElasticSearch for autocomplete and full-text search, or to other databases to support multi-cloud or hybrid-cloud architectures, trigger downstream actions using Pub/Sub and Cloud Functions, even integrate with Vertex AI to deliver ML-driven experiences.

With change streams, retailers can leverage change streams to monitor product catalog changes like pricing or availability to trigger in-app, text or email notifications to alert their customers. Meanwhile, banking apps can pass uploaded documents to Cloud Vision AI to parse their contents.

Save 20-40% on your Bigtable nodes with committed use discounts

Bigtable now provides deeply discounted prices in exchange for your commitment to continuously use Bigtable compute capacity (as measured in Bigtable nodes) for a one- or three-year period. A one-year commitment provides a 20% discount; a three-year commitment provides a 40% discount! Bigtable committed use discounts are spend-based and can be used flexibly across all Google Cloud regions and projects.

Go multi-cloud with bi-directional Bigtable-HBase replication

Multi-cloud has become an important part of the organizations’ IT strategies, however not every cloud database service is amenable to multi-cloud deployments because of the presence of proprietary APIs and data models. Luckily, Bigtable not only offers a compatible API for open-source Apache HBase, but also shares its common wide-column data model, which has historically made migrations between the two systems very easy. And with a new bi-directional replication capability, you can also use this compatibility to deliver multi-cloud or hybrid-cloud deployments. 

Hybrid transactional and analytical processing with request priorities 

As organizations continue to digitally transform, one of the biggest barriers they face is their inability to securely run batch and ad-hoc analytical workloads against live operational databases without risking disruptions. Database owners implement strict controls and limits on any such scenarios because these operations can consume a lot of resources, and if unchecked, can disrupt latency-sensitive serving workloads. Some teams work around these limitations by creating additional replicas, limiting batch writes to low-traffic hours or over-provisioning their databases — but this comes with significant cost and management overhead. Others try building complex pipelines to deliver data to analytical systems; these can be expensive, error-prone, and introduce data freshness and correctness issues. These limitations prevent many organizations from taking advantage of their data effectively, hindering data-driven innovation. 

This week at Google Cloud Next, we announced request priorities for Bigtable. Now, you can execute large workloads that are not time-sensitive, e.g., analytical queries and batch writes, as low priority jobs on a Bigtable cluster that’s also serving latency-sensitive queries as high priority jobs, significantly minimizing the impact of batch processing on the serving workloads. Request priorities are also supported through common Bigtable access paths for analytics such as BigQuery federation, Dataflow and Spark connectors, allowing analysts, data engineers and data scientists to work with operational data with ease, while giving admins the confidence that operations such as batch data loading or online model training will have minimal effect on operational performance.

You can sign up to try out request priorities here.

Export from BigQuery to Bigtable, no ETL tools required

Applications often need to serve analytics to their end users, whether it’s serving app analytics on a mobile app, or a machine learning model delivering millions of personalized ads every second. This pattern is sometimes referred to as “Reverse ETL.” However, getting this kind of data into operational databases involves ETL pipelines, a step that sometimes requires your teams to file tickets for help from your data engineers. We think there is a better way: Why not empower a developer or a data scientist to move data from their data warehouse into their operational databases in a self-service manner instead?

Many Google Cloud customers publish engagement metrics for their social media content or time-series data for IoT through dashboards backed by Bigtable. Then there are the data scientists building ML features in BigQuery, who often materialize their features into Bigtable to support the low-latency, high-throughput online feature store access patterns required by machine learning models.

We worked closely with the BigQuery team to build these export capabilities directly into BigQuery to take full advantage of the scalability of Google’s two “Big” databases (BigQuery and Bigtable) so that developers can easily export analytics necessary for their applications, while data scientists can materialize their features directly from the BigQuery console — all without having to touch any ETL tools. 

This feature is available to sign up for preview today.

Retain backups longer across multiple regions for additional resilience

Last but not least, you can now create a copy of a Cloud Bigtable backup and store it in any project or region where you have a Bigtable instance. You can also retain your backups for up to 90 days.

Tell us what you think

Are you as excited about these features as we are? Sign up for previews to try out the new features and let us know what you think:

Explore change streams 

Learn more about exporting data from BigQuery to Bigtable

Read more about request priorities