Today, AWS announces that AWS Control Tower supports an additional 223 managed Config rules in Control Catalog for various use cases such as security, cost, durability, and operations. With this launch, you can now search, discover, enable and manage these additional rules directly from AWS Control Tower and govern more use cases for your multi-account environment.
To get started, in AWS Control Tower go to the Control Catalog and search for controls with the implementation filter AWS Config, you will then see all the AWS Config rules present in the Catalog. If you find rules that are relevant for you, you can then directly enable them from the AWS Control Tower console. You can also use ListControls, GetControl and EnableControl APIs. With this launch we’ve updated ListControls and GetControl APIs to support three new fields: Create Time, Severity and Implementation, that you can use when searching for a control in Control Catalog. For example, you can now programmatically find high severity Config rules which were created after your previous evaluation.
You can search the new AWS Config rules in all AWS Regions where AWS Control Tower is available, including AWS GovCloud (US). When you want to deploy a rule, reference the list of supported regions for that rule to see where it can be enabled. To learn more, visit the AWS Control Tower User Guide.
AWS announces the end of sale for AWS Elemental Link HD devices effective April 15, 2024. AWS Elemental Link UHD devices will continue to be available for purchase. To support HD content contribution workflows, Link UHD has now added HD ingest pricing, providing a seamless path for new deployments. Existing Link HD devices will continue to be supported, with Link UHD now serving as the recommended solution for both HD and UHD contribution workflows.
To enable HD pricing on Link UHD devices, you can configure the device’s input resolution on the Link device configuration page when the device is not actively streaming. The configuration option provides the flexibility to optimize costs when contributing HD content through Link UHD devices.
This feature is available immediately in all AWS Regions where Link UHD is supported. The input resolution configuration option is accessible through the AWS Management Console for all Link UHD devices.
Gateway Load Balancer (GWLB) now supports Load Balancer Capacity Unit (LCU) Reservation that allows you to proactively set a minimum bandwidth capacity for your load balancer, complementing its existing ability to auto-scale based on your traffic pattern.
Gateway Load Balancer helps you deploy, scale, and manage third-party virtual appliances. With this feature, you can reserve a guaranteed capacity for anticipated traffic surge. The LCU reservation is ideal for scenarios such as onboarding and migrating new workload to your GWLB gated services without the need to wait for organic scaling, or maintaining a minimum bandwidth capacity for your firewall applications to meet specific SLA or compliance requirements. When using this feature, you pay only for the reserved LCUs and any additional usage above the reservation. You can easily configure this feature through the ELB console or API.
The feature is available for GWLB in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Hong Kong), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm) AWS Regions. This feature is not supported on Gateway Load Balancer Endpoint (GWLBe). To learn more, please refer to the GWLB documentation.
Starting today, Amazon S3 Express One Zone has reduced pricing for storage by 31%, PUT requests by 55%, and GET requests by 85%. In addition, S3 Express One Zone has reduced its per-gigabyte data upload and retrieval charges by 60% and now applies these charges to all bytes rather than just portions of requests exceeding 512 kilobytes.
Amazon S3 Express One Zone is a high-performance, single-Availability Zone storage class purpose-built to deliver consistent single-digit millisecond data access for your most frequently accessed data and latency-sensitive applications, such as machine learning training, analytics for live streaming events, and market analysis for financial services.
Amazon Bedrock Knowledge Bases now extends support for hybrid search to knowledge bases created using Amazon Aurora PostgreSQL and MongoDB Atlas vector stores. This capability, which can improve relevance of the results, previously only worked with Opensearch Serverless and Opensearch Managed Clusters in Bedrock Knowledge Bases.
Retrieval augmented generation (RAG) applications use semantic search, based on vectors, to search unstructured text. These vectors are created using foundation models to capture contextual and linguistic meaning within data to answer human-like questions. Hybrid search merges semantic and full-text search methods, executing dual queries and combining results. This approach improves results relevance by retrieving documents that match conceptually from semantic search or that contain specific keywords found in full-text search. The wider search scope enhances result quality, particularly for keyword-based queries.
You can enable hybrid search through the Knowledge Base APIs or through the Bedrock console. In the console, you can select hybrid search as your preferred search option within Knowledge Bases, or choose the default search option to use semantic search only. Hybrid search with Aurora PostgreSQL is available in all AWS Regions where Bedrock Knowledge Bases is available, excluding Europe (Zurich) and GovCloud (US) Regions. Hybrid search with Mongo DB Atlas is available in the US West (Oregon) and US East (N. Virginia) AWS Regions. To learn more, refer to Bedrock Knowledge Bases documentation. To get started, visit the Amazon Bedrock console.
AWS Compute Optimizer now supports 57 additional Amazon Elastic Compute Cloud (Amazon EC2) instance types. The newly supported instance types include the latest generation accelerated computing instances (P5e, P5en, G6e), storage optimized instances (I7ie, I8g), and compute optimized instances (M8g), as well as high memory instances (U7i) and new instance sizes for C7i-flex and M7i-flex. With these newly supported instance types, AWS Compute Optimizer delivers recommendations to help you identify cost and performance optimization opportunities across a wider range of EC2 instance types, helping you improve performance and cost savings for your workloads.
This new feature is available in all AWS Regions where AWS Compute Optimizer is available, except the AWS GovCloud (US) and the China Regions. For more information about Compute Optimizer, visit our product page and documentation. You can start using AWS Compute Optimizer through the AWS Management Console, AWS Services CLI, or AWS SDK.
IAM Identity Center has released a new SDK plugin that simplifies AWS resource authorization for applications that authenticate with external identity providers (IdPs) such as Microsoft EntraID, Okta, and others. The plugin which supports trusted identity propagation (TIP), streamlines how external IdP tokens are exchanged for IAM Identity Center tokens. These tokens enable precise access control to AWS resources (e.g., Amazon S3 buckets) leveraging user and group memberships as defined in the external IdP.
The new SDK plugin automates the token exchange process eliminating the need for complex, custom-built workflows. Once configured, it seamlessly handles the IAM Identity Center token creation and the generation of user identity-aware credentials. These credentials can be used for creating identity-aware IAM role sessions while requesting access to different AWS resources. Currently available for Java 2.0 and JavaScript v3 SDK, this TIP plugin is AWS’s recommended solution for implementing user identity-aware authorization.
IAM Identity Center enables you to connect your existing source of workforce identities to AWS once, and access the personalized experiences offered by AWS applications such as Amazon Q, define and audit user identity-aware access to data in AWS services, and manage access to multiple AWS accounts from a central place. For instructions on installation of this plug-in, see here. For an example of how Amazon Q business developers can integrate into this plugin to build user identity-aware GenAI experiences, see here. This plugin is available at no additional cost in all AWS Regions where IAM Identity Center is supported.
Today, Amazon Web Services (AWS) announces the launch of two new EC2 I7ie bare metal instances. These instances are now available in US East (N. Virginia, Ohio), US West (Oregon), Europe (Frankfurt, London), and Asia Pacific (Tokyo) regions. The I7ie instances feature 5th generation Intel Xeon Scalable processors with a 3.2GHz all-core turbo frequency. Compared to I3en instances, they deliver 40% better compute performance and 20% better price performance. I7ie instances offer up to 120TB local NVMe storage density (highest in the cloud) for storage optimized instances. Powered by 3rd generation AWS Nitro SSDs, I7ie instances deliver up to 65% better real-time storage performance, up to 50% lower storage I/O latency, and 65% lower storage I/O latency variability compared to I3en instances.
EC2 bare metal instances provide direct access to the 5th generation Intel Xeon Scalable processor and memory resources. They allow EC2 customers to run applications that benefit from deep performance analysis tools, specialized workloads that require direct access to bare metal infrastructure, legacy workloads incompatible with virtual environments, and licensing-restricted business critical applications. These instances feature three Intel accelerator technologies: Intel Data Streaming accelerator (DSA), Intel In-Memory Analytics Accelerator (IAA), and Intel QuickAssist Technology (QAT). These accelerators optimize workload performance through efficient data operation offloading and acceleration.
I7ie instances offer metal-24xl and metal-48xl sizes with 96 and 192 vCPUs respectively and deliver up to 100Gbps of network bandwidth and 60Gbps of bandwidth for Amazon Elastic Block Store (EBS).
AWS Transfer Family announces new configuration options for SFTP connectors, providing you more flexibility and performance when connecting with remote SFTP servers. These enhancements include support for OpenSSH key format for authentication, ability to discover remote server’s host key for validating server identity, and ability to perform concurrent remote operations for improved transfer performance.
SFTP connectors provide a fully managed and low-code capability to copy files between remote SFTP servers and Amazon S3. You can now authenticate connections to remote servers using OpenSSH keys, in addition to the existing option of using PEM-formatted keys. Your connectors can now scan the remote servers for their public host keys that are used to validate the host identity, eliminating the need for manual retrieval of this information from server administrators. To improve transfer performance, connectors can now create up to five parallel connections with remote servers. These enhancements provide you greater control when connecting with remote SFTP servers to execute file operations.
The new configuration options for SFTP connectors are available in all AWS Regions where the Transfer Family is available. To learn more about SFTP connectors, visit the documentation. To get started with Transfer Family’s SFTP offerings, take the self-paced SFTP workshop.
Starting today, customers can use Amazon Managed Service for Apache Flink in the Mexico (Central) Region to build real-time stream processing applications.
Amazon Managed Service for Apache Flink makes it easier to transform and analyze streaming data in real time with Apache Flink. Apache Flink is an open source framework and engine for processing data streams. Amazon Managed Service for Apache Flink reduces the complexity of building and managing Apache Flink applications and integrates with Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Kinesis Data Streams, Amazon OpenSearch Service, Amazon DynamoDB streams, Amazon Simple Storage Service (Amazon S3), custom integrations, and more using built-in connectors.
You can learn more about Amazon Managed Service for Apache Flink here. For Amazon Managed Service for Apache Flink region availability, refer to the AWS Region Table.
Amazon Lex now allows you to disable automatic intent switching during slot elicitation using request attributes. This new capability gives you more control over conversation flows by preventing unintended switches between intents while gathering required information from users. The feature helps maintain focused conversations and reduces the likelihood of interrupting the process.
This enhancement is particularly valuable for complex conversational flows where completing the current interaction is crucial before allowing transitions to other intents. By setting certain attributes, you can ensure that your bot stays focused on collecting all necessary slots, or conformations for the current intent, even if the user’s utterance matches another intent with higher confidence. This helps create more predictable and controlled conversation experiences, especially in scenarios like multi-step form filling or sequential information gathering.
This feature is supported for all Lex supported languages and is available in all AWS Regions where Amazon Lex operates.
To learn more about controlling intent switching behavior, please reference the Lex V2 Developer Guide.
Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R6id instances are available in Europe (Spain) Region. These instances are powered by 3rd generation Intel Xeon Scalable Ice Lake processors with an all-core turbo frequency of 3.5 GHz and up to 7.6 TB of local NVMe-based SSD block-level storage. R6id instances are built on AWS Nitro System, a combination of dedicated hardware and lightweight hypervisor, which delivers practically all of the compute and memory resources of the host hardware to your instances for better overall performance and security. Customers can take advantage of access to high-speed, low-latency local storage to scale performance of applications such data logging, distributed web-scale in-memory caches, in-memory databases, and real-time big data analytics.
These instances are generally available today in the US East (Ohio, N.Virginia), US West (Oregon), Canada West (Calgary), Mexico (Central), Asia Pacific (Malaysia, Mumbai, Seoul, Singapore, Sydney, Thailand, Tokyo), Europe (Frankfurt, Ireland, London, Spain), Israel (Tel Aviv), and AWS GovCloud (US-West) Regions.
Customers can purchase the new instances via Savings Plans, Reserved, On-Demand, and Spot instances. To learn more, see Amazon R6id instances. To get started, visit AWS Command Line Interface (CLI), and AWS SDKs.
Starting today, Amazon Elastic Compute Cloud (Amazon EC2) M6id instances are available in US West (N. California) Region. These instances are powered by 3rd generation Intel Xeon Scalable Ice Lake processors with an all-core turbo frequency of 3.5 GHz and up to 7.6 TB of local NVMe-based SSD block-level storage.
M6id instances are built on AWS Nitro System, a combination of dedicated hardware and lightweight hypervisor, which delivers practically all of the compute and memory resources of the host hardware to your instances for better overall performance and security. Customers can take advantage of access to high-speed, low-latency local storage to scale performance of applications such data logging, distributed web-scale in-memory caches, in-memory databases, and real-time big data analytics.
These instances are generally available today in the US East (Ohio, N. Virginia), US West (Oregon, N. California), Canada West (Calgary), Canada (Central), Mexico (Central), South America (Sao Paulo), Asia Pacific (Tokyo, Sydney, Seoul, Singapore, Malaysia, Mumbai, Thailand), Europe (Zurich, Ireland, Frankfurt, London), Israel (Tel Aviv) Regions.
Customers can purchase the new instances via Savings Plans, Reserved, On-Demand, and Spot instances. To get started, visit AWS Command Line Interface (CLI), and AWS SDKs. To learn more, visit our product page for M6id.
AWS CodeBuild now supports Node 22, Python 3.13, Go 1.24 and Ruby 3.4 in Lambda Compute. These new runtime versions are available in both x86_64 and aarch64 architectures. AWS CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages ready for deployment.
The new Lambda Compute runtime versions are available in US East (N. Virginia), US East (Ohio), US West (Oregon), South America (São Paulo), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Mumbai), Europe (Ireland), and Europe (Frankfurt).
To learn more about runtime versions provided by CodeBuild, please visit our documentation. To learn more about CodeBuild’s Lambda Compute mode, see CodeBuild’s documentation for Running builds on Lambda.
Amazon Relational Database Service (Amazon RDS) for SQL Server now supports new minor versions for SQL Server 2019 (CU32 – 15.0.4430.1) and SQL Server 2022 (CU18 – 16.0.4185.3). These minor versions include performance improvements and bug fixes, and are available for SQL Server Express, Web, Standard, and Enterprise editions. Review the Microsoft release notes for CU32 and CU18 for details.
We recommend that you upgrade to the latest minor versions to benefit from the performance improvements and bug fixes. You can upgrade with just a few clicks in the Amazon RDS Management Console or by using the AWS SDK or CLI. Learn more about upgrading your database instances from the Amazon RDS User Guide.
These minor versions are available in all AWS regions where Amazon RDS for SQL Server is available. See Amazon RDS for SQL Server Pricing for pricing details and regional availability.
At Google Cloud Next, we introduced H4D VMs, our latest machine type for high performance computing (HPC). Building upon existing HPC offerings, H4D VMs are designed to address the evolving needs of demanding workloads in industries such as manufacturing, weather forecasting, EDA, and healthcare and life sciences.
H4D VMs are powered by the 5th Generation AMD EPYCTM Processors, offering improved whole-node VM performance of more than 12,000 gflopsand improved memory bandwidth of more than 950 GB/s. H4D provides low-latency and 200 Gbps network bandwidth using Cloud Remote Direct Memory Access (RDMA) on Titanium, the first of our CPU-based VMs to do so.This powerful combination enables you to efficiently scale your HPC workloads and achieve insights faster.
VM and core performance, as well as memory bandwidth for H4D vs. C2D and C3D, showing generational improvement
For open-source High-Performance Linpack (OSS-HPL), a widely-used benchmark for measuring the floating-point computing power of supercomputers, H4D offers 1.8x higher performance per VM and 1.6x higher performance per core compared to C3D. Additionally, H4D offers 5.8x higher performance per VM and 1.7x higher performance per core compared to C2D.
For STREAM Triad, a benchmark to measure memory bandwidth, H4D offers 1.3x higher performance per VM and 1.4x higher performance per core compared to C3D. Additionally, H4D offers 3x higher performance per VM and 1.4x higher performance per core compared to C2D.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud infrastructure’), (‘body’, <wagtail.rich_text.RichText object at 0x3ece6d2a1f70>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/compute’), (‘image’, None)])]>
Improved HPC application performance
H4D VMs deliver strong compute performance and memory bandwidth, significantly outperforming previous generations of AMD-based VMs like C2D and C3D, allowing for faster simulations and analysis, and delivering significant performance gains (relative to a prior generation AMD-based HPC VM, C2D) across various HPC applications and benchmarks, as illustrated below:
Manufacturing
CFD apps like SiemensTM Simcenter STAR-CCM+TM/HIMach show up to 3.6x improvement.
CFD apps like Ansys Fluent/f1_racecar_140 show up to 3.6x improvement.
FEA Explicit apps like Altair Radioss/T10m show up to 3.6x improvement.
CFD apps like OpenFoam/Motorbike_20m show up to 2.9x improvement.
FEA Implicit apps like Ansys Mechanical/gearbox shows up to 2.7x improvement.
Healthcare and life sciences:
Molecular Dynamics (GROMACS) shows up to 5x improvement.
Weather forecasting
Industry standard benchmark WRFv4 shows up to 3.6x improvement.
Figure 2: Single VM HPC Application performance (speed-up) of H4D, C3D and C2D relative to C2D. Applications ran on single VMs using all cores.
“Our deep collaboration with Google Cloud powers the next generation of cloud-based HPC with the announcement of the new H4D VMs. Google Cloud has leveraged the architectural advances of our 5th Gen AMD EPYC CPUs to create an offering that delivers impressive performance uplift compared to previous generations across a variety of HPC benchmarks. This will empower customers to achieve fast insights and accelerate their most demanding HPC workloads.” – Ram Peddibhotla, corporate vice president, Cloud Business, AMD
Faster HPC with Cloud RDMA on Titanium
H4D’s performance is made possible with Cloud RDMA, a new Titanium offload that’s available for the first time on these VMs. Cloud RDMA is specifically engineered to support HPC workloads that rely heavily on inter-node communication, such as computational fluid dynamics, weather modeling, molecular dynamics, and more. By offloading network processing, Cloud RDMA provides predictable, low-latency, high-bandwidth communication between compute nodes, thus minimizing host CPU bottlenecks.
Under the hood, Cloud RDMA uses Google’s innovative Falcon hardware transport for reliable, low-latency communication over our Ethernet-based data center networks, effectively resolving the traditional challenges of RDMA over Ethernet while helping to ensure predictable, high performance at scale.
Cloud RDMA over Falcon speeds up simulations by efficiently utilizing more computational resources. For example, for smaller CFD problems like OpenFoam/motorbike_20m and Simcenter Star-CCM+/HIMach10, which have limited inherent parallelism and are typically challenging to accelerate, H4D results in 3.4x and 1.9x speedup, respectively, on four VMs compared to TCP.
Figure 3: Left: OpenFoam/Motorbike_20m offers a 3.4x improvement with H4D Cloud RDMA over TCP at four VMs. Right: Simcenter STAR-CCM+/HIMach10 offers a 1.9x improvement with H4D Cloud RDMA over TCP at four VMs.
For larger models, Falcon also helps maintain strong scaling. Using 32 VMs, Falcon achieved a 2.8x speedup over TCP for GROMACS/Lignocellulose and a 1.3x speedup for WRFv4/Conus 2.5km.
Figure 4: Left: GROMACS/Lignocellulose offers a 2.8x improvement with H4D Cloud RDMA over TCP at 32 VMs. Right: WRFv4/Conus 2.5km offers a 1.3x improvement with H4D Cloud RDMA over TCP at 32 VMs.
Cluster management and scheduling capabilities
H4D VMs will support both Dynamic Workload Scheduler (DWS) and Cluster Director (formerly known as Hypercompute Cluster).
DWS helps schedule HPC workloads for optimal performance and cost-effectiveness, providing resource availability for time-sensitive simulations and flexible HPC jobs.
Cluster Director, which lets you deploy and scale a large, physically-colocated accelerator cluster as a single unit, is now extending its capabilities to HPC environments. Cluster Director simplifies deploying and managing complex HPC clusters on H4D VMs by allowing researchers to easily set up and run large-scale simulations.
VM sizes and regional availability
We offer H4D VMs in both standard and high-memory configurations to cater to diverse workload requirements. We also provide options with local SSD for workloads that demand high-speed storage, such as CPU-based seismic processing and structural mechanics applications (e.g., Abaqus, NASTRAN, Altair OptiStruct and Ansys Mechanical).
VM
Cores
Memory
Local SSD
h4d-highmem-192-lssd
192
1488
3.75TB
h4d-standard-192
192
720
N/A
h4d-highmem-192
192
1488
N/A
H4D VMs are currently available in us-central1-a (Iowa), and europe-west4-b (Netherlands), with additional regions in progress.
What our customers and partners are saying
“With the power of Google’s new H4D-based clusters, we are poised to simulate systems approaching a trillion particles, unlocking unprecedented insights into circulatory functions and diseases. This leap in computational capability will dramatically accelerate our pursuit of breakthrough therapeutics, bringing us closer to effective precision therapies for blood vessel damage in heart disease.” – Petros Koumoutsakos, Jr. Professor of Computing in Science and Engineering, Harvard University
“The launch of Google Cloud’s H4D platform marks a significant advancement in engineering simulation. As GCP’s first VM with RDMA over Ethernet, combined with higher memory bandwidth, generous L3 cache, and AVX-512 instruction support, H4D delivers up to 3.6x better performance for Ansys Fluent simulations compared to C2D VMs. This performance boost allows our customers to run simulations faster, explore a wider range of design options, and drive innovation with greater efficiency.” – Wim Slagter, Senior Director of Partner Programs, Ansys
“The generational performance leap achieved with Google H4D VMs, powered by the 5th Generation AMD EPYC™, is truly remarkable. For compute-intensive, highly non-linear simulations, such as car crash analysis, Altair® Radioss® delivers a stunning 3.6x speedup. This breakthrough paves the way for faster and more accurate simulations, which is crucial for our customers in the era of the digital thread!” – Eric Lequiniou, SVP Radioss Development and Altair Solvers HPC
“The latest H4D VMs, powered by 5th Generation AMD EPYC Processors and Cloud RDMA, allow our customers to realize faster time-to-results for their Simcenter STAR-CCM+ simulations. For HIMach10, we’re seeing up to 3.6x performance gains compared to the C2D instance and 1.9x speedup on four H4D Cloud RDMA VMs compared to TCP. Our partnership with Google has been key to achieving these reduced simulation times.”– Lisa Mesaros, Vice President, Simcenter Solution Domains Product Management, Siemens
Want to try it out?
We’re excited to see how H4D VMs will empower you to achieve faster results with your HPC workloads! Sign up for the preview by filling out thisform.
For decades, businesses have wrestled with unlocking the true potential of their data for real-time operations. Bigtable, Google Cloud’s pioneering NoSQL database, has been the engine behind massive-scale, low-latency applications that operate at a global scale. It was purpose-built for the challenges faced in real-time applications, and remains a key piece of Google infrastructure, including YouTube and Ads.
This week at Google Cloud Next, we announced continuous materialized views, an expansion of Bigtable’ SQL capabilities. Bigtable SQL and continuous materialized views enable users to build fully-managed, real-time application backends using familiar SQL syntax, including specialized features that preserve Bigtable’s flexible schema — a vital aspect of real-time applications.
Whether you’re building streaming applications, real-time aggregations, or global AI analysis on a continuous data stream, Bigtable just got a whole lot easier — and much more powerful.
Bigtable’s SQL interface, now generally available
Bigtable recently transformed the developer experience by adding SQL support, now generally available. SQL support makes it easier for development teams to work with Bigtable’s flexibility and speed.
Bigtable SQL interface in Bigtable Studio
The Bigtable SQL interface enhances accessibility and streamlines application development by facilitating rapid troubleshooting and data analysis. This unlocks new use cases, like real-time dashboards utilizing distributed counting for instant metric retrieval and improved product search through K nearest neighbors (KNN) similarity search. A wide range of customers, spanning innovative AI startups to traditional financial institutions, are enthusiastic about Bigtable SQL’s potential to broaden developer access to Bigtable’s capabilities.
“Imagine coding with AI that understands your entire codebase. That’s Augment Code, an AI coding platform that gives you context in every feature. Bigtable’s robustness and scaling enable us to work with large code repositories. Its ease of use allowed us to build security features that safeguard our customers’ valuable intellectual property. As our engineering team grows, Bigtable SQL will make it easier to onboard new engineers who can immediately start to work with Bigtable’s fast access to structured, semi-structured, or unstructured data while using a familiar SQL interface”saidIgorOstrovsky, cofounder and CTO, Augment.
“Equifax leverages Bigtable within our proprietary data fabric for the high-performance storage of financial journals. Our data pipeline team evaluated Bigtable’s SQL interface and found it to be a valuable tool for directly accessing our enterprise data assets and improved Bigtable’s ease of use for SQL-experienced teams. This means more of our team can work efficiently with Bigtable and we anticipate boosted productivity and better integration capabilities,” said Varadarajan Elangadu Raghunathan and Lakshmi Narayanan Veena Subramaniyam, vice-presidents, Data Fabric Decision Science.
Bigtable SQL has also been praised for offering a smooth migration path from databases with distributed key-value architectures and SQL-based query languages, including Cassandra (CQL) and HBase with Apache Phoenix.
“At Pega, we are building real-time decisioning applications that require very low latency query responses to make sure our clients get real-time data to drive their business. The new SQL interface in Bigtable is a compelling option for us as we look for alternatives to our existing database,” said Arjen van der Broek, principal product manager, Data and Integrations, Pega.
This week, Bigtable is also adding new preview functionalities to its SQL language including GROUP BYs and aggregations, an UNPACK transform for working with timestamped data, and structured row keys for working with data that is stored in a multi-part row key.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud databases’), (‘body’, <wagtail.rich_text.RichText object at 0x3ece6c19bf70>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/products?#databases’), (‘image’, None)])]>
Continuous materialized views, now in preview
Bigtable SQL integrates with Bigtable’s recently introduced continuous materialized views (preview), offering a solution to traditional materialized view limitations like data staleness and maintenance complexity. This allows for real-time aggregation and analysis of data streams across applications such as media streaming, e-commerce, advertising, social media, and industrial monitoring.
Bigtable materialized views are fully managed and make updates incrementally without impacting your user queries from applications. Bigtable materialized views also support a rich SQL language including functions and aggregations.
“With Bigtable’s new Materialized Views, we’ve unleashed the full potential of low-latency use cases for clients of our Customer Data Platform. By defining SQL-based aggregations/transformations at ingestion, we’ve eliminated the complexities and delays of ETL in our time series use cases. Moreover, using data transformations during ingestion, we’ve unlocked the ability for our AI applications to receive perfectly prepared data with minimal latencies,” said Sathish KS, Chief Technology Officer, Zeotap.
Continuous Materialized Views workflow
Ecosystem integrations
To get useful real-time analytics, you often need to pull data from many sources and do so with very low latency. As Bigtable expands its SQL interface, it is also expanding its ecosystem compatibility making it easier to build end to end applications using simple connectors and SQL.
Open-source Apache Kafka Bigtable Sink Customers often rely on Google Cloud Managed Service for Apache Kafka to build pipelines that stream data into Bigtable and other analytics systems. To help customers build high-performance data pipelines, the Bigtable team has open-sourced a new Bigtable Sink for Apache Kafka so you can send data from Kafka to Bigtable in milliseconds.
Open-source Apache Flink Connector for Bigtable Apache Flink is a stream-processing framework that lets you manipulate data in real time. With the recently launched Apache Flink to Bigtable Connector, you can construct a pipeline that lets you transform streaming data and write the outputs into Bigtable using both the high-level Apache Flink Table API and the more granular Datastream API.
“BigQuery continuous queries enables our application to use real-time stream processing and ML predictions by simply writing a SQL statement. It’s a great service that allows us to launch products quickly and easily,” said Shuntaro Kasai and Ryo Ueda, MLOps Engineers, DMM.com.
Real-time Analytics in Bigtable overview
Bigtable CQL Client: Bigtable is now in preview and Cassandra-compatible
The Cassandra Query Language (CQL) is the query language of Apache Cassandra. With the launch of Bigtable CQL Client, developers can now migrate their applications to Bigtable with minimal to no code change, and enjoy the familiarity of CQL on enterprise-grade, high-performance Bigtable. Bigtable also supports common tools in the Cassandra ecosystem like the CQL shell (CQLsh), as well as Cassandra’s own data migration utilities which enable seamless migrations from Cassandra, with no downtime significantly reducing operational overhead.
Get started using the Bigtable CQL Client and migration utilities here.
Convergence: NoSQL’s embrace of SQL power
In this blog, we discussed a significant advancement that empowers developers to use SQL with Bigtable. You can easily get started with the flexible SQL language from any existing Bigtable cluster using Bigtable Studio and start to create materialized views on streams of data coming from Kafka and Flink.
As an object storage service, Google Cloud Storage is popular for its simplicity and scale, a big part of which is due to the stateless REST protocols that you can use to read and write data. But with the rise of AI and as more customers look to run data-intensive workloads, two major obstacles to using object storage are its higher latency and lack of file-oriented semantics. With the launch of Rapid Storage on Google Cloud, we’ve added a stateful gRPC-based streaming protocol that provides sub-millisecond read/write latency and the ability to easily append data to an object, while maintaining the high aggregate throughput and scale of object storage. In this post, we’ll share an architectural perspective into how and why we went with this approach, and the new types of workloads it unlocks.
It all comes back to Colossus, Google’s internal zonal cluster-level file system that underpins most (if not all) of our products. As we discussed in a recent blog post, Colossus supports our most demanding performance-focused products with sophisticated SSD placement techniques that deliver low latency and massive scale.
Another key ingredient in Colossus’s performance is its stateful protocol — and with Rapid Storage, we’re bringing the power of the Colossus stateful protocol directly to Google Cloud customers.
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3ece8b51b9d0>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
When a Colossus client creates or reads a file, the client first opens the file and gets a handle, a collection of state that includes all the information about how that file is stored, including which disks the file’s data is stored on. Clients can use this handle when reading or writing to talk directly to the disks via an optimized RDMA-like network protocol, as we previously outlined in our Snap networking system paper.
Handles can also be used to support ultra-low latency durable appends, which is extremely useful for demanding database and streaming analytics applications. For example, Spanner and Bigtable both write transactions to a log file that requires durable storage and that is on the critical path for database mutations. Similarly, BigQuery supports streaming to a table while massively parallel batch jobs perform computations over recently ingested data. These applications open Colossus files in append mode, and the Colossus client running in the application uses the handle to write their database mutations and table data directly to disks over the network. To ensure the data is stored durably, Colossus replicates its data across several disks, performing writes in parallel and using a quorum technique to avoid waiting on stragglers.
Figure 1: Steps involved in appending data to a file in Colossus.
The above image shows the steps that are taken to append data to a file.
The application opens the file in append mode. The Colossus Curator constructs a handle and sends it to the Colossus Client running in-process, which caches the handle.
The application issues a write call for an arbitrary-sized log entry to the Colossus Client.
The Colossus Client, using the disk addresses in the handle, writes the log entry in parallel to all the disks.
Rapid Storage builds on Colossus’s stateful protocol, leveraging gRPC-based streaming for the underlying transport. When performing low-latency reads and writes to Rapid Storage objects, the Cloud Storage client establishes a stream, providing the same request parameters used in Cloud Storage’s REST protocols, such as the bucket and object name. Further, all the time-consuming Cloud Storage operations such as user authorization and metadata accesses are front-loaded and performed at stream creation time, so subsequent read and write operations go directly to Colossus without any additional overhead, allowing for appendable writes and repeated ranged reads with sub-millisecond latency.
This Colossus architecture enables Rapid Storage to support 20 million requests per second in a single bucket — a scale that is extremely useful in a variety of AI/ML applications. For example, when pre-training a model, pre-processed, tokenized training data is fed into GPUs or TPUs, typically in large files that each contain thousands of tokens. But the data is rarely read sequentially, for example, because different random samples are read in different orders as the training progresses. With Rapid Storage’s stateful protocol, a stream can be established at the start of the training run before executing massively parallel ranged-reads at sub-millisecond speeds. This helps to ensure that accelerators aren’t blocked on storage latency.
Likewise, with appends, Rapid Storage takes advantage of Colossus’s stateful protocol to provide durable writes with sub-millisecond latency, and supports unlimited appends to a single object up to the object size limit. A major challenge with stateful append protocols is how to handle cases where the client or server hangs or crashes. With Rapid Storage, the client receives a handle from Cloud Storage when creating the stream. If the stream gets interrupted but the client wants to continue reading or appending to the object, the client can re-establish a new stream using this handle, which streamlines this flow and minimizes any latency hiccups. It gets trickier when there is a problem on the client, and the application wants to continue appending to an object from a new client. To simplify this, Rapid Storage guarantees that only one gRPC stream can write to an object at a time; each new stream takes over ownership of the object, transactionally locking out any prior stream. Finally, each append operation includes the offset that’s being written to, ensuring that data correctness is always preserved even in the face of network partitions and replays.
Figure 2: A new client taking over ownership of an object.
In the above image, a new client takes over ownership of an object, locking out the previous owner.
Initially, client 1 appends data to an object stored on three disks.
The application decides to fail over to client 2, which opens this object in append mode. The Colossus Curator transactionally locks out client 1 by increasing a version number on each object data replica.
Client 1 attempts to append more data to the object, but cannot because its ownership was tied to the old version number.
To make it as easy as possible to integrate Rapid Storage into your applications, we are also updating our SDKs to support gRPC streaming-based appends and expose a simple application-oriented API. Writing data using handles is a familiar concept in the filesystems world, so we’ve integrated Rapid Storage into Cloud Storage FUSE, which provides clients with file-like access to Cloud Storage buckets, for low-latency file-oriented workloads. Rapid Storage also natively enables Hierarchical Namespace as part of its zonal bucket type, providing enhanced performance, consistency, and folder-oriented APIs.
In short, Rapid Storage combines the sub-millisecond latency of block-like storage, the throughput of a parallel filesystem, and the scalability and ease of use of object storage, and it does all this in large part due to Colossus. Here are some interesting workloads we’ve seen our customers explore during the preview:
AI/ML data preparation, training, and checkpointing
Distributed database architecture optimization
Batch and streaming analytics processing
Video live-streaming and transcoding
Logging and monitoring
Interested in trying Rapid Storage? Indicate your interest here or reach out through your Google Cloud representative.
As organizations continue to prioritize cloud-first strategies to accelerate innovation and gain competitive advantage, legacy databases remain a bottleneck by hindering modernization and stifling growth with unfriendly licensing, complex agreements, and rigid infrastructure.
That’s why this week at Google Cloud Next, we’re announcing that Database Migration Service (DMS) is extending its comprehensive database modernization offering to support SQL Server to PostgreSQL migrations, enabling you to unlock the potential of open-source databases in the cloud and build modern, scalable, and cost-effective applications.
While holding great benefits, migrating from SQL Server to a modern, managed PostgreSQL offering like AlloyDB or Cloud SQL can be a highly complex task. Even though SQL Server and PostgreSQL both adhere to SQL standards, they still have fundamental differences in their architectures, data types, and procedural languages which require deep expertise in both technologies for a successful migration.
For example, SQL Server’s T-SQL syntax and built-in functions often require manual translation to PostgreSQL’s PL/pgSQL. Data type mappings can be intricate, as SQL Server’s DATETIME precision and NVARCHAR handling differ from PostgreSQL’s equivalents.
Furthermore, features like SQL Server’s stored procedures, triggers, and functions often necessitate significant refactoring to align with PostgreSQL’s implementation. This requires deep knowledge in both database systems, as well as specific migration expertise that developers typically don’t possess, and it requires hours of painstaking work, even with the benefit of an automated conversion tool.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud databases’), (‘body’, <wagtail.rich_text.RichText object at 0x3ece6c0c5460>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/products?#databases’), (‘image’, None)])]>
Simplifying database modernization with Database Migration Service
DMS is a fully-managed, serverless cloud service that offers a complete set of capabilities to simplify database “lift and shift” migrations and database modernization journeys.
For modernization efforts, DMS offers an interactive experience that includes data migration, as well as schema and resident code conversion, all in the same powerful user interface. For data migration, it offers high-throughput database initial loads followed by low-latency change data capture to reduce downtime and minimize the impact on business critical applications.
Announcing SQL Server to PostgreSQL migration
The new SQL Server to PostgreSQL.migration experience supports the migration of both self-managed and cloud-managed SQL Server offerings to Cloud SQL for PostgreSQL and AlloyDB to accelerate your database modernization journey. Similar to the existing database modernization offerings, this new experience features a high-throughput initial load of the database followed by seamless change data capture (CDC) replication to synchronize the SQL Server source and PostgreSQL destination, all while the production application is up and running to ensure minimal business interruption.
Database Migration Service is designed to automate the most difficult SQL Server to PostgreSQL migration steps.
For SQL Server schema and code conversion, DMS offers a fast, customizable algorithmic code conversion engine that automates the conversion of most of the database schema and code to the appropriate PostgreSQL dialect, leaving minimal manual conversion work for the user to complete.
The algorithmic conversion engine maps the source database data types and SQL commands to the most suitable PostgreSQL ones, and even refactors complex source features which have no direct PostgreSQL equivalents to achieve the same functionality using available PostgreSQL capabilities. Algorithmic engines are extremely accurate, by nature, for the scenarios they are programmed for. However, they’re limited to just those scenarios, and in real-life usage some of the database code will consist of scenarios that can’t be anticipated.
For these situations, we’re pushing the boundaries of automated database modernization with the introduction of the Gemini automatic conversion engine. This new engine automatically augments the output of the algorithmic conversion, further automating the conversion tasks and reducing the amount of remaining manual work. It also provides a comprehensive conversion report, highlighting which parts of the code were enhanced, why they were changed, and how they were converted.
Instead of spending time researching suitable PostgreSQL features and fixing conversion issues, you can simply review the Gemini recommendations in the conversion report and mark the conversion as verified. Reviewing the completed conversions instead of having to research and fix issues can significantly reduce the manual migration effort and speed up the conversion process.
To further empower SQL Server DBAs, DMS offers a Gemini conversion assist with targeted yet comprehensive SQL Server to PostgreSQL conversion training. Gemini analyzes both the source and the converted code and explains the conversion rationale, highlighting the chosen PostgreSQL features, why they were used, and how they compare to the SQL Server ones. It can then optimize the migrated code for better performance and automatically generate comprehensive comments, for better long-term maintainability.
Database Migration Service provides detailed explanations of SQL Server to PostgreSQL conversions.
At Google Cloud, we’ve been working closely with customers looking to modernize their database estate. One of them is Wayfair LLC, an American online home store for furniture and decor.
“Google Cloud’s Database Migration Service simplifies the process of modernizing databases. Features like Change Data Capture to reduce downtime and AI-assisted code conversion help evolve our database usage more efficiently. This makes the migration process less manual and time-consuming, allowing teams to spend more time on development and less on infrastructure,” said Shashank Srivastava, software engineering manager, Data Foundations, Wayfair.
How to get started
To start your Gemini-powered SQL Server migration, navigate to the Database Migration page in the Google Cloud console, and follow these simple steps:
Create your source and destination connection profiles, which contain information about the source and destination databases. These connection profiles can later be used for additional migrations.
Create a conversion workspace that automatically converts your source schema and the code to a PostgreSQL schema and compatible SQL. Make sure you choose to enable the new Gemini-powered conversion workspace capabilities.
Review the converted schema objects and SQL code, and apply them to your destination Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL instance.
Create a migration job and choose the conversion workspace and connection profiles previously created.
Test your migration job and get started whenever you’re ready.
To learn more about how Database Migration Service can help you modernize your SQL Server databases, please review our DMS documentation and start your migration journey today.
Supporting customers where they want to be is a core value at Google Cloud, and a big part of the reason that we have partnered with Oracle — so that you can innovate faster with the best of Google and the best of Oracle.
This week at Google Cloud Next, we announced significant expansions to our Oracle Database offerings, including the preview of Oracle Base Database Service for a flexible and controllable way to run Oracle databases in the cloud; general availability of Oracle Exadata X11M,bringing the latest generation of the Oracle Exadata platform to Google Cloud; and additional enterprise-ready capabilities including customer managed encryption keys (CMEK).
We are continuing to invest in global infrastructure for Oracle, with a total of 20 locations available in the coming months, adding Oracle Database@Google Cloud presence in Australia, Brazil, Canada, India, Italy, and Japan.
These announcements follow our developments with Oracle since last July, when we launched Oracle Database@Google Cloud. This partnership enables customers to migrate and modernize their Oracle workloads and start taking advantage of Google’s industry-leading data and AI capabilities such as BigQuery, Vertex AI platform, and Gemini foundation models.
Additional features provide customers with even more options in their modernization journey, such as the fully managed Oracle Autonomous Database Serverless. They can also benefit from increased reliability and resiliency features, such as cross-region disaster recovery and Oracle Maximum Availability Gold certification.
“Banco Actinver is committed to providing innovative financial solutions to our clients. By combining the security and performance of Oracle Database with Google Cloud’s data analytics and AI tools, we’re gaining deeper insights into market trends, enhancing our services, and delivering personalized experiences to our customers,” said Jorge Fernandez, CIO, Banco Actinver.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud databases’), (‘body’, <wagtail.rich_text.RichText object at 0x3ece8b0a2370>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/products?#databases’), (‘image’, None)])]>
Innovative new capabilities
We’re expanding our offerings to empower customers with the flexibility to manage a diverse set of database workloads cost effectively.
Oracle Base Database Service: The new Base Database Service delivers a highly controllable and customizable foundational database platform, built on Oracle Cloud Infrastructure (OCI) virtual machines and general-purpose infrastructure. It can empower businesses with the flexibility to manage a diverse range of database workloads directly.
Enhanced Oracle Database Services: In addition to the availability of Exadata Cloud Service, Autonomous Database Service, Oracle Linux, and Oracle on Google Compute Engine (GCE) and Google Kubernetes Engine (GKE), we are pleased to share general availability of Oracle Exadata X11M. Oracle Database@Google Cloud now offers the latest generation of Oracle Exadata machines, the X11M, with enhanced performance and scalability for demanding database workloads. These new machines provide significant performance gains and increased capacity, enabling customers to run even the most intensive Oracle applications with ease. X11M will be available in all new regions.
Customers are embracing Oracle Database@Google Cloud, and to support their global needs, we’re expanding our footprint while maintaining the highest standards of application performance and reliability.
Expanding to 20 Oracle Database@Google Cloud Locations in the coming months: To further support the growing demand for Oracle workloads on Google Cloud, we are launching in more locations, including U.S. Central 1 (Iowa), North America-Northeast 1 (Montreal), North America-Northeast 2 (Toronto), Asia-Northeast 1 (Tokyo), Asia-Northeast 2 (Osaka), Asia-South 1 (Mumbai), Asia-South 2 (Delhi), South America-East 1 (Sao Paulo), Europe-West (Italy), Australia-Southeast2 (Melbourne), and Australia-Southeast1 (Sydney) — and additional zones in Ashburn, Frankfurt, London, Melbourne, and Italy. The new regions and expanded capacity are in addition to Google Cloud regions across U.S. East (Ashburn), U.S. West (Salt Lake City), U.K. South (London), and Germany Central (Frankfurt) that are available today.
New Partner Cross-Cloud Interconnect availability: Partner Cross-Cloud Interconnect for OCI is pleased to expand our global network offerings with new multicloud connectivity between Google and Oracle Cloud Infrastructure in Toronto and Zurich. This further complements our existing 11 regions already served, ensuring the lowest possible latency between both clouds while keeping traffic private and secure.
Cross Region Disaster Recovery: Cross Region Disaster Recovery support for Oracle workloads on Oracle Autonomous Database ensures high availability and resilience, protecting against potential outages and providing continuous operation for critical applications.
Enterprise-grade networking upgrades: Advanced networking upgrades enable enterprises to efficiently deploy their Oracle resources along with Google Cloud and share resources.
Industry-leading certifications and user experience
Google Cloud is committed to providing a seamless and efficient experience for Oracle customers, ensuring that managing and utilizing Oracle databases is straightforward and effective. We offer a combination of native Google Cloud tools and Oracle Cloud Infrastructure (OCI) interfaces, along with robust support for various applications and systems.
Enhanced user experience: Google Cloud is committed to providing an easy-to-use experience for Oracle customers, offering a Google Cloud integrated user experience for application developers and routine database operations, alongside an OCI-native experience for advanced database management. This includes support for Shared VPC, APIs, SDKs, and Terraform.
Application support: Google Cloud is pleased to announce the support for Oracle applications running on Google Cloud, ensuring compatibility and optimal performance, including Oracle E-Business Suite, Peoplesoft Enterprise, JD Edwards Enterprise One, Hyperion Financial Management, and Retail Merchandising.
SAP and Oracle Capability: Oracle workloads on Google Compute Engine are now supported by SAP and Oracle, further validating Google Cloud as a trusted platform for running enterprise applications.
Integration with Google Cloud Monitoring: Provides enterprises a unified monitoring and alerting mechanism across all their Google Cloud database services, now including Oracle Database.
New support in Google Cloud Backup and DR: Our backup service now provides central, policy-based management for backup of Oracle workloads along with other Google Cloud services using secure backup vaults for data protection — isolating and protecting data from threats like ransomware and accidental deletion.
Google Cloud’s strengths make it the preferred hyperscaler for running mission-critical Oracle workloads.
Get started right away from your Google Cloud Console or learn more here.