Starting today, Amazon Elastic Compute Cloud (Amazon EC2) R7g instances are available in AWS Asia Pacific (Melbourne) region. These instances are powered by AWS Graviton3 processors that provide up to 25% better compute performance compared to AWS Graviton2 processors, and built on top of the the AWS Nitro System, a collection of AWS designed innovations that deliver efficient, flexible, and secure cloud services with isolated multi-tenancy, private networking, and fast local storage.
Amazon EC2 Graviton3 instances also use up to 60% less energy to reduce your cloud carbon footprint for the same performance than comparable EC2 instances. For increased scalability, these instances are available in 9 different instance sizes, including bare metal, and offer up to 30 Gbps networking bandwidth and up to 20 Gbps of bandwidth to the Amazon Elastic Block Store (EBS).
Amazon Relational Database Service (Amazon RDS) for Oracle offers Reserved Instances for R7i and M7i instances with up to 46% cost savings compared to On-Demand prices. These instances are powered by custom 4th Generation Intel Xeon Scalable processors and provide larger sizes up to 48xlarge with 192 vCPUs and 1536 GiB of latest DDR5 memory.
Reserved instance benefits apply to both Multi-AZ and Single-AZ configurations. This means that customers can move freely between configurations within the same database instance class type, making them ideal for varying production workloads. Amazon RDS for Oracle Reserved Instances also provide size flexibility for the Oracle database engine under the Bring Your Own License (BYOL) licensing model. With size flexibility, discounted rate for Reserved Instances will automatically apply to usage of any size in the same instance family.
Customers can now purchase Reserved Instances for Amazon RDS Oracle in all AWS regions where R7i and M7i instances are available. For information on specific Oracle database editions and licensing options that support these database instance types, refer to the Amazon RDS user guide.
To get started, purchase Reserved Instances through the AWS Management Console, AWS CLI, or AWS SDK. For detailed pricing information and purchase options, visit the Amazon RDS for Oracle pricing page.
Starting today, Amazon Elastic Compute Cloud (Amazon EC2) C7i-flex and C7i instances are available in the Asia Pacific (Hong Kong) and Europe (Zurich) Regions. These instances are powered by powered by custom 4th Generation Intel Xeon Scalable processors (code-named Sapphire Rapids) custom processors, available only on AWS, and offer up to 15% better performance over comparable x86-based Intel processors utilized by other cloud providers.
C7i-flex instances expand the EC2 Flex instances portfolio to provide the easiest way for you to get price performance benefits for a majority of compute intensive workloads, and deliver up to 19% better price-performance compared to C6i. C7i-flex instances offer the most common sizes, from large to 16xlarge, and are a great first choice for applications that don’t fully utilize all compute resources. With C7i-flex instances, you can seamlessly run web and application servers, databases, caches, Apache Kafka, and Elasticsearch, and more.
C7i instances deliver up to 15% better price-performance versus C6i instances and are a great choice for all compute-intensive workloads, such as batch processing, distributed analytics, ad serving, and video encoding. C7i instances offer larger instance sizes, up to 48xlarge, and two bare metal sizes (metal-24xl, metal-48xl). These bare-metal sizes support built-in Intel accelerators: Data Streaming Accelerator, In-Memory Analytics Accelerator, and QuickAssist Technology that are used to facilitate efficient offload and acceleration of data operations and optimize performance for workloads.
Starting today, Amazon Elastic Compute Cloud (Amazon EC2) M7i-flex and M7i instances powered by custom 4th Gen Intel Xeon Scalable processors (code-named Sapphire Rapids) are available in Asia Pacific (Hong Kong) region. These custom processors, available only on AWS, offer up to 15% better performance over comparable x86-based Intel processors utilized by other cloud providers.
M7i-flex instances are the easiest way for you to get price-performance benefits for a majority of general-purpose workloads. They deliver up to 19% better price-performance compared to M6i. M7i-flex instances offer the most common sizes, from large to 16xlarge, and are a great first choice for applications that don’t fully utilize all compute resources such as web and application servers, virtual-desktops, batch-processing, and microservices.
M7i deliver up to 15% better price-performance compared to M6i. M7i instances are a great choice for workloads that need the largest instance sizes or continuous high CPU usage, such as gaming servers, CPU-based machine learning (ML), and video-streaming. M7i offer larger instance sizes, up to 48xlarge, and two bare metal sizes (metal-24xl, metal-48xl). These bare-metal sizes support built-in Intel accelerators: Data Streaming Accelerator, In-Memory Analytics Accelerator, and QuickAssist Technology that are used to facilitate efficient offload and acceleration of data operations and optimize performance for workloads.
Starting today, you can use E-RTMP (Enhanced Real-Time Messaging Protocol) multitrack video to send multiple video qualities to your Amazon Interactive Video Service (Amazon IVS) stages. This feature enables adaptive bitrate streaming, allowing viewers to watch in the best quality for their network connection. Multitrack video is supported in OBS Studio and complements the existing simulcast capabilities in the IVS broadcast SDK. There is no additional cost for using multitrack video with Real-Time Streaming.
Amazon IVS is a managed live streaming solution designed to make low-latency or real-time video available to viewers around the world. Visit the AWS region table for a full list of AWS Regions where the Amazon IVS console and APIs for control and creation of video streams are available.
Anthropic’s Claude 3.7 Sonnet hybrid reasoning model is now available in Europe (London). Claude 3.7 Sonnet offers advanced AI capabilities with both quick responses and extended, step-by-step thinking made visible to the user. This model has strong capabilities in coding and brings enhanced performance across various tasks, like instruction following, math, and physics.
Claude 3.7 Sonnet introduces a unique approach to AI reasoning by integrating it seamlessly with other capabilities. Unlike traditional models that separate quick responses from those requiring deeper thought, Claude 3.7 Sonnet allows users to toggle between standard and extended thinking modes. In standard mode, it functions as an upgraded version of Claude 3.5 Sonnet. In extended thinking mode, it employs self-reflection to achieve improved results across a wide range of tasks. Amazon Bedrock customers can adjust how long the model thinks, offering a flexible trade-off between speed and answer quality. Additionally, users can control the reasoning budget by specifying a token limit, enabling more precise cost management.
Claude 3.7 Sonnet is also available on Amazon Bedrock in the Europe (Frankfurt), Europe (Ireland), Europe (Paris), Europe (Stockholm), US East (N. Virginia), US East (Ohio), and US West (Oregon) regions. To get started, visit the Amazon Bedrock console. Integrate it into your applications using the Amazon Bedrock API or SDK. For more information, see the AWS News Blog and Claude in Amazon Bedrock.
AWS Lambda now provides native support for Avro and Protobuf formatted Kafka events with Apache Kafka’s event-source-mapping (ESM), and integrates with AWS Glue Schema registry (GSR), Confluent Cloud Schema registry (CCSR), and self-managed Confluent Schema registry (SCSR) for schema management. This enables you to validate your schema, filter events, and process events using open-source Kafka consumer interfaces. Additionally, customers can use Powertools for AWS Lambda to process their Kafka events without writing custom deserialization code, making it easier to build their Kafka applications with AWS Lambda.
Kafka customers use Avro and Protobuf formats for efficient data storage, fast serialization and deserialization, schema evolution support, and interoperability between different programming languages. They utilize schema registry to manage, evolve, and validate schemas before data enters processing pipelines. Previously, customers were required to write custom code within their Lambda function, in order to validate, de-serialize, and filter events when using these data formats. With today’s launch, Lambda natively supports Avro and Protobuf as well as integration with GSR, CCSR and SCSR, enabling customers to process their Kafka events using these data formats, without writing custom code. Additionally, customers can optimize costs through event filtering to prevent unnecessary function invocations.
This feature is generally available in all AWS Commercial Regions where AWS Lambda Kafka ESM is available, except Israel (Tel Aviv), Asia Pacific (Malaysia), and Canada West (Calgary).
To get started, provide your schema registry configuration for your new or existing Kafka ESM in the ESM API, AWS Console, AWS CLI, AWS SDK, AWS CloudFormation, and AWS SAM. Optionally, you can setup filtering rules to discard irrelevant Avro or Protobuf formatted events before function invocations. To build your function with Kafka’s open-source ConsumerRecords interface, add Powertools for AWS Lambda as a dependency within your Lambda function. To learn more, read Lambda ESM documentation and AWS Lambda pricing.
AWS License Manager now supports license type conversions for AWS Marketplace products, initially for Red Hat Enterprise Linux (RHEL) and RHEL for SAP products. Using AWS License Manager, Amazon EC2 customers can now switch Red Hat subscriptions between AWS-provided and Red Hat-provided options from AWS Marketplace without re-deploying instances.
License conversion empowers customers to optimize their licensing strategy by seamlessly transitioning between different subscription models, whether purchased directly through EC2 or from the vendor in AWS Marketplace. Utilizing the license type conversion process, customers are now no longer required to re-deploy instances when switching licenses, reducing downtime and IT operational overhead. By switching their license, customers can negotiate custom pricing directly with vendors and transact through private offers in AWS Marketplace. This new flexibility allows customers to consolidate their vendor spend in AWS Marketplace and maintain preferred vendor relationships for support.
License type conversion for select AWS Marketplace products is available in all AWS Commercial and AWS GovCloud (US) Regions where AWS Marketplace is available.
To get started, customers can configure Linux subscriptions discovery through the AWS License Manager console, AWS CLI, or License Manager Linux subscription API, and create a license type conversion. For more information and to begin using this capability, visit the AWS License Manager page or AWS Marketplace Buyer Guide.
Starting today, Amazon EC2 High Memory U7i instances with 8TB of memory (u7i-8tb.112xlarge) are now available in the US West (Oregon) region. U7i-8tb instances are part of AWS 7th generation and are powered by custom fourth generation Intel Xeon Scalable Processors (Sapphire Rapids). U7i-8tb instances offer 8TiB of DDR5 memory enabling customers to scale transaction processing throughput in a fast-growing data environment.
U7i-8tb instances offer 448 vCPUs, support up to 60Gbps Elastic Block Storage (EBS) for faster data loading and backups, deliver up to 100Gbps of network bandwidth, and support ENA Express. U7i instances are ideal for customers using mission-critical in-memory databases like SAP HANA, Oracle, and SQL Server.
Today, AWS announced major enhancements to its AWS Government Competency, introducing three categories to help public sector customers effectively identify and engage with validated AWS Partners. This update consolidates and streamlines AWS’s public sector partner offerings by merging the AWS Public Safety Competency and AWS Smart City Competency under the Government Competency.
This update features three distinct categories: Citizen Services, Defense & National Security, and Public Safety. The new structure enables government customers to quickly find partners with specific expertise aligned to their mission requirements. Partners in the program must meet rigorous technical validation requirements and demonstrate proven success in their designated categories, ensuring customers can confidently select partners who understand their unique compliance, security, and procurement needs.
AWS has also enhanced the program benefits for qualified partners, including new technical and go-to-market enablement resources, early access to new solutions development tools, and exclusive networking opportunities. Partners will receive specialized support tailored to their focus areas, helping them better serve government customers’ evolving needs.
The AWS Government Competency Program, which has grown from 24 partners in 2016 to more than 169 partners globally, will maintain its high standards through a new re-validation process. This ensures that partners continue to meet the technical expertise, customer success, and compliance requirements that government customers expect.
To learn more about the AWS Government Competency Program and find qualified partners, visit the AWS Government Competency webpage. Government organizations interested in working with AWS Government Competency Partners can start exploring partner solutions today.
Amazon Web Services (AWS) announces the availability of Amazon EC2 I7ie instances in the AWS Europe (Spain) region. Designed for large storage I/O intensive workloads, these new instances are powered by 5th generation Intel Xeon Scalable processors with an all-core turbo frequency of 3.2 GHz, offering up to 40% better compute performance and 20% better price performance over existing I3en instances.
I7ie instances offer up to 120TB local NVMe storage density—the highest available in the cloud for storage optimized instances—and deliver up to twice as many vCPUs and memory compared to prior generation instances. Powered by 3rd generation AWS Nitro SSDs, these instances achieve up to 65% better real-time storage performance, up to 50% lower storage I/O latency, and 65% lower storage I/O latency variability compared to existing I3en instances. Additionally, the 16KB torn write prevention feature, enables customers to eliminate performance bottlenecks for database workloads.
I7ie instances are high-density storage-optimized instances, for workloads that demand rapid local storage with high random read/write performance and consistently low latency for accessing large data sets. These versatile instances are offered in eleven different sizes including 2 metal sizes, providing flexibility to match customers computational needs. They deliver up to 100 Gbps of network performance bandwidth, and 60 Gbps of dedicated bandwidth for Amazon Elastic Block Store (EBS), ensuring fast and efficient data transfer for the most demanding applications.
AWS is expanding resource control policies (RCPs) support to include two additional services: Amazon Elastic Container Registry and Amazon OpenSearch Serverless. This expansion enhances your ability to centrally establish a data perimeter across a wider range of AWS resources in your organization.
RCPs are available in all AWS commercial Regions and AWS GovCloud (US) Regions. To learn more about RCPs and view the full list of supported AWS services, visit the Resource control policies (RCPs) documentation in the AWS Organizations User Guide.
Today, AWS announced support for job metadata logs for AWS Parallel Computing Service (PCS). With this launch, PCS can be configured to emit job completion logs to Amazon CloudWatch Logs, Amazon S3, and Amazon Data Firehose. Each job log will contain detailed metadata including information such as job submission time, start time, completed time, user who submitted the job, queue that processed the job, and Amazon EC2 instances used to run the job. Using these logs, you can analyze usage patterns, identify and troubleshoot job failures, track job wait times in the queue, generate user-level usage reports, and more.
This feature is available in all AWS Regions where PCS is available. You can enable job completion metadata logging on all newly-created PCS clusters in just a few clicks using the AWS Management Console. Visit our job completion documentation page to learn more.
Amazon EC2 Auto Scaling now offers the ability to filter out instance details from the DescribeAutoScalingGroups API with a new parameter. With IncludeInstances set to false, you can quickly access metadata and configurations about your Auto Scaling Groups without the overhead of instance details, reducing the size of the API response and improving API response time.
The new parameter is available in all commercial AWS Regions, and AWS GovCloud (US) Regions. To learn more, see the EC2 Auto Scaling API Reference.
AWS Parallel Computing Service (PCS) is now available in the AWS GovCloud (US-East, US-West) Regions
Today, AWS launches AWS Parallel Computing Service (PCS) in the AWS GovCloud (US-East, US-West) Regions, enabling you to easily build and manage High Performance Computing (HPC) clusters using the Slurm workload manager.
AWS PCS is a managed service that makes it easier for you to run and scale your high performance computing (HPC) workloads and build scientific and engineering models on AWS using Slurm. You can use AWS PCS to build complete, elastic environments that integrate compute, storage, networking, and visualization tools. AWS PCS simplifies cluster operations with managed updates and built-in observability features, helping to remove the burden of maintenance. You can work in a familiar environment, focusing on your research and innovation instead of worrying about infrastructure.
AWS Payment Cryptography has expanded its regional presence in Asia Pacific with availability in two new regions – Asia Pacific (Mumbai) and Asia Pacific (Osaka). This expansion enables customers with latency-sensitive payment applications to build, deploy or migrate into additional AWS Regions without depending on cross-region support. For customers processing payment workloads in Asia Pacific (Tokyo), the new Osaka region offers an additional option for multi-region high availability.
AWS Payment Cryptography is a fully managed service that simplifies payment-specific cryptographic operations and key management for cloud-hosted payment applications. The service scales elastically with your business needs and is assessed as compliant with PCI PIN Security requirements, eliminating the need to maintain dedicated payment HSM instances. Organizations performing payment functions – including acquirers, payment facilitators, networks, switches, processors, and banks can now position their payment cryptographic operations closer to their cloud applications while reducing dependencies on auxiliary data centers or colocation facilities with dedicated payment HSMs.
AWS Payment Cryptography is available in the following AWS Regions: US East (Ohio, N. Virginia), US West (Oregon), Europe (Ireland, Frankfurt) and Asia Pacific (Singapore, Tokyo, Osaka, Mumbai).
To learn more about the service, see the AWS Payment Cryptography user guide, and visit the AWS Payment Cryptography page for pricing details and availability in additional regions.
Valkey announces general availability of General Language Independent Driver for the Enterprise (GLIDE) 2.0, the latest release of one of its official open source Valkey client libraries. Valkey is the most permissive open source alternative to Redis stewarded by the Linux Foundation, which means it will always be open source. Valkey GLIDE is a reliable, high-performance, multi-language client that supports all Valkey commands. GLIDE 2.0 brings new capabilities that expand developer support, improve observability, and optimize performance for high-throughput workloads.
Valkey GLIDE 2.0 extends its multi-language support to Go, joining Java, Python, and Node.js to provide a consistent, fully compatible API experience across all four languages—with more on the way. With this release, Valkey GLIDE now supports OpenTelemetry, an open source, vendor-neutral framework enabling developers to generate, collect, and export telemetry data and critical client-side performance insights. Additionally, GLIDE 2.0 introduces pipeline capabilities, reducing network overhead and latency for high-frequency use cases by allowing multiple commands to be grouped and executed as a single operation.
Valkey GLIDE is compatible with versions 7.2, 8.0 and 8.1 of Valkey, as well as versions 6.2, 7.0, and 7.2 of Redis OSS. Valkey GLIDE 2.0 is available now through the Valkey repository on GitHub. For more information about Valkey’s official client libraries, visit the Valkey website.
Amazon S3 Express One Zone now supports renaming objects with the new RenameObject API. For the first time in S3, you can rename existing objects atomically (with a single operation) without any data movement.
The RenameObject API simplifies data management in S3 directory buckets by transforming a multi-step rename operation into a single API call. You can now rename objects in S3 Express One Zone by specifying an existing object’s name as the source and the new name of the object as the destination within the same S3 directory bucket. With no data movement involved, this capability accelerates applications like log file management, media processing, and data analytics while lowering costs. For example, renaming a 1-terabyte log file can now complete in milliseconds, instead of hours, significantly accelerating applications and reducing cost.
You can use the RenameObject API in the S3 Express One Zone storage class in all AWS Regions where the storage class is available. You can get started with the new capability in S3 Express One Zone using the AWS SDKs, AWS CLI, AWS Management Console, Amazon S3 API, or Mountpoint for Amazon S3 (version 1.19.0 or higher). To learn more, visit the S3 User Guide.
Under the hood, there’s a lot of technology and expertise that goes into delivering the performance you get from BigQuery, Google Cloud’s data to AI platform. Separating storage and compute provides unique resource allocation flexibility and enables petabyte-scale analysis, while features like compressed storage, compute autoscaling, and flexible pricing contribute to its efficiency. Then there’s the infrastructure — technologies like Borg, Colossus, Jupiter, and Dremel, as we discussed in a previous post.
BigQuery is continually pushing the limits of query price/performance. Google infrastructure innovations such as L4 in Colossus, userspace host networking, optimized BigQuery storage formats, and a cutting-edge data center network have allowed us to do a complete modernization of BigQuery’s core data warehousing technology. We do this while adhering to core principles of self-tuning and zero user intervention, to guarantee the best possible price/performance for all queries. Collectively, we group these improvements into BigQuery’s advanced runtime. In this blog post, we introduce you to one of these improvements, enhanced vectorization, now in preview. Then, stay tuned for future blog posts where we’ll go deep on other technologies and techniques in the advanced runtime family.
Before diving into enhanced vectorization, let’s talk about vectorized query execution. In vectorized query execution, columnar data is processed in blocks the size of the CPU cache using Single Instruction Multiple Data (SIMD) instructions, which is now the de-facto industry standard for efficient query processing. BigQuery’s enhanced vectorization expands on vectorized query execution by applying it to key aspects of query processing, such as filter evaluation in BigQuery storage, support for parallel execution of query algorithms, and through specialized data encodings and optimization techniques. Let’s take a closer look.
aside_block
<ListValue: [StructValue([(‘title’, ‘$300 in free credit to try Google Cloud data analytics’), (‘body’, <wagtail.rich_text.RichText object at 0x3e7a058daf40>), (‘btn_text’, ‘Start building for free’), (‘href’, ‘http://console.cloud.google.com/freetrial?redirectPath=/bigquery/’), (‘image’, None)])]>
Data encodings Modern columnar storage formats use space-efficient data encodings such as dictionary and run-length encodings. For instance, if a column has a million rows but only 10 unique values, dictionary encoding stores those 10 values once and assigns a smaller integer ID to each row rather than repeating the full value. Enhanced vectorization can directly process this encoded data, eliminating redundant computations and significantly boosting query performance. The smaller memory footprint of this encoded data also improves cache locality, creating more opportunities for vectorization.
Figure 1: Dictionary and run-length encodings
For example, as figure 1 demonstrates, “Sedan”, “Wagon” and “SUV” string values are encoded in the dictionary, replacing the repeated string literals with integers that represent indices in the dictionary built from those string values. Subsequently, the repeated integer values can be further represented with run-length encoding. Both types of encodings can offer substantial space and processing savings.
Expression folding and common subexpression elimination Enhanced vectorization integrates native support for dictionary and run-length encoded data directly into its algorithms. This, combined with optimization techniques such as expression folding, folding propagation, and common subexpression elimination, allows it to intelligently reshape query execution plans. The result can be a significant reduction, or indeed complete removal, of unnecessary data processing.
Consider a scenario where REGEXP_CONTAINS(id, '[0-9]{2}$') AS shard receives dictionary-encoded input. The REGEXP_CONTAINS calculation is performed only once for each unique dictionary value, and the resulting expression is also dictionary-encoded, reducing the number of evaluations significantly and leading to performance improvements.
Figure 2: Dictionary folding
Here, the calculation is applied to the input dictionary-encoded data directly, producing output of dictionary-encoded data and skipping the dictionary expansion.
With enhanced vectorization, we take expression folding optimization even further by, in some cases, converting an expression into a constant. Consider this query:
code_block
<ListValue: [StructValue([(‘code’, “SELECT SUM(number) FROM tablernWHERE REGEXP_CONTAINS(id, ‘^.*[0-9]{2}’);”), (‘language’, ‘lang-sql’), (‘caption’, <wagtail.rich_text.RichText object at 0x3e7a06941640>)])]>
If the id in the Capacitor file for this table is dictionary-encoded, the system’s expression folding will evaluate all dictionary values, and, because none of its values contain two digits, determine that the REGEXP_CONTAINS condition is always false, and replace the WHERE clause with a constant false. As a result, BigQuery completely skips scanning the Capacitor file for this table, significantly boosting performance. Of course, these optimizations are applicable across a broad range of scenarios and not just to the query used in this example.
Data-encoding-enabled optimizations Our state-of-the art join algorithm tries to preserve dictionary and run-length-encoded data wherever possible and makes runtime decisions taking data encoding into account. For example, if the probe side in the join key is dictionary-encoded, we can use that knowledge to avoid repeated hash-table lookups. Also, during aggregation, we can skip building a hashmap if data is already dictionary-encoded and its cardinality is known.
Parallelizable join and aggregation algorithms Enhanced vectorization harnesses sophisticated parallelizable algorithms for efficient joins and aggregations. When parallel execution is enabled in a Dremel leaf node for certain query-execution modes, the join algorithm can build and probe the right-hand side hash table in parallel using multiple threads. Similarly, aggregation algorithms can perform both local and global aggregations across multiple threads simultaneously. This parallel execution of join and aggregation algorithms leads to a substantial acceleration of query execution.
Tighter integration with Capacitor We re-engineered Capacitor for the enhanced vectorization runtime, making it smarter and more efficient. This updated version now natively supports semi-structured and JSON data, using sophisticated operators to rebuild JSON data efficiently. Capacitor enables enhanced vectorization runtime to directly access dictionary and run-length-encoded data and apply various optimizations based on data. It intelligently applies folding to a constant optimization when an entire column has the same value. And it can prune expressions in functions expecting NULL, such as IF_NULL and COALESCE, when a column is confirmed to be NULL-free.
Filter pushdown in Capacitor Capacitor leverages the same vectorized engine as enhanced vectorization to efficiently push down filters and computations. This allows for tailored optimizations based on specific file characteristics and the expressions used. When combined with dictionary and run-length-encoded data, this approach delivers exceptionally fast and efficient data scans, enabling further optimizations like expression folding.
Enhanced vectorization in action
Let’s illustrate the power of these techniques with a concrete example. Enhanced vectorization accelerated one query by 21 times, slashing execution time from over one minute (61 seconds) down to 2.9 seconds.
The query that achieved this dramatic speedup was:
code_block
<ListValue: [StructValue([(‘code’, ‘SELECTrn ANY_VALUE(id) AS id,rn hash_idrnFROM (rn SELECTrn CAST(source_id AS STRING) AS id,rn TO_HEX(SHA1(CAST(source_id AS STRING))) AS hash_idrn FROM `source_data`)rnWHERErn hash_id IS NOT NULLrnGROUP BYrn hash_id’), (‘language’, ‘lang-sql’), (‘caption’, <wagtail.rich_text.RichText object at 0x3e7a06582850>)])]>
This query ran against a table with over 13 billion logical rows spread across 167 partitions, stored in Capacitor columnar storage format and optimized with dictionary and run-length-encoded data.
Without enhanced vectorization
Executing this query with a regular query engine would involve several steps:
Reading all data for each partition, fully expanding the dictionary and run-length-encoded columnar data.
Computing CAST(source_id AS STRING) and TO_HEX(SHA1(CAST(source_id AS STRING))) for every single columnar data value.
Building a hashmap from all the non-NULL hash_id values.
With enhanced vectorization
When enhanced vectorization processed the same query over the same dataset, it automatically applied these crucial optimizations:
It directly scanned the columnar data in the Capacitor file while preserving its dictionary-encoded data.
It detected and eliminated duplicate computations for CAST(source_id AS STRING) by identifying them as common subexpressions.
It folded the TO_HEX(SHA1(CAST(source_id AS STRING))) computation, propagating the resulting dictionary-encoded data directly to the aggregation step.
The aggregation step recognized the data was already dictionary-encoded, allowing it to completely skip building a hashmap for aggregation.
This example of 21-times query speedup vividly demonstrates how tight integration between enhanced vectorization runtime and Capacitor and various optimization techniques can lead to substantial query performance improvements.
What’s next
BigQuery’s enhanced vectorization significantly improves query price/performance. Internally, we’ve seen a substantial reduction in query latency with comparable or even lower slot utilization with enhanced vectorization runtime, though individual query results can differ. This performance gain comes from innovations in both enhanced vectorization and BigQuery’s storage formats.
We’re dedicated to continuously improving both, applying even more advanced optimizations alongside Google’s infrastructure advancements in storage, compute, and networking to further boost query efficiency and expand the range of queries that the advanced runtime can handle. Over the coming months, BigQuery’s advanced runtime’s enhanced vectorization will be enabled for all customers by default, but you can enable it earlier for your project today. Next up: We’ll offer BigQuery enhanced vectorization for Parquet files and Iceberg tables!
Cloud backups were once considered as little more than an insurance policy. Now, your backups should do more! They should be autonomous, cost-efficient, and analytics-ready by default.
That’s why Eon built a platform purposefully aligned with Google Cloud to eliminate backup blind spots, simplify recovery, and unlock the value inside backup data without requiring teams to become policy experts or infrastructure wranglers.
Still, no matter what platform you use, it’s critical to understand what resilient cloud backup looks like and how to get there with Google Cloud’s native capabilities.
What makes cloud backup resilient?
Before diving into tooling, it’s worth asking: What does a resilient backup strategy look like in the cloud? In our work with Google Cloud users across industries, we’ve found five common criteria:
5 signs your backup posture may be at risk
You can’t easily see what’s backed up (or not)
Retention policies vary across projects and teams
Data is duplicated or stored inefficiently, driving up spend
Cloud ransomware protection is reactive rather than policy-driven
Recovery requires full restores even when you only need one object
aside_block
<ListValue: [StructValue([(‘title’, ‘Try Google Cloud for free’), (‘body’, <wagtail.rich_text.RichText object at 0x3e79daa4f550>), (‘btn_text’, ‘Get started for free’), (‘href’, ‘https://console.cloud.google.com/freetrial?redirectPath=/welcome’), (‘image’, None)])]>
Best practices for data protection
Google Cloud provides foundational capabilities to protect your data if you configure and use them consistently. Here’s how to maximize native protection:
1. Versioning and retention: first lines of defense
Enable Object Versioning in Cloud Storage to retain multiple object versions, making it easier to recover from accidental deletions. Pair this with Retention Policies to enforce minimum storage lifetimes for regulatory or critical datasets.
Tip: Use Bucket Lock for write-once-read-many (WORM) protection in the areas where compliance matters most.
2. Monitor for gaps in coverage
Use native services like Cloud SQL backups, GKE snapshots, and Persistent Disk images, but be mindful that backup responsibilities can fall to different teams. Without centralized visibility, coverage becomes inconsistent.
Tip: Use Cloud Asset Inventory or scheduled BigQuery queries to audit coverage.
3. Design for granular recovery
Plan for partial restores since not everything needs a full rollback. Whether it’s a single BigQuery table or a specific Cloud Storage object, restoring only what you need saves time and cost.
Tip: Use Object Lifecycle Management to automatically transition older or less critical Cloud Storage objects to colder storage classes.
Automating the complexity away
Managing cloud backup at scale is hard to do manually. From onboarding new workloads to applying consistent policies, human-led approaches don’t scale well.
That’s why more teams are exploring autonomous Cloud Backup Posture Management (CBPM) solutions, like Eon, that detect new assets in real time, apply smart backup rules automatically, and enforce consistent protection across environments.
With Eon, you don’t have to tag resources or write custom scripts. Our platform classifies and protects your Google Cloud assets out of the box—whether you’re working with GKE, Cloud SQL, BigQuery, or another solution.
From backups to business insights
Traditionally, backup data was siloed, underused, and only meant to be retrieved in emergencies. But, increasingly, teams are unlocking that data to:
Run analysis directly on backups using BigQuery and Dataproc,
Feed training and monitoring pipelines via Vertex AI,
Deliver audit-ready dashboards with Looker, powered by backup snapshots.
With Eon, this is built-in. We transform backups into zero-ETL data lakes that reduce pipeline costs and provide immediate access to structured data with no reprocessing required.
What a “mature” backup posture looks like
The end goal for many cloud-native teams is not just to “have backups.” It’s to develop a resilient, intelligent backup strategy that adapts to scale and risk.
Here’s what that looks like:
Automated discovery of new resources
Policy-driven protection tailored to data type and criticality
Immutable backups with time-locked retention
Search-first recovery instead of full snapshot restores
Cost-aware tiering and storage deduplication
Eon helps Google Cloud users reach this level of maturity faster without the burden of custom tooling or constant policy updates.
Ready to simplify backup?
If your team spends hours managing scripts, storage tiers, or backup tags across cloud environments, it may be time to rethink your approach.
Eon was built to make cloud backup resilient, autonomous, and actually useful. From ransomware protection to instant, object-level recovery—and now, zero-ETL access to analytics—we’re here to help you unlock the full potential of your backup data.
Book a demo to see how Eon can modernize your Google Cloud data protection strategy.
To discover how Google Cloud can support your startup, visit our program page. You can also sign up for our newsletter to stay informed about community activities, digital events, special offers, and more.
