Cloud

Storage-dense workloads need consistent performance, high SSD density, and predictable maintenance that preserves SSD data. Last week at Google Cloud Next ‘24, we announced the general availability of the Z3 machine series, our first storage-optimized VM family. With an industry-leading 6M 100% random-read and 6M write IOPs, an incredibly dense storage configuration of up to 409 SSD (GB) per vCPU and a highmem configuration (1 vCPU : 8 GB RAM), Z3 VMs provide a consistent performance and maintenance experience with minimal disruptions for storage-dense workloads such as horizontal, scale-out databases and log analytics workloads, allowing you to reduce total cost of ownership by offering more storage capacity for fewer cores.

The Z3 machine series brings together the enterprise-grade performance and reliability of the 4th Gen Intel Xeon scalable processor (code-named Sapphire Rapids), Google’s custom Intel Infrastructure Processing Unit (IPU), and the latest generation of Local SSD to Google Compute Engine and Google Kubernetes Engine customers. Z3 is also the debut of next generation Local SSD, with up to 3x disk throughput over prior-generation instances and up to 35% lower disk latency. Z3 is ideal for horizontal, scale-out databases, flash-optimized databases, data warehouses, and other applications with dense storage requirements.

Initially, Z3 comes in two shapes: 176 vCPU with 1.4T of DDR5 memory, and 88 vCPU with 704 GB of DDR5 memory, both with 36TB of next generation Local SSD.

What our customers are saying

“Google’s Z3 instances help fulfill Aerospike’s commitment to deliver superior cost performance for real-time database users. Our testing shows that they not only meet the high expectations of our mutual customers, but can also reduce their cluster sizes by more than 70%, simplifying their environments and reducing overall costs.” – Srini Srinivasan, Chief Technology Officer, Aerospike

“bi(OS) is the real-time database that ‘scales up and scales out.’ GCP’s Z3 instance is the first cloud VM that can unleash its true potential. Over 72 hours, using only three Z3-88 instances, bi(OS) delivered ~164,000 rows/sec of throughput at a mean latency < 32 ms. All inserts (49%), upserts (12%), and selects (39%) were performed with five 9s reliability across three zones.” – Darshan Rawal, Founder and CEO, Isima

“When we tested ScyllaDB on the new Z3 instances, ScyllaDB exhibited a significant throughput improvement across workloads versus the previous generation of N2 instances. We observed a 23% increase in write throughput, 24% for mixed workloads, and 14% for reads – and that’s with 8 fewer cores (z3-highmem-88 vs n2-highmem-96). On these new instances, a cluster of just three ScyllaDB nodes can achieve around 2.2M OPS for writes and mixed workloads and around 3.6M OPS for reads. We are excited for the incredible performance and value that these new instances will offer our customers.” – Avi Kivity, Co-founder and CTO, ScyllaDB

Enhanced maintenance experience

Z3 VMs come with a variety of new infrastructure lifecycle technologies that provide tighter control and specificity around maintenance. Z3 VMs receive notice from the system several days in advance of a maintenance event. You can then schedule the maintenance event at a time of your choosing, or default to the scheduled time. This allows you to more predictively plan ahead of a disruptive event, while allowing us to deliver more performant and secure infrastructure. You’ll also receive in-place upgrades that preserve your data through planned maintenance events. 

Powered by Titanium

Z3 VMs are built on Titanium, Google’s system of purpose-built custom silicon, security microcontrollers, and tiered scale-out offloads. The end result is better performance, lifecycle management, reliability, and security for your workloads. Titanium enables Z3 to deliver up to 200 Gbps of fully encrypted networking, 3x faster packet-processing capabilities than prior generation VMs, near-bare-metal performance, integrated maintenance updates for the majority of workloads, and advanced controls for the more sensitive workloads.

“Building on our successful partnership with Google Cloud since 2016, we’re proud to collaborate on Google Cloud’s first storage-optimized VM family. This collaboration delivers Intel’s 4th Gen Intel Xeon processor and Google’s custom Intel IPU that unlocks new levels of efficiency and performance.” – Suzi Jewett, General Manager – Intel Xeon Products, Intel Corporation

Hyperdisk storage

Hyperdisk is Google Cloud’s latest-generation block storage. Built on Titanium, Hyperdisk delivers significantly higher levels of performance, flexibility, and efficiency by decoupling storage processing from the virtual machine host. With Hyperdisk, you can dynamically scale storage performance and capacity independently to efficiently meet the storage I/O needs of data-intensive workloads such as data analytics and databases. Now, you don’t have to choose expensive, large compute instances just to get higher storage performance.

Get started with Z3 today

Z3 VMs are available today in the following regions: us-central1 (Iowa), europe-west4 (Netherlands), and asia-southeast1 (Singapore). To start using Z3 instances, select Z3 under the new Storage-Optimized machine family when creating a new VM or GKE node pool in the Google Cloud console. Learn more at the Z3 machine series page. Contact your Google Cloud sales representative for more information on regional availability.

Organizations are increasingly adopting streaming technologies, and Google Cloud offers a comprehensive solution for streaming ingestion and analytics. Cloud Pub/Sub is Google Cloud’s simple, highly scalable and reliable global messaging service. It serves as the primary entry point for you to ingest your streaming data into Google Cloud and is natively integrated with BigQuery, Google Cloud’s unified, AI-ready data analytics platform. You can then use this data for downstream analytics, visualization, and AI applications. Today, we are excited to announce recent Pub/Sub innovations answering customer needs for simplified streaming data ingestion and analytics.

One-click Streaming Import (GA)

Multi-cloud workloads are becoming a reality for many organizations where customers would like to run certain workloads (e.g., operational) on one public cloud and want to run their analytical workloads on another. However, it can be a challenge to gain a holistic view of their business data. Through data consolidation in one public cloud, you can run analytics across their entire data footprint. For Google Cloud customers it is common to consolidate data in BigQuery, providing a source of truth for the organization. 

To ingest streaming data from external sources such as AWS Kinesis Data Streams into Google Cloud, you need to configure, deploy, run, manage and scale a custom connector. You also need to monitor and maintain the connector to ensure the streaming ingestion pipeline is running as expected. Last week, we launched a no-code, one-click capability to ingest streaming data into Pub/Sub topics from external sources, starting with Kinesis Data Streams. The Import Topics capability is now generally available (GA) and offers multiple benefits:

Simplified data pipelines: You can streamline your cross-cloud streaming data ingestion pipelines by using the Import Topics capability. This removes the overhead of running and managing a custom connector.

Auto-scaling: Streaming pipelines created with managed import topics scale up and down based on the incoming throughput.

Out-of-the-box monitoring: Three new Pub/Sub metrics are now available out-of-the-box to monitor your import topics.

Streaming analytics with Pub/Sub Apache Flink connector (GA)

Apache Flink is an open-source stream processing framework with powerful stream and batch processing capabilities, with growing adoption across enterprises. Customers often use Apache Flink with messaging services to power streaming analytics use cases. We are pleased to announce that a new version of the Pub/Sub Flink Connector is now GA with active support from the Google Cloud Pub/Sub team. The connector is fully open source under an Apache 2.0 license and hosted on our GitHub repository. With just a few steps, the connector allows you to connect your existing Apache Flink deployment to Pub/Sub. 

The connector allows you to publish an Apache Flink output into Pub/Sub topics or use Pub/Sub subscriptions as a source in Apache Flink applications. The new GA version of the connector comes with multiple enhancements. It now leverages the StreamingPull API to achieve maximum throughput and low latency. We also added support for automatic message lease extensions to enable setting longer checkpointing intervals. Finally, the connector supports the latest Apache Flink source streaming API.

Enhanced Export Subscriptions experience

Pub/Sub has two popular export subscriptions — BigQuery and Cloud Storage. BigQuery subscriptions can now be leveraged as a simple method to ingest streaming data into BigLake Managed Tables, BigQuery’s recently announced capability for building open-format lakehouses on Google Cloud. You can use this method to transform your streaming data into Parquet or Iceberg format files in your Cloud Storage buckets. We also launched a number of enhancements to these export subscriptions.

BigQuery subscriptions support a growing number of ways to move your structured data seamlessly. The biggest change is the ability to write JSON data into columns in BigQuery without defining a schema on the Pub/Sub topic. Previously, the only way to get data into columns was to define a schema on the topic and publish data that matched that schema. Now, with the use table schema feature, Pub/Sub can write JSON messages to the BigQuery table using its schema. Basic types are supported now and support for more advanced types like NUMERIC and DATETIME is coming soon.

Speaking of type support, BigQuery subscriptions now handle most Avro logical types. BigQuery subscriptions now support non-local timestamp types (compatible with the BigQuery TIMESTAMP type) and decimal types (compatible with the BigQuery NUMERIC and BIGNUMERIC types, coming soon). You can use these logical types to preserve the semantic meaning of fields across your pipelines.

Another highly requested feature coming soon to both BigQuery subscriptions and Cloud Storage subscriptions is the ability to specify a custom service account. Currently, only the per-project Pub/Sub service account can be used to write messages to your table or bucket. Therefore, when you grant access, you enable anyone who has permission to use this project-wide service account to write to the destination. You may prefer to limit access to a specific service account via this upcoming feature.

Cloud Storage subscriptions will be enhanced in the coming months with a new batching option allowing you to batch Cloud Storage files based on the number of Pub/Sub messages in each file. You will also be able to specify a custom datetime format in Cloud Storage filenames to support custom downstream data lake analysis pipelines. Finally, you’ll soon be able to use topic schema to write data to your Cloud Storage bucket.

Getting started

We’re excited to introduce a set of new capabilities to help you leverage your streaming data for a variety of use cases. You can now simplify your cross-cloud ingestion pipelines with Managed Import. You can also leverage Apache Flink with Pub/Sub for streaming analytics use cases. Finally, you can now use enhanced Export Subscriptions to seamlessly get data in either BigQuery or Cloud Storage. We are excited to see how you use these Pub/Sub features to solve your business challenges.

At Google Cloud Next 2024 we announced the general availability of Hyperdisk Storage Pools, the first block storage offering from a hyperscale cloud provider to offer thin-provisioning, data reduction, and capacity pooling. Hyperdisk Storage Pools help you lower your Total Cost of Ownership (TCO) by as much as 30-50%, helping you achieve higher efficiency, modernize SAN-based workloads, and simplify block storage management. You can get started with Storage Pools in the Google Cloud console today.  In this blog post, let’s take a look at how Hyperdisk Storage Pools work, and how to incorporate them into your environment.  

Hyperdisk Storage Pools you can lower your block storage TCO in two ways. First, Hyperdisk Storage Pools leverage thin-provisioned capacity pooling to drive up to 80% capacity utilization. Instead of provisioning hundreds of volumes and only using a fraction of your purchased capacity, you can provision a single Storage Pool which will only be consumed by the data you write. Second, Hyperdisk Storage Pools leverage data reduction technologies to further lower the capacity you need to purchase up front, allowing you to provision less storage and you use more of what you buy. 

Take for example a database workload with 1 PiB (1,000 TiB) of written data. Outside of a Hyperdisk Storage Pool, this workload runs at 38% utilization, and as a result requires the customer to provision 2.6 PiB of capacity (to account for the low utilization). With Hyperdisk Storage Pools, the customer can use thin-provisioning to drive 80% utilization and benefit from 22% data reduction. As a result, this workload would require almost 2.5 times less capacity than if run on other leading clouds, enabling Hyperdisk Storage Pools to lower the capacity TCO of this workload by as much as 56%.

Storage Pools are part of Google Cloud’s next-generation Hyperdisk block storage portfolio. With Hyperdisk volumes you can workload-optimize your storage by provisioning the capacity and performance you need, independently and dynamically. Storage Pools with Advanced Capacity are available for Hyperdisk Balanced and Throughput volume types. Hyperdisk Storage Pools let you share Hyperdisk capacity across the volumes in that pool, without changing how the included Hyperdisk volumes work. Once you’ve created your Storage Pool, simply provision and attach Hyperdisk volumes as normal. That’s why we call Hyperdisk Storage Pools “effortlessly efficient.”

Simplify block-storage management

The benefits of Hyperdisk Storage Pools go beyond lowering your TCO. Infrastructure managers know that workloads have a broad range of capacity and performance needs, and that because application owners often lack insight into their requirements, the needs of those applications can change significantly over time. As a result of that uncertainty, deciding how much block storage capacity and performance each workload will need can take weeks, or even months. Managing your block storage to ensure every workload has the capacity it needs can be a painful, time-consuming process — especially as your cloud adoption grows.

With the explosion of workloads like generative AI, scale-out analytics workloads such as Kafka, intelligent databases, cloud modernization initiatives, and cloud desktops, the last few years have only made these challenges worse. It became obvious to us that customers need a new approach to managing block storage in the cloud. 

That’s why we built Hyperdisk Storage Pools: a radically simpler way to manage cloud block storage. With Hyperdisk Storage Pools, you can provision a pool of thin-provisioned and data-reduced capacity that is then shared across Hyperdisk volumes. Because the capacity of the pool is only consumed by written data (and the data is reduced), you no longer have to choose between efficiency and availability. Now you can provide your workloads with the capacity they need while only consuming the storage they actually use. This makes getting SAN-based workloads such as Virtual Desktops (VDI) or databases to the cloud much simpler. Instead of lengthy planning cycles, simply create a Hyperdisk Storage Pool and begin building your VDI or database workloads on a pool of shared capacity. 

Early Hyperdisk Storage Pools customers such as REWE are already seeing the value of the offering. 

“At REWE, block storage plays a critical role in serving applications from analytics to end-user computing. We see Hyperdisk Storage Pools as the foundation for our next-generation block storage applications. We are excited to use Hyperdisk Storage Pools to drive storage efficiency, while making storage management easier.” – Jan Rundshagen, Infrastructure Manager, REWE

Get started

You can get started with Hyperdisk Storage Pools by logging into the Google Cloud console, navigating to Compute Engine, and looking under Storage. Then create your Storage Pool, select its volume type (Balanced or Throughput), Advanced Capacity (if you want to take advantage of thin-provisioning and data reduction), and set the total amount of capacity and performance you will need in the pool. You can now begin creating new Hyperdisk volumes in the pool that share the pool’s capacity. 

We think that Hyperdisk Storage Pools represent a fundamentally new approach to managing block storage in the cloud. For the first time, you no longer have to choose between efficiency and availability, or between cost and simplicity. As we look into the future, we see Hyperdisk Storage Pools as a new paradigm for cloud storage management.

Hyperdisk Storage Pools are rolling out to regions and zones now. Get started with Storage Pools in the Google Cloud console today or learn more in our documentation about how to create, use and manage your pools.

Written by: Gabby Roncone, Dan Black, John Wolfram, Tyler McLellan, Nick Simonian, Ryan Hall, Anton Prokopenkov, Luke Jenkins, Dan Perez, Lexie Aytes, Alden Wahlstrom

With Russia’s full-scale invasion in its third year, Sandworm (aka FROZENBARENTS) remains a formidable threat to Ukraine. The group’s operations in support of Moscow’s war aims have proven tactically and operationally adaptable, and as of today, appear to be better integrated with the activities of Russia’s conventional forces than in any other previous phase of the conflict. To date, no other Russian government-backed cyber group has played a more central role in shaping and supporting Russia’s military campaign. 

Yet the threat posed by Sandworm is far from limited to Ukraine. Mandiant continues to see operations from the group that are global in scope in key political, military, and economic hotspots for Russia. Additionally, with a record number of people participating in national elections in 2024, Sandworm’s history of attempting to interfere in democratic processes further elevates the severity of the threat the group may pose in the near-term. 

Given the active and diffuse nature of the threat posed by Sandworm globally, Mandiant has decided to graduate the group into a named Advanced Persistent Threat: APT44. As part of this process, we are releasing a report, “APT44: Unearthing Sandworm”, that provides additional insights into the group’s new operations, retrospective insights, and context on how the group is adjusting to support Moscow’s war aims.

Key Findings 

Sponsored by Russian military intelligence, APT44 is a dynamic and operationally mature threat actor that is actively engaged in the full spectrum of espionage, attack, and influence operations. While most state-backed threat groups tend to specialize in a specific mission such as collecting intelligence, sabotaging networks, or conducting information operations, APT44 stands apart in how it has honed each of these capabilities and sought to integrate them into a unified playbook over time. Each of these respective components, and APT44’s efforts to blend them for combined effect, are foundational to Russia’s guiding “information confrontation” concept for cyber warfare.

APT44 has aggressively pursued a multi-pronged effort to help the Russian military gain a wartime advantage and is responsible for nearly all of the disruptive and destructive operations against Ukraine over the past decade. Throughout Russia’s war, APT44 has waged a high intensity campaign of cyber sabotage inside of Ukraine. Through the use of disruptive cyber tools, such as wiper malware designed to disrupt systems, APT44 has sought to impact a wide range of critical infrastructure sectors. At times, these operations have been coordinated with conventional military activity, such as kinetic strikes or other forms of sabotage, in an attempt to achieve joint military objectives. 

However, as the war has endured, APT44’s relative focus has transitioned away from disruption to intelligence collection. The group’s targets and methods have shifted significantly in the second year of the war, with increasing emphasis placed on espionage activity intended to provide battlefield advantage to Russia’s conventional forces. For example, one long-running APT44 campaign has assisted forward-deployed Russian ground forces to exfiltrate communications from captured mobile devices in order to collect and process relevant targeting data. APT44’s approach to supporting Russia’s military campaign has evolved considerably over the past two years.

We assess with high confidence that APT44 is seen by the Kremlin as a flexible instrument of power capable of servicing Russia’s wide ranging national interests and ambitions, including efforts to undermine democratic processes globally.  

Despite being an arm of Russia’s military, the group’s sabotage activity is not limited to military objectives and also spans Russia’s wider national interests, such as driving the Kremlin’s political signaling efforts, responses to crises, or intended non-escalatory responses to perceived slights to Moscow’s stature in the world.  

APT44’s support of the Kremlin’s political objectives has resulted in some of the largest and most consequential cyber attacks in history. These operations include first-of-their-kind disruptions of Ukraine’s energy grid in the winters of 2015 and 2016, the global NotPetya attack timed to coincide with Ukraine’s Constitution Day in 2017, and the disruption of the opening ceremony of the 2018 Pyeongchang Olympics in response to Russia’s doping ban from the games, to name a few. 

Due to its history of aggressive use of network attack capabilities across political and military contexts, APT44 presents a persistent, high severity threat to governments and critical infrastructure operators globally where Russian national interests intersect. The combination of APT44’s high capability, risk tolerance, and far-reaching mandate to support Russia’s foreign policy interests places governments, civil society, and critical infrastructure operators around the world at risk of falling into the group’s sights on short notice. 

We also judge APT44 to present a significant proliferation risk for new cyber attack concepts and methods. Continued advancements and in-the-wild use of the group’s disruptive and destructive capabilities has likely lowered the barrier of entry for other state and non-state actors to replicate and develop their own cyber attack programs. Russia itself is almost certainly alert to and concerned about this proliferation risk, as Mandiant has observed Russian cybersecurity entities exercise their ability to defend against categories of disruptive cyber capabilities originally used by APT44 against Ukraine.  

Looking Ahead

APT44 will almost certainly continue to present one of the widest and highest severity cyber threats globally. It has been at the forefront of the threat landscape for over a decade and is responsible for a long list of firsts that have set precedents for future cyber attack activity. Patterns of historical activity, such as efforts to influence elections or retaliate against international sporting bodies, suggest there is no limit to the nationalist impulses that may fuel the group’s operations in the future.

As Russia’s war continues, we anticipate Ukraine will remain the principal focus of APT44 operations. However, as history indicates, the group’s readiness to conduct cyber operations in furtherance of the Kremlin’s wider strategic objectives globally is ingrained in its mandate. We therefore assess that changing Western political dynamics, upcoming elections, and emerging issues in Russia’s near abroad will also continue to shape APT44’s operations for the foreseeable future.

Protecting the Community

As part of our research, we take various steps to protect customers and the community:

Google’s Threat Analysis Group (TAG) uses the results of our research to improve the safety and security of Google’s products.

Upon discovery, all identified websites and domains are added to Safe Browsing to protect users from further exploitation.
All targeted Gmail and Workspace users are sent government-backed attacker alerts, notifying them of the activity, encouraging potential targets to enable Enhanced Safe Browsing for Chrome, and ensuring them that all devices are updated.

Where possible, Mandiant sends victim notifications via the Victim Notification Program.
If you are a Google Chronicle Enterprise+ customer, Chronicle rules were released to your Emerging Threats rule pack, and IOCs are available for prioritization with Applied Threat Intelligence.
A VirusTotal Collection featuring APT44-related indicators of compromise is now available for registered users.

We are committed to sharing our findings with the security community to raise awareness, and with companies and individuals that might have been targeted by these activities. 

Read the APT44 report for our full analysis of this group, a detailed list of malware used by APT44 since 2018, hunting rules for detecting the malware, and a list of Mandiant Security Validation actions organizations can use to validate their security controls.