Cloud

Google Cloud is delighted to introduce the latest offering in the area of SAP HANA Scale-out high availability deployment, a new automated deployment guide powered byTerraform Google Cloud Platform Provider.

Scaling out refers to the process of combining multiple independent Compute Engine instances to create a unified SAP system. This approach is primarily adopted to overcome the hardware limitations associated with fitting an ERP (Enterprise Resource Planning) or Business Warehouse (BW) system into a single large virtual machine.

Earlier this year, we released a comprehensive step-by-step, manual deployment guidethat provided detailed instructions for setting up SAP HANA Scale-out, ensuring high availability through utilizingSUSE Linux Enterprise Server for SAP Applications andSUSE Linux Enterprise High Availability Extension. This guide served as a valuable resource for our joint clients.

Today, we are pleased to present an enhanced option that further simplifies the deployment process and streamlines the implementation of SAP HANA Scale-out high availability, allowing for increased efficiency and productivity.

A robust SAP HANA scale-out high availability cluster for uninterrupted operations

Customers likePayPal have successfully implemented theworld’s largest SAP HANA scale-out high availability cluster, sized at 96TB, hosted on robust Google Cloud infrastructure.

To attain optimal uptime and ensure continuous operations, a best practice is to deploy two SAP HANA systems that are replicated across separate zones within a region. By distributing the SAP HANA systems across different zones within the same region, any localized outages in one zone will not impact the availability of the entire system. This approach guarantees high availability and mitigates potential risks while also accommodating the expansion of SAP capabilities.

Advantages of choosing the SAP HANA Scale-out HA solution on Google Cloud

While Google already providesextra-large Bare Metal Solutions tailored for SAP workloads for scale-up scenarios, hosting your workloads in Compute Engine instances presents numerous advantages:

Horizontal scalability, making it easier to expand your infrastructure as your needs grow.

Transparent host maintenance throughlive migration, which ensures uninterrupted operations and minimal disruptions.

A broader range ofregions and zones for deployment, affording you greater flexibility and options when configuring your infrastructure.

Currently, Google Cloud supports scale-out configuration with up to 12TB memory, which PayPal uses as part of its deployment. As part of the commitment to enhancing services, we plan to extend scale-out support to even larger machines, providing customers with even more scalability and performance options to meet their evolving business needs.

Overview of the solution architecture

When it comes to SAP high-availability solutions operating on Google Cloud, SUSE-based offerings are a go-to choice for many partners and customers, gaining widespread adoption among enterprises. And like in life, good things come in threes:

The high availability layer of the SAP HANA scale-out solution is managed by the SUSE High Availability Extension, which leverages the power of the Pacemaker open-source software framework. Pacemaker automates HANA scale-out failover, letting you tackle planned and unplanned downtime challenges.

To ensure optimal performance and compatibility, the solution is built upon the SUSE Linux Enterprise Server for SAP Applications, anSAP Endorsed Application. This signifies that the solution meets SAP’s standards for seamless integration and optimal performance within SAP environments. The preloaded saptune tool is a valuable configuration tool for optimizing system performance with SAP workloads. It simplifies the task for system administrators by implementing the recommended configurations from diverse SAP notes.

The solution harnesses the robustness and capabilities of the Google Cloud infrastructure ecosystem.

A strategic partnership for SUSE and Google Cloud

SUSE has been at the forefront of collaboration with Google Cloud, delivering groundbreaking solutions for SAP HANA scale-up, scale-out, NetWeaver, and S/4HANA high-availability solutions on Google Cloud.[1]

As an industry leader in this space, SUSE continues to drive innovation by optimizing SUSE software performance. SUSE has implemented three HA/DR provider hook scripts that bolster the reliability and integrity of an SAP environment. These hook scripts play a crucial role in minimizing downtime, mitigating the risk of data loss, and optimizing monitoring intervals for SAP HANA. Refer to the Appendix below for more details.

Notably, SUSE introduced a newFencing Agent and Floating IP resource agent for SAP NetWeaver and S/4HANA high-availability environments, which will soon also be available for SAP HANA deployments. These components significantly improve the resiliency and stability of SAP environments, helping to ensure uninterrupted operations and minimizing potential disruptions. To ensure a seamless experience for our clients, all of these software components and enhancements are accessible through SUSE enterprise-grade software repositories.

SLES image for for SAP applications on Google Cloud Marketplace

All the software and features from SUSE mentioned above are conveniently preloaded in the Pay-As-You-Go (PAYG) image of the SUSE Linux Enterprise Server for SAP Applications, available from the Google Cloud Marketplace. By opting for this image and signing a three-year contract, you can significantly reduce your operational expenses related to the operating system, with potential savings of up to 63%.[2]

As an added advantage, customers with ten or more instances of SUSE Linux Enterprise Server for SAP Applications in the PAYG model can benefit from additional complimentary SUSE subscriptions that provide unrestricted access to:

1. SUSE Manager: An infrastructure management solution that manages, simplifies, and secures your entire Linux environment.

2. SUSE Linux Enterprise Live Patching: Apply patches and critical updates to your Linux kernel that hosts your SAP environment without rebooting your system.

APPENDIX

The table provided below presents a concise summary of the functionality of each HA/DR script:

Footnotes

[1] Supported High Availability Solutions by SLES for SAP Applications
[2] Resource-based committed use discounts
[3] Emergency Braking for SAP HANA Dying Indexserver
[4] SAP HANA Cockpit with SUSE HA integration greatly improves data integrity

In this blog we will give you an overview of the manifest support for BigQuery and also explain how it enables querying open table formats like Apache Hudi and Delta Lake in BigQuery.

Open table formats rely on embedded metadata to provide transactionally consistent DML and time travel features. They keep different versions of the data files and are capable of generating manifests, which are lists of data files that represent a point-in-time snapshot. Many data runtimes like Delta Lake and Apache Hudi can generate manifests, which can be used for load and query use cases. BigQuery now supports manifest files, which will make it easier to query open table formats with BigQuery.

BigQuery supports manifest files in SymLinkTextInputFormat, which is simply a newline-delimited list of URIs. Customers can now set the file_set_spec_type flag to NEW_LINE_DELIMITED_MANIFEST in table options to indicate that the provided URIs are newline-delimited manifest files, with one URI per line. This feature also supports partition pruning for hive-style partitioned tables which leads to better performance and lower cost. 

Here is an example of creating a BigLake table using a manifest file.

Querying Apache Hudi using Manifest Support 

Apache Hudi is an open-source data management framework for big data workloads. It’s built on top of Apache Hadoop and provides a mechanism to manage data in a Hadoop Distributed File System (HDFS) or any other cloud storage system.

Hudi tables can be queried from BigQuery as external tables using the Hudi-BigQuery Connector. The Hudi-BigQuery integration only works for hive-style partitioned Copy-On-Write tables. The implementation precludes the use of some important query processing optimizations, which hurts performance and increases slot consumption.

To overcome these pain points, the Hudi-BigQuery Connector is upgraded to leverage BigQuery’s manifest file support. Here is a step by step process to query Apache Hudi workloads using the Connector.

Step 1: Download and build the BigQuery Hudi connector

Download and build the latest hudi-gcp-bundle to run the BigQuerySyncTool. 

Step 2: Run the spark application to generate a BigQuery external table

Here are the steps to use the connector using manifest approach:

Drop the existing view that represents the Hudi table in BigQuery [if old implementation is used]

The Hudi connector looks for the table name and if one exists it just updates the manifest file. Queries will start failing because of a schema mismatch. Make sure you drop the view before triggering the latest connector.

Run the latest Hudi Connector to trigger the manifest approach

Run the BigQuerySyncTool with the –use-bq-manifest-file flag.

If you are transitioning from the old implementation, append –use-bq-manifest-file flag to the current spark submit that runs the existing connector. Using the same table name is recommended as it will allow keeping the existing downstream pipeline code.

Running the connector with the use-bq-manifest-file flag will export a manifest file in a format supported by BigQuery and use it to create an external table with the name specified in the –table parameter.

Here is a sample spark submit for the manifest approach.

Step 3: Recommended: Upgrade to an accelerated BigLake table

Customers running large-scale analytics can upgrade external tables to BigLake tables to set appropriate fine-grained controls and accelerate the performance of these workloads by taking advantage of metadata caching and materialized views. 

Querying Delta Lake using Manifest Support 

Delta Lake is an open-source storage framework that enables building a lakehouse architecture. It extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling. It also provides an option to export a manifest file that contains a list of data files that represent the point-in-time snapshot. 

With the manifest support, users can create a BigLake table to query the Delta Lake table on GCS. It is the responsibility of the user to generate the manifest whenever the underlying Delta Lake table changes and this approach only supports querying Delta Lake reader v1 tables.

Here is a step by step process to query Delta Lake tables using manifest support.

Step 1: Generate the Delta table’s manifests using Apache Spark

Delta Lake supports exporting manifest files.  The generate command generates manifest files at <path-to-delta-table>/_symlink_format_manifest/. The files in this directory will contain the names of the data files (that is, Parquet files) that should be read for reading a snapshot of the Delta table.

Step 2: Create a BigLake table on the generated manifests 

Create a manifest file based BigLake table using the manifest files generated from the previous step.  If the underlying Delta Lake table is partitioned, you can create a hive style partitioned BigLake table.

Step 3: Recommended: Upgrade to an accelerated BigLake table

Customers running large-scale analytics on Delta Lake workloads can accelerate the performance by taking advantage of metadata caching and materialized views. 

What’s Next?

 If you are an OSS customer looking to query your Delta lake or Apache Hudi workloads on GCS, please leverage the manifest support and if you are also looking to further accelerate the performance, you can do that by taking advantage of metadata caching and materialized views. 

For more Information

Accelerate BigLake performance to run large-scale analytics workloads

Introduction to BigLake tables.

Visit BigLake on Google Cloud.

Acknowledgments: Micah Kornfield, Brian Hulette, Silvian Calman, Mahesh Bogadi, Garrett Casto, Yuri Volobuev, Justin Levandoski, Gaurav Saxena and the rest of the BigQuery Engineering team.

This comprehensive blog presents various approaches for monitoring, troubleshooting, and minimizing DAG parse times, leading to notable performance improvements in Cloud Composer / Airflow:

Increase environment scalability by efficiently handling larger workloads and accommodating more DAGs.

Improve environment stability by limiting the chance of task overlaps and resource contention. 

Enhance productivity and overall efficiency for developers through faster feedback loops and reduced processing time.

A low DAG parse time serves as a reliable indicator of a healthy Cloud Composer / Airflow environment

Getting started

What is an Airflow DAG?

An Airflow DAG (Directed Acyclic Graph) is a collection of tasks that are organized in a way that reflects their relationships and dependencies. DAGs are defined in Python scripts, and they are the core concept of Airflow.

A DAG defines four things:

The tasks that need to be run

The order in which the tasks need to be run

The dependencies between the tasks

The schedule for running the tasks

DAGs are a powerful way to define and manage complex workflows. They can be used to automate tasks, schedule tasks, and monitor the execution of tasks.

What is the Airflow Scheduler?

The Airflow Scheduler monitors all tasks and DAGs, then triggers the task instances once dependent tasks are complete. Once every 30 seconds by default, the Scheduler collects DAG parsing results and checks whether any active tasks can be triggered.

What is the DAG Processor? 

As of Airflow 2.3.0, the DAG Processor is separate from the Airflow Scheduler. For more information about this change, check out AIP-43 DAG Processor separation.

Monitoring and alerting

Monitoring DAG parse times

In Google Cloud console you can use the Monitoring page and the Logs tab to inspect DAG parse times. 

On Cloud Composer environment

Run the following commands to check DAG parse times on the Cloud Composer environment.:

Locally using time command

Make sure to run it several times in succession to account for caching effects. Compare the results before and after the optimization (in the same conditions – using the same machine, environment etc.) in order to assess the impact of any optimization.

Sample output:

The important metric is the “real time” – which tells you how long time it took to process the DAG. Note that when loading the file this way, you are starting a new interpreter so there is an initial loading time that is not present when Airflow parses the DAG. You can assess the time of initialization by running:

Result:

In this case the initial interpreter startup time is ~ 0.07s which is about 10% of time needed to parse the example_python_operator.py above so the actual parsing time is about ~ 0.62 s for the example DAG.

What is an ideal parse time metric?

On the Monitoring dashboard, in the DAG Statistics section, observe graphs for the total DAG parse time. If the number exceeds about 10 seconds, your Schedulers might be overloaded with DAG parsing and cannot run DAGs effectively.

How can I receive alerts for long parse times?

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition. This can also be done through the Composer Monitoring Dashboard.

DAG code optimization

Generalized DAG code improvements

Check out Optimize Cloud Composer via Better Airflow DAGs to view a generalized checklist of activities when authoring Apache Airflow DAGs.  These items follow best practices determined by Google Cloud and the open source community.  A collection of performant DAGs will enable Cloud Composer to work optimally and standardized authoring will help developers manage hundreds or thousands of DAGs.  Each item will benefit your Cloud Composer environment and your development process. The two highest priorities should be limiting top-level code and avoiding the use of variables/xcoms in top-level code.

Limit top-level code

Follow established best practices. You should avoid writing the top level code which is not necessary to create Operators and build DAG relations between them. This is because of the design decision for the Scheduler of Airflow and the impact the top-level code parsing speed on both performance and scalability of Airflow.

One of the important factors impacting DAG loading time, that might be overlooked by Python developers is that top-level imports might take surprisingly a lot of time (in the order of seconds) and they can generate a lot of overhead and this can be easily avoided by converting them to local imports inside Python callables for example.

Avoid the use of Variables and Xcoms in top-level code

If you are using Variable.get() in top level code, every time the .py file is parsed, Airflow executes a Variable.get() which opens a session to the DB. This can dramatically slow down parse times. 

Use JSON dictionaries or Jinja templates as values if absolutely necessary. (one connection for many values inside dict)

DAG folder cleanup

Remove unused DAGs, unnecessary files from the DAGs folder

Airflow Scheduler wastes time and resources parsing files in DAGs folder that aren’t used.

Use .airflowignore

 An .airflowignore file specifies the directories or files in DAG_FOLDER or PLUGINS_FOLDER that Airflow should intentionally ignore. Airflow supports two syntax flavors for patterns in the file, as specified by the DAG_IGNORE_FILE_SYNTAX configuration parameter (added in Airflow 2.3): regexp and glob.

More files ignored = less files parsed by Airflow Scheduler.

Review paused DAGs

Paused DAGs are still continuously parsed by the Airflow Scheduler.  Determine why each DAG is paused and whether it should be removed, ignored, or unpaused.

Airflow configurations

min_file_process_interval

The Scheduler parses your DAG files every min_file_process_interval number of seconds. Airflow starts using your updated DAG code only after this interval ends.

Consider increasing this interval when you have a high number of DAGs that do not change too often, or observe a high Scheduler load in general. Consider decreasing this interval to parse your DAGs faster. Updates to DAGs are reflected after this interval. Keeping this number low will increase CPU usage.

For example, if you have >1000 dag files, raise the min_file_process_interval to 600 (10 minutes), 6000 (100 minutes), or a higher value.

dag_dir_list_interval

Dag_dir_list_interval determines how often Airflow should scan the DAGs directory in seconds. A lower value here means that new DAGs will be processed faster, but this comes at the cost of CPU usage.

Increasing the DAG directory listing interval reduces the Scheduler load associated with discovery of new DAGs in the environment’s bucket. Consider increasing this interval if you deploy new DAGs infrequently. Consider decreasing this interval if you want Airflow to react faster to newly deployed DAG files.

parsing_processes

The DAG Processor can run multiple processes in parallel to parse DAGs, and parsing_processes (formerly max_threads) determines how many of those processes can run in parallel. Increasing this value can help to serialize DAGs if you have a large number of them. By default, this is set to 2.

file_parsing_sort_mode 

Evaluate the following file_parsing_sort_mode options if you are running more than one Airflow Scheduler. The Scheduler will list and sort the dag files to decide the parsing order.

modified_time: Sort by modified time of the files. This is useful on a large scale to parse the recently modified DAGs first. (default)

random_seeded_by_host: Sort randomly across multiple Schedulers but with the same order on the same host. This is useful when running with Scheduler in HA mode where each Scheduler can parse different DAG files.

alphabetical: Sort by filename

When there are a lot (>1000) of dags files, you can prioritize parsing of new files by changing the file_parsing_sort_mode to modified_time.

Cloud Composer upgrades

If you’ve gotten this far and still observe long DAG parse times, you’ll need to consider adding more resources to your Cloud Composer Environment. Note: this will add to the overall cost of your Cloud Composer environment.

Change/Increase the number of Airflow Schedulers

Adjusting the number of Schedulers improves the Scheduler capacity and resilience of Airflow scheduling. Caution: Don’t configure more than three Airflow Schedulers in your Cloud Composer environment without special consideration.

If you increase the number of Schedulers, this increases the traffic to and from the Airflow database. We recommend using two Airflow Schedulers in most scenarios.

Increase CPU/Memory of Airflow Schedulers

You can specify the amount of CPUs, memory, and disk space used by your environment. In this way, you can increase performance of your environment, in addition to horizontal scaling provided by using multiple workers and Schedulers.

Conclusion

By following these next steps, you can maximize the benefits of Cloud Composer / Airflow, enhance the performance of your environment, and create a smoother development experience.

Memorystore is a fully managed caching service that supports the open-source in-memory databases Redis and Memcached. Memorystore makes it easy for applications built on Google Cloud to leverage in-memory data stores based on open-source Redis and Memcached. With a single click, Memorystore for Redis offers a highly available, zonally redundant Redis instance with automatic fast failover. 

As more users depend on Memorystore, it is important for Memorystore to support the latest open-source software versions. Today, we’re excited to announce that Memorystore now supports Redis 7.0, which brings the following features and improvements: 

Introduction of Redis Functions 

Improved in-transit encryption (TLS) performance 

You can now create new Redis instances for Memorystore with the Redis 7.0 version. You can also upgrade your Memorystore instances to Redis 7.0 by following these instructions. As always, we suggest reviewing the version release notes to ensure that there are no breaking changes before performing an upgrade of your existing instance. Redis 7.0 on Memorystore is generally available.  

What’s new in Redis 7.0 on Memorystore

Redis Functions 

Redis Functions was introduced in version 7.0 as an evolutionary successor for Lua scripting via the EVAL command. Functions enable the scripting logic to be stored in Memorystore rather than requiring application developers to resend script logic with every EVAL command. This new design provides benefits such as 

reduced networking bandwidth overhead from continuously sending script logic

removing the need for all client applications to maintain a copy of all scripts

simplified troubleshooting of errant scripts. 

You can learn more about Redis Functions in the official documentation.

Improved in-transit encryption performance

We’ve also further optimized in-transit encryption performance for all Redis 7.0 instances. Memorystore for Redis 7.0 in-transit encryption delivers up to 5x the throughput and 85% lower latency when compared to prior versions on the service. In addition, instances using in-transit encryption and Redis 7.0 are now able to support a maximum connection limit of 65,000 for all instance capacity tiers. Instances using in-transit encryption on older Redis versions are still subject to lower maximum connection limits. We strongly recommend that existing Memorystore in-transit encryption users upgrade their instances to Redis 7.0 to realize these significant performance improvements at no additional cost. 

Next steps 

We are thrilled to offer support for Redis 7.0 and will continue to add support for the latest Redis version releases. For more information on how to upgrade your instance, just see the documentation. We also recommend reviewing our best practices which can be found here.

Last year, we introduced Managed Service for Prometheus so that you can scale your Prometheus environment more easily that hosting it yourself. As part of that announcement, we also added support for Prometheus’ popular PromQL query language in Cloud Monitoring. Since then, we’ve heard from experienced and new Prometheus users that they’d like to manage alerts in a single ecosystem. Today, we’re excited to announce that Cloud Monitoring now supports alerting using PromQL in Public Preview. You can now create globally scoped alerting policies based on PromQL queries alongside your Cloud Monitoring metrics and dashboards, without having to maintain backend services. 

In this release, you can:

Write globally scoped PromQL-based alerting policies in Cloud Monitoring 

Reference Prometheus, GCP system, and custom metrics in your alerting policy 

Route the notification to any Cloud Monitoring-supported notification channel. Use Email, Slack, SMS, and Mobile push to send the notification to your team members. Use Webhooks to send the notification to any public endpoint or Pub/Sub for any private endpoint. 

Customize the subject line in an Email notification channel 

Easily migrate your existing Prometheus alert_rules to Cloud Monitoring 

Manage your configuration with Terraform 

If you already have Prometheus alert rules, then you can migrate them to Cloud Monitoring alerting policies containing a PromQL query. You can also create your own PromQL alerting policies directly in Cloud Monitoring by using the Monitoring API or gCloud CLI.

Understanding alerting policies 

The following Prometheus alert rule triggers if your Kuberentes volume is at 90% of the available disk space. Let’s review a few of its fields:

Alert: Sets a name for the alert to help users identify what’s happening

Expr: The PromQL expression to evaluate. In this case, if the volume’s capacity exceeds 90%, then an alert will fire.  

For: Specifies the length of time during which each evaluation of the query must generate a `true` value before the alert fires.

Summary: The customized subject line to be used in the alert.

The labels and annotations fields provide additional information about the alert and can be used for passing additional context or actions.

Migrate alert rules to Cloud Monitoring 

For most Prometheus users, alert rules are stored in a Prometheus config file or rules file. You can now migrate these files to Cloud Monitoring alerting policies with PromQL queries. This way, you can store all of your alert policies in one place. Let’s look at a sample migration process:

The following PromQL expression sends an alert if any instance latency spikes above the 95th percentile during the weekdays:

The following Prometheus rules file sample shows the latency evaluation query and several other important fields:

alerting_rules.yaml

You can migrate this alert rule by providing it to the gCloud migration command:

Cloud Monitoring then creates an alerting policy that contains the PromQL query:

These examples show how easy it is to get started with Cloud Monitoring’s new PromQL Alerting capabilities. You can migrate your existing Prometheus alert rules to Cloud Monitoring alert policies, or create PromQL alerting policies in Cloud Monitoring with the API or CLI.  

If you want to install alerts locally in your clusters and have your clusters route them to a Prometheus Alert Manager, then you can continue to use the Managed Service for Prometheus rule evaluation options for managed or self-deployed collection. Otherwise, you can reduce the time you spend on alert management by migrating your existing Prometheus alert rules to Cloud Monitoring alerting policies.

To learn more, check out our documentation:

For a general overview of PromQL Alerting, including a list of migration options and alerting rule-to-alerting policy field mapping, see Alerting policies with PromQL.

For detailed information about how to migrate alerting rules and receivers with the Google Cloud CLI, see Migrate alerting rules and receivers from Prometheus.

For a walkthrough of how to use Cloud Monitoring API to create alerting policies with a PromQL query, including several examples, see Create alerting policies with a PromQL query.

As always, please leave use feedback during the preview so we can improve the experience!

At the Cyber Week conference in June, we were excited to announce our strategic partnership with the Israel National Cyber Directorate (INCD) to modernize security operations under the wider scope of Israel’s own Cyber Dome mission. Our teams worked together to leverage Google Cloud’s Chronicle Security Operations platform, which includes automation, analytics, threat intelligence, and AI to create a multi tier national solution that includes Israel’s national security operations center (SOC), sector SOCs, and participating organizations. 

“Combining the right people with the right technology is how you do cyber defense the right way,” said Gaby Portnoy, Director General of the Israel National Cyber Directorate. “The Israel National Cyber Directorate, which leads Israel’s national cyber activities, is pleased to enter into this strategic partnership with Google Cloud and Mandiant to create the next generation of national level security solutions and technology, and to help build better cyber immunity.”

Our partnership with INCD over the last year helped us develop unique offerings for governments based on our Chronicle Security Operations platform. Today, threats are on the rise as attackers use new tactics, techniques, and procedures augmented by AI and automation to carry out cheaper and more efficient attacks. Governments and private organizations around the world combat adversaries primarily on their own, with little coordination. Mutual defense is organized — at best — through chat and email. 

As we know, threat actors often use the same tactics and techniques repeatedly against different organizations — until enough organizations have developed protections that the attackers move on to new techniques. Although defenders outnumber bad actors, today’s reality creates an asymmetric advantage for attackers. This has to change. 

Building on our partnership with INCD, we enhanced our offering and developed a cybersecurity solution tailored specifically for governments, which we are proud to announce today: Chronicle CyberShield, inspired by the concept that organizations inside a nation and around the world need to “lock shields” and stand united in their defense against shared attackers. 

“At Google Cloud, we are laser-focused on improving cybersecurity and doing our part to support collective defense against malicious threats,” said Kevin Mandia, CEO of Mandiant, Google Cloud. “By bringing together Mandiant frontline threat intelligence and expertise with Google Cloud technology, we’re able to help governments around the world enhance their security capabilities and respond holistically at the national level to keep their citizens and critical assets safe.”

Chronicle CyberShield is a comprehensive solution that provides technology, processes, capabilities, and resources to deliver unparalleled situational awareness of the threat landscape. CyberShield can help improve national security at scale, accelerate innovation, and support continuous cyber defense. Chronicle CyberShield is organized around three pillars:

Government SOC: Chronicle Security Operations’ unique approach to speed, scale, and intelligence allows simple onboarding and maintenance for a number of interconnected SOCs across public and/or private organizations. With Chronicle, security teams can continuously feed unlimited data to the government SOC for analysis and hunting, and push curated detections to protect different sectors or even the entire nation from new and novel threats. Powered by unparalleled AI and threat intelligence from Google, VirusTotal, and Mandiant, governments can help prevent threats from spreading beyond the first infection, reduce the toil of repetitive tasks, and pair novices and security experts with AI expertise to make it easier to “do” security.

Digital Security: CyberShield’s unique security capabilities can help secure a nation’s digital assets and infrastructure, providing defense in depth at scale to mitigate malware, cryptomining, DDoS, bot attacks, and more. 

Capability and Intelligence Excellence: Google Cloud’s professional services and Mandiant’s government consulting solutions and expertise can help organizations identify and research threats, enhance knowledge sharing and collaboration, and build the skills required to action Mandiant frontline threat intelligence and a nation’s own threat intelligence to quickly and easily identify and respond to threats.

For more information, please read our in-depth look at Chronicle CyberShield, or reach out to one of our security experts. 

One of the primary functions of any government is to protect its citizens, institutions, infrastructure and way of life. With the rise of the global Internet, the world is more connected and traditional borders do not exist, meaning those same citizens, institutions, infrastructure, and way of life are at greater risk of malicious activity online. The threat profile of many governments has evolved and it is more important than ever to protect and defend critical online services.

Of the intrusions investigated byMandiant in 2022, response efforts for government-related organizations captured 25% of all investigations, compared to 9% in 2021. This primarily reflects the extensive work Mandiant has conducted in support of customers affected by the Russian invasion of Ukraine.

To help governments around the world continue to realize the benefits of digital transformation while mitigating the risk of cyber threats, we’ve developed Chronicle CyberShield to  provide government agencies with a solution that integrates threat intelligence, detection, and response. Chronicle CyberShield is unique in that it enables multiple government entities to proactively and rapidly share threat information, accelerate investigations and initiate a united response. 

Situational threat awareness at a national level 

Governments need to invest in improving their cybersecurity capabilities and cultivate a collaborative culture of enhanced information-sharing and threat awareness at scale. They need to reduce the impact and severity of cyber attacks on critical national infrastructure and develop capabilities to secure the networks that support it. Further, as governments adopt the cloud to accelerate innovation and drive repeatable outcomes, they need to ensure that it’s secure and reliable. Lastly, and most importantly, government entities need to be empowered with advanced skills and capabilities to defend against an ever-evolving threat landscape.

The large attack surface across a government makes visibility and situational awareness of the threat landscape paramount. Even governments with mature cybersecurity postures are at risk to most advanced persistent threat actors who constantly evolve their techniques. As a result, rapid aggregation of security events and real-time sharing of actionable cyber threat intelligence widely across the government sector is necessary to prevent widespread cyber incidents. 

Chronicle CyberShield

Chronicle CyberShield enables governments to build an enhanced cyber threat intelligence capability; protect web-facing infrastructure from cyber attacks; monitor and detect indicators of compromise, malware, and intrusions; and rapidly respond to cyber attacks to limit widespread impacts. In addition, it enables governments to raise threat and situational awareness, build cybersecurity skills and capabilities, and facilitate knowledge sharing and collaboration to raise the bar for security at a national level.

Improving situational threat awareness with a modern security operations center

In the digital world, operating a sophisticated and streamlined Security Operations Center (SOC) is at the core of maintaining digital integrity and security. A primary component of Chronicle CyberShield is establishing a modern government SOC, comprising a network of interconnected SOCs to scale and aggregate security threats. This empowers governments to operate a cyber defense center for enhanced detection, protection against major threats, and automated response and incident management across multiple entities. 

As part of Chronicle CyberShield, governments can leverage cyber threat intelligence from Google and Mandiant, now part of Google Cloud, to build a scalable and centralized threat intelligence and analysis capability. This is integrated operationally into the government SOC to identify suspicious indicators and enrich the context for known vulnerabilities.

In addition, Chronicle CyberShield allows governments to build a coordinated monitoring capability with Chronicle SIEM to simplify threat detection, investigation, and hunting with the intelligence, speed, and scale of Google. By implementing Chronicle across a network of SOCs, attack patterns and correlated threat activity across multiple entities is available for investigation and analysis. With Chronicle’s cloud-focused scalable architecture and innovative pricing model, governments can analyze large volumes of security telemetry within seconds without sacrificing visibility, performance, or costs.      

Once threats are identified in Chronicle SIEM, automated playbooks can be developed in Chronicle  SOAR to address root causes and reduce the impact of threats and cyber attacks. Integration with third party solutions enables Chronicle SOAR to enrich data with threat intelligence and additional context to get faster insights. Analysts in the government SOC can focus on resolving cases faster and reducing dwell time by uncovering threats faster and containing them more rapidly. 

When major cyber attacks take place, time is of essence to clearly understand the scope and magnitude of impact. Governments need additional support to augment their in-house capabilities to respond to the full lifecycle of any major security incident. With Chronicle CyberShield, governments can agree on pre-established terms and conditions for incident management and response support from Mandiant, saving precious time when it matters the most.   

Lastly, staying ahead of attackers requires continuous validation to strengthen detection and response capabilities. Governments need to continuously test security controls by launching real-world attacks against critical assets to identify vulnerabilities and harden systems. Chronicle CyberShield includes continuous red teaming and penetration testing services delivered by Mandiant to test security controls and protect critical assets by identifying and mitigating security gaps and vulnerabilities.

By continuously assessing security controls and capabilities, governments can rapidly identify and respond to threats. This results in heightened situational awareness and prepares teams to mobilize quickly in response to major threats.

Protect web applications from cyberattacks

In addition to monitoring and responding to threats, Chronicle CyberShield provides governments with the capability to protect web applications from large-scale cyber attacks. With the Digital Security component of Chronicle CyberShield, governments can integrate with existing solutions and build anti-DDoS, anti-bot, web application firewall (WAF), and API protection to protect against new and existing threats. 

Cloud Armor protects applications from DDoS attacks and mitigates against OWASP Top 10 risks. Integration with reCAPTCHA Enterprise identifies fraudulent activity, spam, and abuse like scraping, credential stuffing, automated account creation, and exploits from automated bots. Lastly, applications and APIs are secured using Apigee API management.

Defend against tomorrow’s attacks today

CyberShield includes consulting services from Google Cloud and Mandiant to further assist governments.

By leveraging Google Cloud’s professional services and Mandiant’s government consulting solutions and expertise, governments can develop core capabilities to improve security governance, upskill talent in government, enhance knowledge sharing and collaboration, and drive effective security operations. Governments can benchmark their capabilities against our National Cybersecurity Capability Framework and establish an Advanced Skills Academy with instructor-led and web-based training on cybersecurity topics including cloud security fundamentals, threat modeling, and secure architecture design. With support from Google Cloud and Mandiant, governments can run cyber attack simulations and table top exercises to test existing controls and be well-prepared for future cyber attacks.

In summary, with Chronicle CyberShield governments will be able to enhance situational threat awareness across a network of interconnected SOCs powered by the speed, scale and performance of Chronicle sec ops suite. In addition, governments get advanced protection for web apps, services, and APIs against DDoS, L7 and bot attacks at a Google scale. Lastly, Chronicle CyberShield empowers governments with the resources to improve security governance, build skills and make strategic decisions to protect the nation. 

To learn more about Chronicle CyberShield, please contact our experts.

Earlier this year we announced the advanced networking demo series. In this blog we will showcase two recent videos that were added to the playlist and what you can expect to learn from them.

Advanced Networking Demo series: Cloud Interconnect

The new videos both touch on the topic of Cloud Interconnect, which allows you to provision high-speed stable links between on-premises and Google Cloud environments. 

Cloud Interconnect initially consisted of Dedicated and Partner Interconnects. Then, in May 2023, we  announced an additional member to the family: Cross-Cloud Interconnect. With Cross-Cloud Interconnect, you can establish high-bandwidth direct links between Google Cloud and other cloud providers.

The advanced networking playlist was updated with two videos on the topics, let’s check out what those videos are about.

# 1 Cloud Interconnect – Cross-Cloud Interconnect Demo Google Cloud and Azure

In this video, a diverse team of experts gives you a wide overview of the Cloud Interconnect product. The opening section covers the overall purpose and scope of Cloud Interconnect, and the scale of Google’s global backbone. The second section introduces you to the Cross-Cloud Interconnect offering and talks about its capabilities and enhancements.

In the final section, there’s a hands-on configuration demo that walks you through how to set up a Cross-Cloud Interconnect link between Google Cloud and Azure.

# 2 – Google Cross-Cloud Interconnect demo Ⅱ

This video offers another focused look at Cross-Cloud Interconnect. The opening section is presented by a developer advocate, providing a quick recap and an overview of the Cross-Cloud interconnect service. Then, the second section is presented by a networking specialist and focuses on console configuration, walking you through the steps to provision a Cross-Cloud Interconnect 10GB link between Google Cloud and AWS.

What’s next?

Subscribe to the Google Cloud Tech YouTube channel to get updates on all our awesome content. Check out and bookmark the Advanced Networking demo series playlist as new videos will be uploaded in the future.

To learn more about configuring Cross-Cloud Interconnect, check out the documentation on the setup process.  Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

In just a few days, Google Cloud Next returns to San Francisco as a large, in-person, three-day event. There, you’ll learn all about the technologies you need to build, connect, and secure all your cloud-first, Kubernetes, and AI/ML workloads. You’ll gain hands-on experience on the latest cloud networking and network security technologies, and you’ll expand your peer network. 

If your role involves designing cloud networks, implementing cybersecurity, or you just want to keep your tabs on the latest network connectivity and security trends, Next ‘23 is the place for you. Here is a list of specially curated content for the networking professional. Click on the links to add these sessions to your schedule.

Spotlight Keynote

SPTL 202 What’s next for architects and IT professionals
How do you ensure you’ve got the right infrastructure to power all the applications that enable your business? How are you preparing to innovate with generative AI and the latest in ML? How are you deploying modern apps with containers? How are you addressing your data sovereignty requirements? 

Get your questions answered in this Spotlight Keynote showcasing the latest Infrastructure advancements and hear from customers that are achieving their cost, sustainability, reliability, and security goals on Google’s planet-scale infrastructure.

Breakout sessions

ARC 201 What’s new in cloud networking: AI-optimized infrastructure, ML-powered security, and more
Whether you’re running AI/ML, data and analytics, web, media, and HPC workloads, cloud networking is delivering simplified and more resilient services to help you connect and secure the workloads. Join this session with our product team for the latest innovations in cloud networking and meet Walmart, Sharechat, and Palo Alto Networks over a fireside chat. 

ARC 202 Design secure enterprise networks for a multi-cloud world
With the shift to a hybrid workforce and distributed application deployment across multiple cloud providers, enterprises are having to build complex network architectures for reliable any-to-any connectivity, enabling optimal user-to-app and app-to-app experience. Get a deep dive from our product team on enterprise cloud network design with Priceline, and Palo Alto Networks. 

ARC 203 Elevate end user experience with planet scale Google Cloud CDN
Delivering engaging content while ensuring quality of experience (QoE) at global scale is becoming a critical differentiator. Discover how Google Cloud helps enable scalable content delivery with our planet-scale network and hear from SonyCrunchyroll on how they’re leveraging Google Cloud.

SEC 203 Network security fundamentals: Creating layered network defenses with built-in tools
Security has become a top of mind concern for line-of-business executives and practitioners alike. Learn how to implement network security and enforce security controls and hear from Wix on how they are using Google Cloud’s built-in network security tools.

SEC 301 Innovations for securing workloads with Google Cloud next-generation firewalls
Firewalls are a critical component of any network security. In the cloud, organizations want advanced threat protection from cloud-first and best-of-breed third-party solutions. Hear about the latest in next-generation firewall innovations from Google, and how our customers, McKesson Inc., CoverMyMeds, and Salesforce implemented advanced threat protection to protect their workloads in their organizations.

Network live and in-person

Everyone is excited to be back in person. Support your fellow customer speakers by stopping by the Showcase Theater at Moscone South: 

 8/29 Tue @ 12:30 – 1 PM: How Walmart simplified multi-cloud adoption with Cross-Cloud Interconnect 
Walmart runs a seamless platform across multiple cloud providers to accelerate innovations. Join this session featuring Walmart and Google speakers to learn how Walmart built a global network to connect, secure and consume multi-cloud services through Cross-Cloud Interconnect.

8/30 Wed @ 4:30 – 5 PM: How Broadcom blocks DDoS attacks with Cloud Armor 
Broadcom is a global provider of enterprise security solutions and have migrated their flagship cybersecurity solutions to Google Cloud. Join this session to hear from Symantec/Broadcom on how Cloud Armor with ML-powered protection prevented intense DDoS attacks, and how it continues to mitigate these threats, while strengthening Broadcom’s infrastructure.

Add these exciting sessions to your schedule. We’re looking forward to seeing you in San Francisco!

Cementos Pacasmayo is a leading producer of construction solutions in Northern Peru whose mission is to “build together the future we dream of.”

The company provides its services through a distribution network of more than 3,500 hardware stores and independent retailers. In recent years, Cementos Pacasmayo has transformed its business model to put the customer at the center. In addition to offering high quality cement, it expanded its portfolio with more specialized and eco-efficient construction solutions that support a more environmentally friendly future.

Cementos Pacasmayo is a signatory on the Peru Cement Roadmap 2030 alongside the Inter-American Cement Federation (FICEM) and the Association of Cement Producers of Peru (ASOCEM) in efforts to achieve carbon neutrality by 2050.

The company received an award for its sustainable management for 10 consecutive years and is the only Peruvian cement company in the Dow Jones MILA 2022.

In addition, for the second consecutive year, they are part of the Sustainability Yearbook published by S&P Global, which only selects companies that have obtained excellent results in SAM’s annual Corporate Sustainability Assessment (CSA).

Greater transparency in the cement sector

Cementos Pacasmayo is a long-time user of SAP ERP and has developed more than 50 remote function calls (RFCs) for communication between SAP systems and other applications, including the legacy Data Warehouse. This required hiring external experts in SAP’s ABAP programming language, which was costly and slowed down data analysis. In addition, these RFCs became “black boxes” due to the lack of transparency about precisely which database tables served as sources of information to build them.

In the absence of easy-to-use business analytics tools, managers frequently explored SAP data in spreadsheets. This required a lot of time and manual labor, which in turn led to potential data security issues and errors in formulas and calculations. Meanwhile, sales related to building solutions could be hampered due to a lack of timely information on product availability, as key data was only updated every 24 hours.

In this context, Cementos Pacasmayo built a data lake using BigQuery to improve access and take advantage of data from SAP and other sources, improving decision-making by all business units. The company also migrated its SAP S/4 HANA platform to RISE with SAP on Google Cloud.

Further, the Google Cloud Cortex Framework provides Cementos Pacasmayo with pre-defined BigQuery data modeling to make it quick and easy to analyze data from SAP.

Faster than setting concrete

Cementos Pacasmayo uses a combined solution for a process that stores base tables, creates replicated tables, and synthesizes new tables. The team arranges its data processing pipelines with Cloud Composer and syncs updates to its data lake every five seconds to the raw layer.

“Using the Google Cloud Cortex Framework with a key integration partner allows our staff to quickly generate an instant, on-the-spot look at what the data was like on a given date. This is a business need that arises very frequently,” says Abrahan Puelles Taboada, Data Architect Leader at Cementos Pacasmayo.

Since most of its staff speaks Spanish, the company appreciates that the names of all the fields appear in this language instead of English or German with the simplified view that the Cortex Framework provides to the data modeler. By connecting the data to Looker for analysis and visualization, the tables are also in Spanish.

“Our data modelers save a lot of time because the information is already in a language that all of our collaborators understand,” says Miguel Loayza, Tribe Leader Data & Analytics for Cementos Pacasmayo. “Between the ease of data access and the linguistic advantages of the Cortex Framework, our modelers can interpret results up to 60% faster.”

Franz Zárate, Technical Tribe Lead Data & Analytics for Cementos Pacasmayo, has also received positive feedback from stakeholders across the company. “Before, three people had to work for eight hours to generate a business analysis. Now business users don’t have to wait for computer scientists or data analysts. They can act autonomously to immediately gain valuable insights thanks to automation and rich, centralized information,” Zárate said.

The solution is also cost effective. “When we factor in the cost of the solution and the money saved by eliminating the need to hire specialized ABAP developers, we are saving approximately 66% in terms of total direct costs and realizing greater business benefits,” added Loayza.

Innovation under construction

Google Cloud Cortex Framework provides Cementos Pacasmayo with machine learning (ML) code samples using Vertex AI. The company is excited to work on a pilot for a gen AI-based virtual assistant through Vertex AI and Looker that will expand business user access to AI.

In a pilot project, the company will analyze the volumes of products shipped and track the location of the trucks to examine the punctuality of the deliveries and the efficiency of the routes. The initiative is expected to help increase customer satisfaction and deliverability.

The team also plans to leverage 110 metrics generated from SAP data for an upcoming operations project. Cementos Pacasmayo integrates IoT sensor data from the production area to BigQuery. Through this approach, the company can develop models that optimize the production process and better adjust to factors such as changes in moisture levels or variations in the performance of additives in its products.

“With the ecosystem we have built using the Cortex Framework, SAP, and Vertex AI on Google Cloud, we provide stakeholders across all departments with fast, broad, and deep visibility into data. This allows us to improve product quality, operational efficiency and customer satisfaction with strategies based on improved data,” concludes Puelles.

Learn more about how the Google Cloud Cortex Framework can help drive value for your business.

We are excited to announce new security features, management options, and product integrations for Workforce Identity Federation, our Identity and Access Management offering that allows you to rapidly onboard user identities from external identity providers (IdPs) for direct, secure access to Google Cloud services and resources. 

Workforce Identity Federation is built on an identity federation approach instead of Directory Synchronization, an option which can simplify identity lifecycle management for the cloud by leveraging your existing identity systems.

Goldman Sachs has already empowered their users by enabling access to BigQuery using Workforce Identity Federation. Ming Ng, managing director and technology fellow at Goldman Sachs, explained why Goldman Sachs started using Workforce Identity Federation.

“We are using Workforce Identity Federation to provide flexible workforce access for our Google Cloud environment. Before using Workforce Identity, if we wanted to grant user-level access to our data warehouse in BigQuery, we had to synchronize our user directory with Google Cloud and have users log into Google Cloud,” said Ng. “Workforce Identity Federation enabled us to grant individual fine-grained, user-level access to BigQuery using our existing identity provider without requiring us to onboard our users to Google Cloud. This saved us significant administrative overhead.”

Here’s what’s new in Workforce Identity Federation:

New security and management capabilities

For OpenID Connect (OIDC) providers, Workforce Identity Federation now supports authorization code flow and implicit flow. Authorization code flow is considered to be more secure because tokens are returned from the IdP in a separate, secure backend transaction directly from the IdP to Google Cloud after the user has been authenticated. As a result, code flow transactions support more claims to use for attribute mapping and attribute conditions. Please see our documentation for details.

Customers who use a SAML-based identity provider are now able to leverage SAML token encryption to encrypt the SAML assertions. When configured, Workforce Identity Federation will encrypt the SAML assertions using the public key from the IdP-stored certificate. Encrypting SAML assertions can protect confidential user information and adds an extra layer of security to Workforce Identity Federation. For a step-by-step guide to enable your SAML 2.0 IdP encrypted SAML assertions to be accepted by Workforce Identity Federation, please see our documentation.

In addition to Google Cloud console access, Workforce Identity Federation now supports programmatic access to Google Cloud services and resources through the API and CLI. We added browser-based sign-in with the gcloud CLI to enable you to create a sign-in configuration file, and then either reference the file in calls to gcloud auth login or activate it so that it is used by default. We have documented detailed configuration steps to help you get started.

Expanding Workforce Identity Federation support to new products 

We continue to expand the list of Google Cloud products that support Workforce Identity Federation. Updates include:

Google Kubernetes Engine (GKE) customers are able to use the identities from an OIDC or SAML 2.0 supported external identity provider to manage their GKE environments.

Chroniclesupports Service Provider Initiated SAML SSO for users. With this capability, users can navigate directly to Chronicle. Chronicle issues a request through Workforce identity federation to the third-party IdP.

Cloud Storage supports Workforce Identity Federation customers to use their third-party IdP for authentication to access the major GCS public APIs and console experiences.

Cloud Billing now enables customers using Workforce Identity Federated to access key billing information; including the cost overview page, cost management, cost optimization, and account management via the federated console.

To see all the new Google Cloud services that support Workforce Identity Federation, please refer to our documentation.

We are excited to share that Goldman Sachs will be presenting at Google Cloud Next ‘23 to discuss their use of Workforce Identity Federation. Please join us at “How Goldman Sachs achieved identity-first security using Google Cloud” to hear what they have been able to accomplish. 

You can learn more about Workforce Identity Federation and get started by visiting our webpage and watching this video.

As one of the most innate and ubiquitous forms of expression, speech is a fundamental pillar of human interaction. It comes as no surprise, then, that Google Cloud’s Speech API  has become a crucial tool for enterprise customers, launched to general availability (GA) over six years ago and, now, processing over 1 billion voice minutes each month.

With the Speech API, we’ve been pleased to serve thousands of customers and provide industry-leading speech recognition quality and cost-effective products across a range of industries. We want to constantly evolve our offerings and bring new benefits to organizations, which is why today, we’re excited to announce the GA release of our new Speech-to-Text v2 API. 

Speech-to-Text v2 modernizes our API interface and introduces several new features. It also migrates all of our existing functionality, so you can use the same models and features that you were using in STT v1 or v1p1beta1 APIs. This new version of our API also allows us to take advantage of significant cost savings in our serving path, and as such we are reducing our base price, as well as adding pricing incentives for large workloads and those willing to accept longer turnaround times.

This new infrastructure also allows us to serve a wide variety of new types of models, including Chirp , our latest 2B-parameter large speech model. All of these are Generally Available to Google Cloud Platform customers and users starting today.

Let’s have a more thorough look though at the enhanced features of Speech-to-Text API V2 and illustrate how your business can benefit from our new capabilities:

Expanding Speech-to-Text features with V2 API

Since the official launch of Speech-to-Text API back in 2017, we’ve utilized Google’s global infrastructure to host and monitor our production-facing transcription models. This robust, well-connected network has been the backbone of our offering for all of our customers. 

However, a unified view of our Speech-to-Text service has been a crucial request for our enterprise customers  who need to satisfy data residency and compliance requirements, especially in regulated industries like banking and public sector. We listened carefully to this feedback, and starting today, our Speech-to-Text v2 API supports full regionalization, allowing our customers to invoke identical copies of all our transcription models in the Google Cloud Platform region of their choice. 

In addition to giving users the flexibility to deploy in any region, we are adding a number of new features to help developers build on the API: 

Recognizers: A user-defined named configuration that combines a model identifier, the language-locale of the audio to be transcribed, and the cloud region for the transcription model to run. Once-created, the recognizer can be referenced to every subsequent transcription request, eliminating the need for users to repeatedly define the same configuration parameters. This resourceful implementation of recognizers allows for greater flexibility in authentication and authorization, as users are not longer required to set up dedicated service accounts.

Cloud Logging: Requests performed using a recognizer object automatically support cloud logging by default. Since the recognizers are defined as named entities, customers can partition traffic based on the recognizer of interest or collectively. 

Audio Format Auto-Detection: Instead of having our users analyze and manually define the audio configuration settings to pass in a transcription request, the new Speech-to-Text V2 API detects settings like encoding, sampling rate, and channel count, then automatically populates the request configuration parameters. 

Increasing accuracy at Enterprise Scale with Chirp

As part of our continuous investment in foundational speech models, in March 2023 we released research results for our Universal Speech Mode (USM), a family of state-of-the-art speech models with 2B parameters and support for transcriptions of 300+ languages. In May 2023, at Google I/O, we announced Chirp in Private Preview, the latest version of the USM family, fine-tuned for our Cloud-specific use-cases. 

Chirp is now GA through the Speech-to-Text v2 API. Following extensive testing and feedback from our customers, we are making the power of pre-trained large models accessible through a simple enterprise-grade API surface. Our early adopters have seen major strides in customer engagement, thanks to the market-leading accuracy and language coverage of the new model, and we cannot wait to see what opportunities our enterprise customers will unlock.

Introducing new pricing, tiers, and options

We have heard from customers that price can be just as important as quality for many workloads. That’s why the Speech-to-Text API v2 features totally new pricing. We have lowered the cost of real-time and batch transcription from $0.024 per minute to $0.016 per minute. Additionally, we know that pricing can be a concern for those that have very large transcription workloads. For that reason, we are also introducing standard volume tiers, allowing costs as low as $0.004 per minute. As always, additional discounts are available to those with even larger workloads.

Staying true to our commitment on flexibility and choice in pricing, we noticed that even though the majority of our customers are interested in our models for real-time transcription scenarios, many are also interested in non-real time transcriptions for data at rest. With our new API v2 infrastructure, we can take greater advantage of capacity that goes unused at certain times. With our new Dynamic Batch pricing, we are passing this savings on to customers that are less latency sensitive. Aimed at users who can wait up to 24 hours for transcription results, Dynamic Batch is a new discounted pricing tier that offers 75% lower price per minute for transcription audio relative to our Standard tier.

For more information on Dynamic Batch and all our new STT v2 API pricing, check out our pricing page.

Learn more and start you Speech-to-Text journey

For more information to help get you started on your migration journey from V1 to V2, head over to the detailed documentation and try our walkthroughs to make the most of regionalization and recognizers.

If you are curious to learn more about how to use Chirp, you can get  started by following our tutorial[link] and learn more about researching our latest large speech models here[link]. 

We are very excited to bring all these improvements to our developer ecosystem and provide you with the tools to leverage the power of voice for your businesses, programs, and applications.