Cloud

Scheduling reports in Looker Studio helps teams predictably and reliably distribute insights to business users in the flow of work and workflows where they spend their time. This helps to foster a data culture, making information more actionable and contextual, and enabling cross-team collaboration. Today, we are introducing two new capabilities in Looker Studio scheduling: the ability to apply distinct filter conditions to reports, and to create and manage multiple schedules within a report. These enhancements will help data teams distribute insights efficiently and make them even more targeted.

Schedule Filters lets you apply specific filter values while creating or editing a schedule, ensuring recipients see the intended filtered view in their delivered report. The Filters section of the Schedule delivery window shows all filters applied to the report along with their default values. You can now modify the filter values for a schedule and they will be applied to the schedule delivery. For example, you might schedule a report to multiple regional teams and apply location- or region-based filters for different sets of recipients.

For an already created schedule, you can also now send a one-time instant delivery using the Send now option from the list view of schedules. You can also toggle the state of a schedule between active and inactive if you wish to temporarily pause delivery.

Multiple schedules feature is exclusively available for Looker Studio Pro users

Transform how your team effectively distributes insights and makes it actionable with Looker Studio Pro. For more information, including features, pricing and trial options, learn more.

In our work on the Big Data solutions architecture, BigQuery, and Dataproc teams, we talk to a lot of customers who are interested in migrating all or part of their data warehouse from Apache Hive to BigQuery, but who have hit some speed bumps along the way. To help, we are proud to announce the public GA release of the Hive-BigQuery Connector.

With this open-source connector, you now can let Apache Hive workloads read and write to BigQuery and BigLake tables. The underlying data can be stored either in BigQuery native storage or in open-source data formats on Cloud Storage. Whether you’re fully migrating from Apache Hive to BigQuery or you want both systems to coexist and interact together, this new connector covers a wide range of use cases.

What is the Hive-BigQuery Connector?

If you’ve run Hadoop or Spark workloads on Google Cloud, you should already be familiar with the Cloud Storage Connector and the Apache Spark SQL connector for BigQuery. The former implements the Hadoop Compatible File System (HCFS) API that lets you store and access data files in Cloud Storage, Google Cloud’s highly-scalable and highly-available object storage service. The latter implements the Spark SQL Data Source API to allow reading BigQuery tables into Spark’s dataframes and writing DataFrames back into BigQuery.

Similarly, the Hive-BigQuery connector implements the Hive StorageHandler API to allow Hive workloads to integrate with BigQuery and BigLake tables. While Hive’s execution engine still handles all compute operations such as aggregates and joins, the connector manages all interactions with the data layer in BigQuery, whether the underlying data is stored in BigQuery native storage or in Cloud Storage buckets via a BigLake connection.

The following diagram illustrates how the Hive-BigQuery connector fits in the architecture:

For its part, Apache Hive is one of the most popular open-source data warehouses, and provides an SQL-like interface to query data stored in various databases and file systems that integrate with Apache Hadoop. Over time Hive evolved from using HDFS on-premises as its exclusive data storage layer, to using cloud storage services. And now, thanks to this new connector, Hive integrates with native storage solutions like BigQuery, simplifying migration.

Migrating a data warehouse to the cloud is a complex process, but it can offer significant benefits, including:

Reduced costs: pay only for the resources you use

Increased scalability: easily scale up or down to meet your evolving needs

Improved reliability: leverage redundant, highly-available systems

Enhanced security: encrypt data in transit and at rest and implement granular access control

Expanded capabilities: integrate directly or indirectly with a vast array of Google Cloud native tools and solutions such as:

BigQuery’s materialized views and BI Engine for increased performance and efficiency

Pub/Sub to transport data with low-latency

Dataflow to process data at scale in batch or streaming mode

Vertex AI to build, deploy, and scale machine learning models

and many, many more

Google Cloud offers the BigQuery Migration Service as a comprehensive solution to accelerate migration from your Hive data warehouse to BigQuery. It includes free-to-use tools that help you with each phase of migration, including assessment and planning, data transfer, and data validation. Two of those tools, the BigQuery batch SQL translator and interactive SQL translator, enable you to translate your Hive queries to BigQuery’s own ANSI-compliant SQL syntax so you can then run those queries natively with BigQuery execution engine.

The new Hive-BigQuery connector offers one additional option: You can keep your original queries in their HiveQL dialect, continue to run those queries with the Hive execution engine on your cluster, but let those queries access data migrated to BigQuery and BigLake tables.

Here’s how the Connector assisted Flipkart’s data lake migration to Google Cloud:

“Flipkart places great importance on interoperability with open source technologies as a part of their reliance on and contribution to the open source community. The Hive-BigQuery Connector has played a crucial role in enabling queries on BigQuery data from Hive, as Hive is the primary query engine on our data lake. This integration has provided Flipkart the flexibility to utilize fast query engines like BigQuery without the need for data duplication or silos across various data stores.” – Venkata Ramana Gollamudi, Principal Architect, Flipkart; Apache Committer

Use cases

The Hive-BigQuery Connector can help you in at least the following core use cases:

Ensure continuity of operations during a wholesale migration. Imagine you decide to move your entire Hive data warehouse to BigQuery and you intend to eventually translate all of your existing Hive queries to BigQuery’s SQL dialect. You expect that the migration will take a significant amount of time due to the sheer size of your data warehouse and the large number of connected applications. You need to ensure smooth continuity of operations during the migration period. To do this, you can first move your data to BigQuery then let your original Hive queries access that data through the Connector while you gradually translate them to BigQuery’s own ANSI-compliant SQL dialect. Once the migration is complete, you can use BigQuery exclusively and retire Hive altogether.

Use BigQuery for some, but not all, data warehouse needs. With the Hive-BigQuery Connector, you can choose to continue using Hive for most workloads and only use BigQuery for certain workloads that you think would benefit from specific BigQuery features like, for example, BI Engine or BigQuery ML. For this use case, you can use the Connector to unify the two environments by letting Hive join its own tables with the tables managed by BigQuery.

Maintain a full open-source software (OSS) stack. Let’s say you want to avoid any potential vendor lock-in and decide to continue using a full OSS stack for your data warehouse. You migrate your data in its original OSS format (e.g. Avro, Parquet or ORC) to Cloud Storage and continue using Hive to execute and process your queries in Hive’s own SQL dialect. For this use case, you can use the Connector to augment your OSS stack foundation by leveraging some BigLake and BigQuery features such as metadata caching for query performance, or Data Loss Prevention, column-level access control, and dynamic data masking for security and governance at scale.

Features

The Connector’s public preview release already ships with many features, including:

Running queries with MapReduce and Tez execution engines

Creating and deleting BigQuery tables from Hive

Joining BigQuery and BigLake tables with Hive tables

Fast reads from BigQuery tables using the Storage Read API streams and the Apache Arrow format

Two methods for writing data to BigQuery:

Direct writes using the BigQuery Storage Write API in pending mode. Use this method for workload that requires low write latency like near real-time dashboards with short refresh time windows.

Indirect writes by staging temporary Avro files to Cloud Storage then loading those files into the destination table using the Load Job API. This method is cheaper than the direct method, since BigQuery load jobs are free. However, it is slower, so it should only be used for workloads that aren’t time critical.

Accessing BigQuery time-partitioned and clustered tables. Here’s an example defining the relation between a Hive table and a table that is natively partitioned and clustered in BigQuery:

Column pruning to avoid retrieving unnecessary columns from the data layer

Predicate pushdowns to pre-filter data rows at the BigQuery storage layer. This can drastically reduce the amount of data traversing the network and therefore improve overall query performance.

Automatic conversion of Hive data types to BigQuery data types

Reading BigQuery views and table snapshots

Getting started

To get started, see the documentation on how to install and configure the connector on your Hive cluster.

We’re continuously adding new features and making improvements, and your feedback is welcome! Send us your thoughts and feature requests either by contacting your Google Cloud sales representative or opening a Github issue. We are excited to see all the innovative ways you will use this new Connector.

Thanks to contributors to the design and development of this connector, in no particular order: David Rabinowitz, Yi Zhang, Dagang Wei, Vinay Londhe, Roderick Yao, Kartik Malik, Guruprasad S, Nikunj Bhartia, Gaurav Khandelwal, Vijay Dulipala, Savitha Sethuram, Palak Mishra, Akanksha Singh, Prathap Reddy.

You can now query Firestore using the OR operator. Your application may often need to compare one or more fields with two or more values and select the documents that match at least one of the values. This is an extension to the IN operator which could be used to compare multiple values with the same field. 

For example, if you’re searching for information about cats and dogs, you could use the OR operator to combine the two search terms into a single query:

‘cats = black OR dogs = brown’

This would return results that include information about both black cats and brown dogs.

Firestore now supports OR queries via server and client SDKs, with streaming and offline compatibility.

Use Cases

Support for OR operator now enables you to utilize Firestore for building applications which need to filter one or more fields with two or more values using any of our existing query constructs. 

Examples 

Let’s say you are running an employee management portal on Firestore and you want the ability to filter on a number of criteria like Title, Organization, Location, Tenure etc., you can now easily utilize OR to be able to do it.

Now, say you want to query for performance reviews for a given employee that occurred within the last two years, or where the given rating was higher than 4. 

You can nest multiple ORs to accomplish such queries as follows :

Nested ORs can be useful when you need to check multiple conditions, but they can also be difficult to read and understand. It is important to use them sparingly and to only use them when necessary.

OR operator can also be used on all our existing query constructs like querying across Collection Groups, or in combination with existing operators like COUNT(). 

Let’s take an example of Collection Groups called Landmarks, you can now use OR operator to filter collections within the group

We know how valuable COUNT() is to you, now you have the ability to combine it with OR to query the data more easily as shown below :

Extended limits for IN and array contains any  

Excitement continues, use multiple IN operators and also combine up to 30 clauses using an IN or array-contains-any operator in Firestore. 

Example :

Note : Not_IN operator cannot be combined with the OR operator or have more than 10 clauses.

Next Steps 

Enjoy building apps more quickly and conveniently with Firestore.

Please refer to official documentation for more detailed information.

Welcome to the second Cloud CISO Perspectives for June 2023. In my previous newsletter, I discussed the growing, important role that accurate threat intelligence can play in helping keep organizations secure, and how we centered that at our Security Summit. 

Today’s cybersecurity threats can pose dangerous risks to organizations, and have a material impact on their business. It’s vital that leaders and boards need to stay abreast of accurate, timely threat intelligence, so I’m turning the mic over to Sandra Joyce, vice president of Mandiant Intelligence at Google Cloud, to talk more about the current threat landscape.

As with all Cloud CISO Perspectives, the contents of this newsletter are posted to the Google Cloud blog. If you’re reading this on the website and you’d like to receive the email version, you can subscribe here.

What you need to know about the ‘zero-day summer’ threat landscape

By Sandra Joyce, vice president, Mandiant Intelligence at Google Cloud

A series of recent zero-day vulnerability-related incidents has led some members of the security community to refer to this season as a “zero-day summer.” The humorous overtone belies the seriousness of the problem, and it’s important for business leaders and boards to understand that these recent major incidents are rooted in larger trends.

Vulnerabilities in file transfer tools and security products have had serious consequences for several of our customers, and concerns that the trend is snowballing are growing. Zero days discovered thus far this year are slightly ahead of last year’s pace, and unfortunately, a longer, multiyear view reveals the number of zero days is growing.

In 2021 and 2022, Chinese cyber espionage actors have been associated with 10 zero days in security devices such as firewalls, VPNs, email security appliances, networking products, and virtualization software. That’s half of all zero days we tracked as exploited by Chinese cyber espionage actors. These vulnerabilities have enabled intrusions on a global scale, allowing actors such as APT41, a state-sponsored Chinese threat actor, to gain access to a variety of sectors.

A concentration in security devices and virtualization software is unlikely to be a coincidence. We believe these actors are deliberately focusing on these devices because they represent multiple tactical advantages. For instance, these devices are challenging to monitor, reducing the likelihood of detection. Many of these devices do not support endpoint detection and response (EDR) solutions, and they lack solutions to detect modifications or collect forensic images. Furthermore, most of these zero days can be exploited without social engineering, which also limits the opportunity for detection.

Chinese cyber espionage has changed dramatically over the last decade from loud, broad intrusion activity to focused targeting in deliberate operations that are far more difficult to detect. In addition to using zero days, these threat actors have largely shifted their attention to the edge of the network, gaining access through internet accessible applications and devices. They are still phishing of course, but they are certainly less reliant on social engineering like actors from North Korea and Iran. 

The infrastructure Chinese actors use to support their operations has changed as well. More frequently, we find them using a complex, ephemeral command and control scheme that involves the use of botnets of compromised systems such as small home and home office Wi-Fi routers. This evolution makes them harder for researchers to track.

Even in the case of vulnerabilities that were not first exploited as zero days by Chinese threat actors, Chinese cyber espionage actors have demonstrated an incredible ability to adopt available exploits quickly to target organizations before they can patch. The speed of their process suggests they have deliberately organized and resourced their operations to move quickly and take advantage of opportunities from recently disclosed vulnerabilities.

FIN11’s recent widespread exploitation of a file transfer software vulnerability is the fourth time we have seen this group exploit this type of software in as many years. From late 2020 to today, FIN11 has exploited multiple zero-day vulnerabilities across software and hardware from Accellion, SolarWinds, Forta, and MOVEit. 

Mass exploitation of a vulnerability in file transfer software can allow threat actors to efficiently gain access to many organizations’ sensitive files without the need for additional lateral movement. For example, in some cases FIN11 was able to start data theft within minutes of exploiting MOVEit systems, almost certainly reducing the time required to monetize access. Limiting the intrusion activity to the file transfer devices also decreases an attacker’s footprint on the network — and therefore the likelihood of detection. Threat actors may also believe that network defenders are less likely to monitor or detect malicious activity targeting these programs compared to other intrusion and post-compromise activity that interacts with more core components of network infrastructure.

The 2020 and 2021 exploitation of Accellion File Transfer Appliance and the 2023 exploitation of MOVEit involved the development of custom malware, which was specifically designed to interact with the targeted programs. Given the apparent success of these campaigns, the threat actors likely view the investment of time and resources into understanding this software, developing or purchasing exploits, and developing malware designed to interact with it as worthwhile. 

Since 2019, financially-motivated actors have exploited 30% of the zero days that we have been able to attribute. The majority of these are linked to extortion operations. This almost certainly reflects the lucrative nature of these operations, as threat actors likely reinvested profits into developing and acquiring exploits. Mandiant regularly observes threat actors advertising and seeking zero-day exploits on underground forums and Telegram channels. 

For example, in June 2023, the English-speaking threat actor “Vars_Sec” advertised a ZTE device zero day with remote code execution for $2500. Similarly, in April 2023, the exploit broker “vulns-rock” advertised a Windows LPE zero day on the Russian-language forum Exploit.in for $150,000. The continued availability of these exploits almost certainly lowers the barrier to entry for acquiring these capabilities.

Even if we aren’t in the midst of a record-breaking “zero-day summer,” the factors behind these recent high-impact cybersecurity events are taking their toll on defenders. Similar incidents are almost inevitable, as long as threat actors continue to experience success with these approaches. 

Predicting the exact nature of the next similar incident is difficult but we can take steps now to mitigate similar risks. One important step organizations can take to mitigate some of these latest zero-day threats to file-transfer systems is to follow our MOVEit Transfer: Containment and Hardening Guide.

The proverbial lazy days of summer will just have to wait.

In case you missed it

Here are the latest updates, products, services, and resources from our security teams so far this month: 

Get ready for Google Cloud Next: Discounted early-bird registration for Google Cloud Next ‘23 has sold out, but you can still register for the conference. This year’s Next comes at an exciting time, with the emergence of generative AI, breakthroughs in cybersecurity, and more. It’s clear that there has never been a better time to work in the cloud industry. Check out our scheduled security sessions, and register now.

Safer spaces: How confidential computing can grow secure data collaboration: Confidential Computing is driving a new era of privacy-enhancing tech to help organizations make the most of the cloud without compromising privacy. Read more.

How to migrate sensitive data using Google Cloud’s CDMC-certified architecture: As enterprises accelerate adoption of cloud-based services and products, they want to secure and govern rapidly-expanding volumes of their most sensitive data. The Enterprise Data Management Council (EDM Council) has accredited a new Google Cloud whitepaper and solution source code as a Cloud Data Management Capabilities (CDMC) Certified Cloud Solution. Read more.

GKE Security Posture dashboard now GA with enhanced features: The Google Kubernetes Engine (GKE) Security Posture dashboard is now generally available. It’s designed to streamline the security management of GKE clusters, and now includes a range of powerful features such as misconfiguration detection and vulnerability scanning to help ensure your applications remain safe and secure. Read more.

Now in Assured Workloads: New regions, capabilities, and supported services: Assured Workloads can help organizations more easily achieve and maintain compliance with relevant regimes around the world without refactoring. Here are several new features and services that are now generally available (GA) in Assured Workloads. Read more.

Introducing client authentication with Mutual TLS on Google Cloud Load Balancing: We are excited to announce the Preview of front-end mutual TLS (mTLS) support, allowing you  to offload client certificate authentication using External HTTPS Load Balancing. Read more.

Protect data from disasters using new Asynchronous Replication: In today’s business landscape, data availability and integrity are paramount. Disasters can disrupt operations and pose a significant risk to critical information. To address this, we have introduced Persistent Disk Asynchronous Replication, which enables disaster recovery for Compute Engine workloads. Read more.

Improving financial control and observability with security monitoring: Over the past few weeks, we’ve introduced new capabilities and solutions to better integrate security monitoring with observability and financial controls, including building better budgets. Read more.

News from Mandiant

How we secure the AI pipeline: Mandiant analysts and responders are already using Bard in their workflows to identify threats faster, eliminate toil, and better scale talent and expertise. Organizations are keen to understand how best to integrate it into their own existing business processes, technology stacks, and delivery pipelines, and ultimately drive business value. Mandiant’s approach to securing the AI pipeline builds on the recently-announced Security AI Framework. Read more. 

Detection, containment, and hardening opportunities on compromised VMware hosts: Mandiant researchers focus on the artifacts, logging options, and hardening steps to detect and prevent tactics and techniques seen being used by threat actor group UNC3886. Read more. 

Google Cloud Security podcasts

We launched a weekly podcast focusing on Cloud Security in February 2021. Hosts Anton Chuvakin and Timothy Peacock chat with cybersecurity experts about the most important and challenging topics facing the industry today. Earlier this month, they discussed:

Can IAM actually be… fun? One of the secrets nobody in cloud security wants you to know is that Identity Access Management (IAM) is exciting, dynamic, and even… enjoyable. At least, that’s according to Ian Glazer, the founder of Weave Identity and co-founder of IDPro. What makes IAM so interesting? And how is it different between cloud providers? Listen here.

Policy-as-Code can help secure your cloud environment: We present a deep dive into the new world of policy-as-code, from who should learn it, to who should manage it, to how it can help secure the cloud. Dominik Richter, founder and head of product at Mondoo, joins us for a Policy-as-Code therapy session, and we even talk about how to tell when it’s succeeding. Listen here.

Will SIEM ever die: What can its past tell us about its future: Security Information and Event Management (SIEM) has been around a long time. We discuss which old SIEM lessons still apply today, which old lessons can harm your organizations, and what are the top modern cloud security use cases for SIEM, with David Swift, security strategist at Netenrich. Listen here.

Mandiant podcasts

Threat Trends: A requirements-driven approach to cyber threat intelligence: Dr. Jamie Collier, senior threat intelligence advisor at Mandiant, joins host Luke McNamara to discuss the recent white paper from Mandiant on developing a requirements-driven approach to intelligence, challenges that organizations face in this area, and the importance of recurring stakeholder feedback to a well-functioning cyber threat intelligence team. Listen here.

To have our Cloud CISO Perspectives post delivered twice a month to your inbox, sign up for our newsletter. We’ll be back in two weeks with more security-related updates from Google Cloud.