Cloud

October 23rd (this past Sunday) was my 5th Googleversery and we just wrapped up an incredible Google Next 2022!  It was great to see so many customers and my colleagues in person this year in New York City. This blog is an attempt to share progress we have made since last year (4th year anniversary blog post 2021 Next). 

Bringing BigQuery to the heart of your Data Cloud

Since last year we have made significant progress across the whole portfolio. I want to start with BigQuery, which is at the heart of our customers’ Data Cloud. We have enhanced BigQuery with key launches like multi-statement transactions, Search and operational log analytics, native JSON support, slot recommender,interactive SQL translation from various dialects like Teradata, Hive, Spark, materialized views enhancements andtable snapshots. Additionally we have launched various enhancements to SQL language, accelerate customer cloud migration with BigQuery migration services and introduced scalable data transformation pipelines in BigQuery using SQL with the Dataform preview. 

One of the most significant enhancements to BigQuery is support for unstructured data in BigQuery through object tables. Object tables enable you to take advantage of common security and governance across your data.  You can now build data products that unify structured and unstructured data in BigQuery.

To support data openness, at Next ’22 we announced the general availability of BigLake, to help you break down data silos by unifying lakes and warehouses. BigLake innovations add support for Apache Iceberg, which is becoming the standard for open source table format for data lakes. And soon, we’ll add support for formats including Delta Lake and Hudi. 

To help customers bring analytics to their data irrespective of where it resides, we launched BigQuery Omni. Now we are adding new capabilities such as cross-cloud transfer and cross-cloud larger query results that will make it easier to combine and analyze data across cloud environments. We also launched on-demand pricing support which enables you to get started at a low cost for BigQuery Omni. 

To help customers break down data boundaries across organizations, we launched Analytics Hub. Analytics Hub is a data exchange platform that enables organizations to create private or public exchanges with their business partners. We have added Google data, which includes highly valuable datasets like Google Trends. With hundreds of partners sharing valuable commercial datasets, Analytics Hub helps customers reach data beyond their organizational walls. We also partnered with the Google Earth Engine team to use BigQuery to get access to and value from the troves of satellite imagery data available within Earth Engine.

We’ve also invested to bring BigQuery together with operational databases to help customers build intelligent, data-driven applications. Innovations include federated queries for Spanner, Cloud SQL and Bigtable, allowing customers to analyze data residing in operational databases in real-time with BigQuery. At Next ’22, we announced Datastream for BigQuery which provides easy replication of data from operational database sources such as AlloyDB, PostgreSQL, MySQL, and Oracle, directly into BigQuery with a few simple clicks.

From Data to AI, with built-in intelligence for BigQuery and Vertex AI

We launched BigQuery Machine Learning in 2018 to make machine learning accessible to data analysts and data scientists across the globe. Now, customers create millions of models and tens of millions of predictions every month using BigQuery ML. Vertex AI enables ML Ops from data model to deployment in production and running predictions in real-time. Over the past year we have tightly integrated BigQuery and Vertex AI to simplify the ML experience. 

Now you can create models in BigQuery using BigQuery ML which are instantly visible inVertex AI model registry. You can then directly deploy these models to Vertex AI endpoints for real-time serving, use VertexAI pipelines to monitor and train models and view detailed explanations for your predictions through BigQuery ML and Vertex AI integration. 

Additionally, we announced an integration between Colab and BigQuery which allows users to explore results quickly with a data science notebook on Colab. “Colab” was developed by Google Research to allow users to execute arbitrary Python code and became a favorite tool for data scientists and machine learning researchers. The BigQuery integration enables seamless workflows for data scientists to run descriptive statistics, generate visualizations, create a predictive analysis, or share your results with others.

Learn more about innovations to bring data and AI closer together, check out my session at Next with June Yang, VP of Cloud AI and Industry Solutions.

Delivering the best of open source

We have always believed in making Google Cloud the best platform to run Open Source Software. Cloud Dataproc enables you to run various OSS engines like Spark, Flink, Hive. We have made a lot of enhancements over the past year in Dataproc. One of the most significant enhancements was to create a Serverless Spark offering that enables you to get away from clusters and focus on just running Spark Jobs. At Cloud Next 2022, we added built-in support for Apache Spark in BigQuery will allow data practitioners to create BigQuery stored procedures unifying their work in Spark with their SQL pipelines. This also provides integrated BigQuery billing with access to a curated library of highly valuable, internal and external assets. 

Powering streaming analytics

Streaming analytics is a key area of differentiation for Google Cloud with products like Cloud Dataflow and Cloud Pub/Sub. This year, our goal was to push the boundaries of innovation in real-time processing through Dataflow Prime and make it seamless to get real-time data coming to Pub/Sub to land into BigQuery for advanced analytics. At the beginning of the year, we introduced over 25 new Dataflow Templates as Generally Available.  At July’s Data Engineer Spotlight, we made Dataflow Prime, Dataflow ML, and Dataflow Go Generally Available. We also introduced a number of new Observability features for Dataflow to give you more visibility and control over your Dataflow pipelines.

Earlier this year we introduced a new type of Pub/Sub subscription called a “BigQuery subscription” that writes directly from Cloud Pub/Sub to BigQuery. With this integration, customers no longer need to pay for data ingestion into BigQuery – you only pay for the Pub/Sub you use.

Unified business intelligence

In Feb 2020 we closed the Looker acquisition and since then we have been busy at work in building Looker capabilities and integrating it into Google Cloud. Additionally, Data Studio has been our self service BI offering for many years. It has the strongest tie-in with BigQuery and many of our BigQuery customers use Data Studio. Announced at Next’22, we are bringing all BI assets under the single umbrella of Looker. Data Studio will become Looker Studio and include a paid version that will provide enterprise support. 

With tight integration between Looker and Google Workspace productivity tools, customers gain easy access via spreadsheets and other documents, to consistent, trusted answers from curated data sources across your organization. Looker integration with Google Sheets is in preview now and increased accessibility of BigQuery to Connected Sheets allows more people to analyze large amounts of data. You can read more details here. 

Intelligent data management and governance

Lastly, a challenge that is top of mind for all data teams is data management and governance across distributed data systems. Our data cloud provides customers with an end-to-end data management and governance layer, with built-in intelligence to help enable trust in data and accelerate time to insights. Earlier this year we launched Dataplex as our Data Management and Governance service. Dataplex helps organizations centrally manage and govern distributed data. Furthermore, we unified Data Catalog with Dataplex to provide a streamlined experience for customers to centrally discover their data with business context and govern and manage that data  with built-in data intelligence. 

At Next we introduced data lineage capabilities with Dataplex to gain end-to-end lineage from ingestion of data to analysis to ML models. Advancements for automatic data quality in Dataplex ensure confidence in your data which is critical to get accurate predictions. Based on customer input we’ve also added enhanced data discovery for automatic cataloging to databases and Looker from a business glossary and added a Spark-powered data exploration workbench. And Dataplex is now fully integrated with BigLake so you can now manage fine grained access control at scale.

An open data ecosystem

Over the past 5 years, the Data Analytics team goal has been to make Google Cloud the best place to run analytics. One of the key tenets of this was to ensure we have the most vibrant partner ecosystem. We have a rich ecosystem of hundreds of tech partner integrations and have 40+ partners who have been certified through the Cloud Ready-BigQuery initiative. 

Additionally, more than 800 technology partners are building their applications on top of our Data Cloud. Data Sharing continues to be one of the top capabilities leveraged by these partners to easily share information at any scale with their enterprise customers.  

We also announced new updates and integrations with Collibra, Elastic, MongoDB, Palantir, ServiceNow, Sisu Data, Reltio, Striim and Qlik to help customers move data between platforms of your choice and bring more Google’s Data Cloud capabilities to partner platforms.

Finally, we established a Data Cloud Alliance  together with 17 of our key partners who provide the most widely-adopted and fastest-growing enterprise data platforms today across analytics, storage, databases and business intelligence.  Our mission is to collaborate to solve modern data challenges providing an acceleration path to value. The first key areas where we are focusing are related to : data interoperability, data governance and solving for skills gap through education. 

Customer momentum across a variety of industries and use cases

We’re super excited for organizations to share their Data Cloud best practices at Next, including Walmart, Boeing, Twitter, Televisa Univision, L’Oreal, CNA Insurance, Wayfair, MLB, British Telecom, Telus, Mercado Libre, LiveRamp, and Home Depot. Check out all the Data Analytics sessions and resources from Next and get started on your Data Cloud journey today. We look forward to hearing your story at a future Google Cloud event.

AI is at a tipping point. We are seeing the impact of AI across more and more industries and use cases. Organizations with varying levels of ML expertise are solving business-critical problems with AI — from creating compelling customer experiences, to optimizing operations, to automating routine tasks, these organizations learn to innovate faster and ultimately, get ahead in the marketplace. However, in many organizations, AI and machine learning systems are often separate and siloed from data warehouses and data lakes. This widens the data to AI gap, limiting data-powered innovation. 

At Google Cloud, we have harnessed our years of experience in AI development to make the data-to-AI journey as seamless as possible for our customers. Google’s data cloud simplifies the way teams work with data. Our built-in AI/ML capabilities are designed to meet users where they are, with their current skills. And our infrastructure, governance, and MLOps capabilities help organizations to leverage AI at scale. 

In this blog, we’ll share how you can simplify your ML workflows using BigQuery ML and Vertex AI and showcase the latest innovations in BigQuery ML.

Simplify machine learning workflows with BigQuery ML and Vertex AI

Organizations that follow a siloed approach to managing databases, analytics and machine learning often need to move data from one system to another. This leads to data duplication with no single source of truth and makes it difficult to adhere to security and governance requirements. 

Additionally, when building ML pipelines, you need to train and deploy your models. Therefore you need to plan your infrastructure for scale. You also need to make sure that your ML models are tuned and optimized to run efficiently on your infrastructure. For example, you may need a large set of kubernetes clusters or access to GPU-based clusters so that you can train your models quickly. This forces organizations to hire highly skilled professionals with deep knowledge of Python, Java and other programming languages. 

Google’s data cloud provides a unified data and AI solution to help you overcome these challenges and simplify your machine learning workflows. BigQuery’s serverless, scalable architecture helps you create a powerful single source of truth for your data. BigQuery ML brings machine learning capabilities directly into your data warehouse through a familiar SQL interface. BigQuery ML’s native integration with Vertex AI allows you to leverage MLOps tooling to deploy, scale, and manage your models.

BigQuery ML and Vertex AI help accelerate the adoption of AI across your organization.

Easy data management: Manage ML workflows without moving data from BigQuery, eliminating security and governance problems. The ability to manage workflows within your datastore removes a big barrier to ML development and adoption.

Reduce infrastructure management overhead: BigQuery takes advantage of the massive scale of Google’s compute and storage infrastructure. You don’t need to manage huge clusters or HPC infrastructure to do ML effectively.

Remove skillset barrier: BigQuery ML is SQL based. This allows many model types to be directly available in SQL, such as regression, classification, recommender systems, deep learning, time series, anomaly detection, and more. 

Deploy models and operationalize ML workflows: Vertex AI Model Registry makes it easy to deploy BigQuery ML models to a Vertex AI REST endpoint for online or batch predictions. Further, Vertex AI Pipelines automate your ML workflows, helping you reliably go from data ingestion to deploying your model in a way that lets you monitor and understand your ML system.

Get started with BigQuery ML in three steps

Step 1: Bring your data into BigQuery automatically via Pub/Sub in real time or in batch using BigQuery utilities or through one of our partner solutions. In addition, BigQuery can access data that may be residing in open source format such as Parquet/Hudi residing in object storage using BigLake. Learn more about loading data into BigQuery.

Step 2: Train a model by running a simple SQL query (create model) in BigQuery and point to the dataset. BigQuery is highly scalable in terms of compute and storage, whether it is a dataset with 1000 rows or billions of rows. Learn more about model training in BigQuery ML.

Step 3: Start running predictions. Use a simple SQL query to run predictions on the new data. There are a vast number of use cases supported through BigQuery ML such as demand forecasting, anomaly detection or even can be used for predicting new segments for your customer. Check out the list of supported models. Learn more about running predictions, detecting anomalies or predicting demand with forecasting.

Increase impact with new capabilities in BigQuery ML

At Next ‘22, we announced several innovations in BigQuery ML that help you to quickly and easily operationalize ML at scale. To get early access and check out these new capabilities, submit this interest form. 

1. Scale with MLOps and pipelines

When you are training a lot of models across your organization, managing models, comparing results, and creating repeatable training processes can be incredibly difficult. New capabilities make it easier to operationalize and scale BigQuery ML models with Vertex AI’s MLOps capabilities. 

Vertex AI Model Registry is now GA, providing a central place to manage and govern the deployment of all your models, including BigQuery ML models. You can use Vertex AI Model Registry for version control and ML metadata tracking, model evaluation and validation, deployment and model reporting. Learn more here. 

Another capability that further helps operationalize ML at scale is Vertex AI Pipelines, a serverless tool for orchestrating ML tasks so that they can be executed as a single pipeline, rather than manually triggered each task (e.g. train a model, evaluate the model, deploy to an endpoint) separately. We are introducing more than 20 BigQuery ML components to simplify orchestrating BigQuery ML operations. This eliminates the need for developers and ML engineers to write their own custom components to invoke BigQuery ML jobs. Additionally, if you are Data Scientist who prefers running code over SQL, you can now use these operators to train and predict in BigQuery ML.

2. Derive insights from unstructured data

We recently announced the preview of object tables, a new table type in BigQuery that enables you to directly run analytics on unstructured data including images, audio, documents and other file types. Using the same underlying framework, BigQuery ML will now help you to unlock that value from unstructured data. You can now execute SQL on image data and predict results from machine learning models using BigQuery ML. For example, you can import either state of the art TensorFlow vision models (e.g. ImageNet and ResNet 50) or your own models to detect objects, annotate photos, extract text from images.

Learn more here and check out this demo of our customer Adswerve, a leading Google Marketing, Analytics and Cloud partner and their client Twiddy & Co, a vacation rental company in North Carolina, who combined structured and unstructured data using BigQuery ML to analyze images of rental listings and predict the click-through rate, enabling data-driven photo editorial decisions. In this work images attributed to 57% of the final prediction results.

3. Inference Engine

BigQuery ML acts an inference engine that works in a number of ways, including using existing models and can be extended to bring your own model:

BigQuery ML trained models

Imported models of various formats

Remote models

BigQuery ML supports several models out-of-the-box. However, some customers want to inference with models that are already trained in other platforms. Therefore, we are introducing new capabilities that allow users to import models beyond TensorFlow into BigQuery ML, starting with TFLite and XGBoost.

Alternatively, if your model is too big to import (see current limitations here) or already deployed at an endpoint and you don’t have the ability to bring that model into BigQuery, BigQuery ML now allows you to do inference on remote models ( resources that you’ve trained outside of Vertex AI, or that you’ve trained using Vertex AI and exported). You can deploy a model on Vertex AI or Cloud Functions and then use BigQuery ML to do prediction.

4. Faster, more powerful feature engineering

Feature preprocessing is one of the most important steps in developing a machine learning model. It consists of the creation of features and the cleaning of the data. Sometimes, the creation of features is also referred to as “feature engineering”. In other words, Feature engineering is all about taking data and representing it in ways that model training results in great models. BQML performs automatic feature preprocessing during training, based on the feature data types. This consists of missing value imputation and feature transformations. Besides these, all numerical and categorical features will be CASTed to double and string, respectively, for BQML training and inference. We are taking feature engineering to the next level by introducing several new numerical functions (such as MAX_ABS_SCALER, IMPUTER, ROBUST_SCALER, NORMALIZER) and categorical functions (such as ONE_HOT_ENCODER, LABEL_ENCODER, TARGET_ENCODER). 

BigQuery ML supports two types of feature preprocessing:

Automatic preprocessing. BigQuery ML performs automatic preprocessing during training. For more information, see Automatic feature preprocessing.

Manual preprocessing. BigQuery ML provides the TRANSFORM clause for you to define custom preprocessing using the manual preprocessing functions. You can also use these functions outside the TRANSFORM clause.

Further, when you export BigQuery ML models by registering with Vertex AI Model Registry or manually, transform clauses will also be exported with it. This really simplifies online model deployment to Vertex.

5. Multivariate time series forecasting

Many BigQuery customers use the natively supported ARIMA PLUS model to forecast future demand and plan their business operations. Until now customers could forecast using only a single input variable. For example, to forecast ice cream sales, along with target metrics the past sales, customers could not forecast using external covariates such as weather. With this launch, users can now make more accurate forecasts by taking more than one variable into account through multivariate time series forecasting with ARIMA_PLUS_XREG (ARIMA_PLUS with external regressors (such as weather, location, etc).

Getting Started

Submit this form to try these new capabilities that help you accelerate your data to AI journey with BigQuery ML. Check out this video to learn more about these features and see a demo of how ML on structured and unstructured data can really transform marketing analytics.

Acknowledgements: It was an honor and privilege to work on this with Amir Hormati, Polong Lin, Candice Chen, Mingge Deng, Yan Sun. We further acknowledge Manoj Gunti, Shana Matthews and Neama Dadkhahnikoo for support, work they have done and their inputs.

At Google Cloud, we continue to invest in our vision of invisible security where advanced capabilities are engineered into our platforms, operations are simplified, and stronger outcomes can be achieved. Assured Workloads is a Google Cloud service that helps customers create and maintain controlled environments that accelerate running more secure and compliant workloads, including enforcement of data residency, administrative and personnel controls, and managing encryption keys.  

Today, we are extending these capabilities to more regions. Assured Workloads for Canada is now generally available, and Assured Workloads for Australia is now available in preview. We are also making it easier for customers to get started with Assured Workloads by automating the onboarding process, and offering new tools to define specific compliance requirements and then validate the compliance posture of your Assured Workloads. These new capabilities can further speed deployment of regulated workloads on Google Cloud. Customers can get started today here.

Assured Workloads for Canada 

Today, Assured Workloads for Canada is generally available to all customers with a premium Assured Workloads subscription, and existing customers can use it right away. Assured Workloads for Canada provides:

Data residency in either a specific Canadian cloud region or across any of our Canadian cloud regions (currently Toronto and Montreal)

Personnel access and technical support services restricted to personnel located in Canada

Cryptographic control over data, including customer-managed encryption keys. Data encrypted with keys stored within a region can only be decrypted in that region.  

Service usage restrictions to centrally administer which Google Cloud products are available within the Assured Workloads environment  

Assured Workloads for Australia

Assured Workloads for Australia is available in Preview today. This launch supports public sector customers in Australia following the certification of Google Cloud against the Hosting Certification Framework (HCF) by Australia’s Digital Transformation Agency. Assured Workloads for Australia provides:

Data residency in either a specific Australian cloud region or across any of our Australian cloud regions (currently Sydney and Melbourne.)

Personnel access and technical support services restricted to five countries (U.S., U.K., Australia, Canada, and New Zealand.) Initially launching with just U.S. persons, we plan on expanding to more support persons next year.

Cryptographic control over data, including customer managed encryption keys. Data encrypted with keys stored within a region can only be decrypted in that region 

Service usage restrictions to centrally administer which Google Cloud products are available within the Assured Workloads environment  

Assured Workloads Monitoring

We are also excited to announce the general availability of Assured Workloads Monitoring. With Assured Workloads Monitoring, customers can get more visibility into organization policy changes that result in compliance violations. Assured Workloads Monitoring scans customer environments in real time and provides alerts whenever a change violates the defined compliance posture.

Assured Workloads Monitoring can also help customers take corrective actions. The monitoring dashboard shows which policy is being violated and provides instructions on how to resolve the finding. Customers also have the ability to mark violations as exceptions and keep an audit trail of approved changes. Click here to learn more about Assured Workloads Monitoring and how you can use it to help keep your organization compliant.

FedRAMP Moderate self-serve onboarding

Assured Workloads provides the ability for customers to create U.S. FedRAMP Moderate environments at no charge. We’ve now made it even easier for customers to create these environments with just a few clicks. Click here to get started.

Conclusion

Assured Workloads helps accelerate running more secure and compliant workloads on Google Cloud. These announcements grow our global capabilities so customers can operate across markets with a consistent set of services and controls. To learn more about Assured Workloads, check out our customer session from this year’s Google Cloud Next and our Assured Workloads video walkthrough series on YouTube.

Google’s Data Cloud’s aim is to help customers close their data-to-value gap. BigQuery, Google Cloud Fully managed, serverless data platform lets customers  combine all data  — structured, semi-structured, and unstructured. Today, we are excited to announce the general availability of search indexes and search functions in BigQuery. This combination enables you to efficiently perform rich text analysis on data that may have been previously hard to explore due to the siloing of text information. With search indexes, you can reduce the need to export text data into standalone search engines and instead build data-driven applications or derive insights based on text data that is combined with the rest of your structured, semi-structured (JSON), unstructured (documents, images, audio), streaming, and geospatial data in BigQuery. 

Our previous post announcing the public preview of search indexesdescribed how search and indexing allow you to use standard BigQuery SQL to easily find unique data elements buried in unstructured text and semi-structured JSON, without having to know the table schemas in advance. 

The Google engineering team ran queries on Google Cloud Logging data of a Google internal test project (10TB, 100TB, and 1PB scales) using the SEARCH function with a search index. We then compared that to the equivalent logic with the REGEXP_CONTAINS function (no search index) and found that for the evaluated use cases, the new capabilities  provided the following overall improvements (more specific details below): 

Execution time: 10x. On average, queries that use BigQuery SEARCH function backed by a search index are 10 times faster than the alternative queries for the common search use cases.

Processed bytes: 2682x. On average, queries with BigQuery SEARCH function backed by a search index process 2682 times fewer bytes than the alternative queries for the common search use cases.

Slot usage (BigQuery compute units): 1271x. On average, queries with BigQuery SEARCH function backed by a search index use 1271 times less slot time than the alternative queries for the common search use cases.

Let’s put these numbers into perspective by discussing the common ways search indexes are used in BigQuery. Please note that all improvement numbers provided were derived from a Google engineering team analysis of common use cases and queries on a Google internal test project’s log data. The results may not map directly to customer queries and we would encourage you to test this on your own data set. 

Rare term search for analytics on logs

Log analytics is a key industry use case enabled by Google’s Data Cloud. In a recent Google Cloud Next ‘22 talk on Operational Data Lakes, The Home Depot discussed how they were able to sunset their existing enterprise log analytics solution and instead use BigQuery and Looker as an alternative for 1,400+ active users in order to reduce costs and improve log retention. Goldman Sachs used BigQuery to solve their multi-cloud and scaling problems for logging data. Goldman Sachs moved from existing logging solutions to BigQuery to improve long term retention, detect PII in their logs with Google DLP, and implement new cost controls and allocations.  

A very common query pattern in analytics on logs is rare-term search or colloquially, “finding a needle in the haystack.” That means quickly searching through millions or billions of rows to identify an exact match to a specific network ID, error code, or user name to troubleshoot an issue or perform a security audit. This is also a quintessential use case for search indexes in a data warehouse. Using a search index on a table of text data allows the BigQuery optimizer to avoid large scanning operations and pinpoint exactly the relevant data required to answer the query. 

Let’s review what the Google engineering team found when they reviewed queries that looked for rare terms with and without a search index.

IP address search in Cloud Logging

Home Depot and Goldman Sachs used BigQuery’s basic building blocks to develop their own customized log analytics applications. However, other customers may choose to use log analytics on Google’s Data Cloud as a pre-built integration within Cloud Logging.  

Log Analytics, powered by BigQuery (Preview) gives customers a managed Log Analytics as a service solution with a specialized interface for logs analysis. It leverages features of BigQuery’s search function which provides specialized ways to look up common logging data elements such as IP addresses, URLs, and e-mails. Let’s take a look at what the Google engineering team found when looking up IP addresses using a search function.

Common term search on recent data for security operations 

Exabeam, an industry leader in security analytics and SIEM MQ, leverages BigQuery search functions and search indexes in their latest Security Operation Platform built on Google’s Data Cloud to search multi-year data in seconds [learn more data journey interview].

Many security use cases are able to leverage a search optimization for queries on recent data that allows you to look up data with common terms using ORDER BY and LIMIT clauses. Let’s take a look at what the Google engineers found for queries on recent data that use an ORDER BY and LIMIT clauses.

Search in JSON objects for ElasticSearch compatibility 

Google technical partner, Mach5 Software, offers its customers an ElasticSearch and OpenSearch-compatible platform powered by BigQuery’s search optimizations and JSON functionality. Using Mach5, customers can migrate familiar tools like Kibana, OpenSearch Dashboards, and pre-built applications, seamlessly to BigQuery, while enjoying a significant reduction in cost and management overhead. Mach5 takes advantage of BigQuery’s search index’s ability to comb through deeply nested data stored in a BigQuery’s native JSON data type. Mach5 Community Edition is freely available for you to deploy and use within your Google Cloud Platform environment.

BigQuery’s SEARCH function operates directly on BigQuery’s native JSON type. Let’s look at some improvements the Google engineering team found when using search with indexing on JSON data.

Learning more

As you can see in the comparisons, there are already significant cost and performance improvements with BigQuery search functions and indexes, even at the petabyte level. Generally speaking, the larger the dataset, the more BigQuery can optimize. This means you can bring petabytes of data to BigQuery and still have them operate effectively. Many customers also combine BigQuery search features with large scale streaming pipelines built with BigQuery’s Storage Write API. This Write API has a default ingestion rate of 3GB per second with additional quota available upon request. It is also 50% lower per GB cost compared to previous streaming APIs offered by BigQuery. These streaming pipelines are fully managed by BigQuery and take care of all the operations from stream to index. Once data is available on the stream, any queries you run with a SEARCH function will have accurate and available data. 

To learn more about how BigQuery search features can help you build an operational data lake, check out this talk on Modern Security Analytics platforms. To see search in action, you can also watch this demo where a search index is built to improve simple searches of label and object data that is generated from running machine learning on vision data. 

You can get started with the BigQuery sandbox and explore these search capabilities at no cost to confirm whether BigQuery fits your needs. The sandbox lets you experience BigQuery and the Google Cloud console without providing a credit card, creating a billing account, or enabling billing for your project.

If organizations can easily analyze unstructured data streams, like live video and images, they can more effectively leverage information from the physical world to create intelligent  business applications. Retailers can improve shelf management by instantly spotting what products are out of stock, manufacturers can  reduce product defects by detecting production errors in real time, and in our communities, administrators could improve traffic management by analyzing vehicle patterns. The possibilities to create new experiences, efficiencies, and insights are endless. 

However, enterprises struggle to ingest, process, and analyze real-time video feeds at scale due to high infrastructure costs, development effort, longer lead times, and technology complexities.

That’s why last week, at Google Cloud Next’ 22, we launched the preview of Vertex AI Vision,  a fully managed end-to-end application development environment that lets enterprises easily build, deploy, and manage computer vision applications for their unique needs. Our internal research shows that Vertex AI Vision can help developers reduce time to build computer vision applications from weeks to hours, at a fraction of the cost of current offerings. As always,  our new AI products also adhere to our AI Principles.

One-stop environment for computer vision applications development 

Vertex AI Vision radically simplifies the process of cost-effectively creating and managing computer vision apps, from ingestion and analysis to deployment and storage. It does so by providing an integrated environment that includes all the tools needed to develop computer vision applications; developers can easily ingest live video streams (all they need is the IP address), add pre-trained models for common tasks such as “Occupancy Analytics,” “PPE Detection,” “Visual Inspection,” add custom models from Vertex AI for specialized tasks, and define a target location for output/ analytics. The application is ready to go.

Vertex AI Vision comprises the following services:

Vertex AI Vision Streams: a geo-distributed managed endpoint service for ingesting video streams & images.  Easily connect cameras or devices from anywhere in the world and let Google handle ingestion and scaling

Vertex AI Vision Applications: a serverless orchestration platform for video models & services enabling developers to stitch together large, auto-scaled media processing and analytics pipelines

Vertex AI Vision Models: a new portfolio of specialized pre-built vision models for common analytics tasks including occupancy counting, PPE detection, face-blurring, retail product recognition and more. Additionally, users can build and deploy their own custom models 

Vertex AI Vision Warehouse: a serverless rich-media storage that provides the best of Google search combined with managed video storage.   Perfect for ingesting, storing, and searching PBs of video data. 

Customers are already seeing the future with Vertex AI Vision 

Customers are thrilled with the possibilities Vertex AI Vision opens. According to Elizabeth Spears, Co-Founder & CPO, Plainsight, a leading developer of computer vision applications,  “Vertex AI Vision is changing the game for use cases that for us have previously been economically non-viable at scale. The ability to run computer vision models on streaming video with up to a 100X cost reduction for Plainsight is creating entirely new business opportunities for our customers.”

Similarly, Brain Corp Vice President Botond Szatmáry said, “Vertex AI Vision is the backend solution that enables Brain Corp’s Shelf Analytics on all BrainOS powered robots, including a new commercial ready reference platform that’s purpose built for end to end inventory analytics. The Vertex AI Product Recognizer and Shelf Recognizer, combined with BigQuery, enable us to efficiently detect products, out of stock events, and low stock events while capturing products, prices, and location within stores and warehouses. Our retail customers can be more competitive in e-commerce, better manage their inventory, improve operational efficiencies, and improve the customer shopping experience with our highly accurate, actionable, and localized inventory shelf insights.” 

You can hear more from Plainsight and Brain Corp in our Next ’22 session. If you are a developer and want to get started on Vertex AI Vision I invite you to experience the magic for yourself here.

Google Government Summit is right around the corner. Join us and government leaders for a full day of networking, best practices, and industry insights on November 15th in Washington, D.C. We can’t wait to see you live in 3D!

Our event will offer over 30 various breakout sessions to engage and explore critical topics such as security and resilience, practical applications of artificial intelligence (AI), collaborative work, and building cultures of innovation and belonging. All sessions will be hosted at the Intercontinental Hotel at the District Wharf. Attendees are eligible to receive up to five continuing professional education (CPE) credits. No advance preparation or prerequisites are required.

View the event agenda breakout sessions to see which sessions you may want to attend, such as:

Advancing Climate Resilience, Sustainability, and Research with Data Analytics

Join Franco Amalfi, Head of Sustainability for the Global Public Sector at Google Cloud, and Duane Piper, CIO and Division Director IT for the South Florida Water Management District, as they discuss how government can prepare for the extreme weather events being made more common by climate change. Learn how the South Florida Water Management District is harnessing earth observation and remote sensing data to inform their climate adaptation strategy with Google Cloud Climate Insights for Natural Resources. 

Speed & Impact: Getting Government onto the Commercial Curve

Learn how government can take advantage of the scale, velocity, and savings driven by innovations in enterprise software and cloud while delivering on the mission requirements of government. Join Joshua Marcuse, Head of Strategy & Innovation at Google Public Sector, Sonny Hashmi, Commissioner for the Federal Acquisition Service at the General Services Administration (GSA), and others as they explain how government leaders can overcome the barriers preventing them from accessing the benefits of technology that advances along the “commercial curve.”

Taking the Wait out of Authority to Operate

Discover how Google Cloud can make the ATO process easier with our unique approach to compliance. David Cruley, Customer Engineer for Google Public Sector, Federal Civilian Agencies, will cover our Google Cloud ATO accelerator, additional tools and techniques you can use, and real customer use cases that will show how easy this process can be with Google Cloud.

Achieve Compliance with Confidence in the Cloud

Join Jeanette Manfra, Senior Director, Global Risk and Compliance, Chris Johnson, Group Product Manager, andGanesh Chilakapati, Group Product Manager, Google Cloud, as they explore ways agencies can achieve modernization and transformation goals while meeting security compliance requirements. Learn about Assured Workloads, Workspace Assured Controls, client-side encryption, and other cloud tools that will allow for innovation without compromising security and compliance.

Using AI to Modernize Government Processes and Drive Mission Outcomes

Speakers Sireesha Kalapala, Industry Solutions Manager, Google, Kevin Cornish, CIO, Covered California, and Jon Horner, Supervisory IT Specialist, United States Patent and Trademark Office (USPTO) will break down what using AI really means, where to start, and how to use it to streamline government processes. Hear from other government agencies about their own AI journeys and how to navigate AI for your agency.

How Can Hybrid and Multicloud Support Your Mission?

Join panelists Leigh Palmer, Vice President, Delivery and Operations, Google Public Sector, Sharon Woods, Director of Hosting and Compute Center, Defense Information Systems Agency,andDeborah Stephens, Deputy CIO, USPTO, as they discuss diversifying your agency’s cloud strategy to fit your needs. Gain insight into hybrid and multicloud models to help you become more agile and flexible and achieve mission success.

There are only a few weeks left to register for Google Government Summit 2022. Save your spot today and check back frequently for agenda updates and news as we get closer to the event date. We look forward to seeing you soon!

Introduction

Machine learning (ML) is essential to video game development. Predicting specific in-game actions and identifying and reaching your most valuable players helps to drive better outcomes. To this end, we need to keep track of all experiments that are happening behind the scenes. Google Cloud recently announced the general availability of a new feature called Vertex AI Experiments that can now help gaming companies do just that – keep track of their ML experiments to uncover insights and best practices across their ML Engineering and Data Science teams. 

In this blog post, we will focus on a popular video game dataset coming from the EA Sports’ FIFA video game series. We will use this dataset to build a ML pipeline to predict the overall rating for a player. We then experiment with different hyperparameters that can then be tracked in Vertex AI Experiments in order to identify the best performing run that’s ultimately put into production by the gaming company.  

The Challenge with Experimentation 

In an ML experiment, you vary different inputs. Your hypothesis might involve testing a specific algorithm or value of a hyperparameter to determine if it is more performant than various alternatives. While ML experiments are common, we often hear the same complaint coming from Data Science teams at gaming companies: 

Data Scientists and other ML practitioners at gaming companies are well versed in experimentation. Where they run into a challenge is keeping a record of all of the rich ML experiments that have been run. 

But why is tracking so challenging? There are a number of reasons.  First, tracking runs of multiple training jobs is cumbersome. It’s easy to lose sight of what’s working versus what’s not. Second, issues compound as your team grows in size. Not all members will be tracking experiments or even sharing their results across the broader team. Third, data capture is time consuming. Manual methods (for example, tracking in spreadsheets) result in inconsistent and often incomplete information, which is difficult to learn from.

Teams can now move away from their old and ineffective way of doing things since platforms now exist that can help Data Science and ML Engineering teams better manage their experiments, compare model or pipeline runs, and uncover new insights. 

What is Vertex AI Experiments and how does it work? 

Let’s briefly discuss what Vertex AI Experiments does. Vertex AI Experiments is a service that more effectively tracks your ML experiments all in one place. Experiments tracked within Vertex AI Experiments consist of a set of runs. While our video game example will be focusing on tracking runs of a ML pipeline, a run within an experiment could be a run of a locally trained model as well. 

In fact, Vertex AI Experiments works hand in hand with other products within the larger Vertex AI suite to better track elements within your various runs. The tools used are Workbench, Pipelines, Experiments, and ML Metadata. The figure below provides a play-by-play on how these tools work together to capture relevant data and uncover the best performing run that is ultimately put into production.

What’s required to get started?

Experiment runs tracked in Vertex AI Experiments are a value add and do not incur additional charges.  Users are only charged for resources that are used during your experiment as described in Vertex AI pricing. Beyond that, as long as you have enabled the relevant APIs such as the Compute Engine API, Vertex AI API, and Notebooks API you are off to the races and can begin tracking and comparing the runs within your experiment. 

Let’s Track ML Pipeline Runs to Better Predict Player Ratings   

To see Vertex AI Experiments in action in a video game context, we now build an ML pipeline and track its various runs within Vertex AI Experiments. While we show you a few highlights here, you can find the full pipeline code for this example in this codelab which again leverages the popular EA Sports FIFA Video Game Series Dataset to predict the overall rating for a player.  Our experiment involves the following: 

Train a custom Keras regression model to predict player ratings using different values of hyperparameters

Use the Kubeflow Pipelines SDK to build a scalable ML pipeline

Create and run a 5-step pipeline that ingests data from Cloud Storage, scales the data, trains the model, evaluates it, and saves the resulting model back into Cloud Storage

Leverage Vertex ML Metadata to save model artifacts such as Models and Model Metrics

Utilize Vertex AI Experiments to track and compare results of the various pipeline runs with different values of hyperparameters

To build, run, and track our ML pipeline runs, we’ll start by importing a couple of libraries:

The Kubeflow Pipelines SDK is used to build our components and connect them together into a pipeline

The Vertex AI SDK allows us to run our pipeline on Vertex Pipelines and track our runs within an experiment

To begin running our experiments, we need to specify our training job by defining it as a pipeline training component. Our component takes in our dataset and hyperparameters (e.g., DROPOUT_RATE, LEARNING_RATE, EPOCHS) as inputs and output model metrics (e.g., MAE and RMSE) and a model artifact.

A few additional callouts. We have included a few parameters in our model architecture so that we can more easily experiment and pass in different values of the hyperparameters we are using to tune the Keras model. In the final step of the code, you can also see how we are able to interact with ML metadata to store our model metrics and the model artifact itself.

After defining and compiling our pipeline, we are ready to submit our pipeline runs. We can speed up our experimentation by creating a variable called “runs” that defines the different hyperparameter values we want to pass in for each run.

We can then leverage a for loop to successfully feed in the different hyperparameter values associated with each run into our pipeline job.

Now we can start to see Vertex AI experiments in action. The Vertex AI SDK for Python currently supports a get_experiment_df method which returns information about your runs. You can use it to get details such as the model metrics and the current state of a given run.

After leveraging the Vertex AI SDK to view experiments, we can now more effectively compare the different runs of our pipeline and identify the run that is best performing and should be put into production. This can be done both programmatically in our managed notebook but also in the console itself.

You might be asking at this point, what could a data scientist at a gaming company learn from the above results? The output from your experiment should contain five rows, one for each run of the pipeline. Both MAE and RMSE are measures of the average model prediction error so a lower value for both metrics is desirable in most cases. 

We can see based on the output from Vertex AI Experiments that our most successful run across both metrics was the final run with a dropout_rate of 0.001, a learning_rate of 0.001, and the count of epochs being 20. Based on this experiment, the final run has the model parameters the gaming company would ultimately use in production since it results in the best model performance.

Conclusion 

ML practitioners can now bid farewell to the old docs and excel sheets they were once using to haphazardly track ML experiments.  With Vertex AI Experiments, we are now able to better harness the power of ML experiments. With more effective tracking comes richer insights, best practices, and the ability to gain insight into the experiments that are happening across your broader team. 

So what’s next? Try out Vertex AI Experiments yourself. Check out our codelab and the other resources below to start getting your feet wet. Thank you for reading! Have a question or want to chat? Find authors here – R.E. Wolfe [Twitter | LinkedIn]

The global workforce is facing a skills gap, affecting more than one billion jobs by 2030. As adoption of technology increases, the World Economic Forum estimates that 50% of all employees will need reskilling by 2025. Globally, 69% of employers report talent shortages and difficulty hiring – mainly due to skills gaps.

BlueCross BlueShield of Tennessee (BCBST) is facing these challenges, with more open cybersecurity and coding jobs than it can fill with local talent. To help address this gap, BCBST and East Tennessee State University (ETSU) forged an innovative partnership to develop the BlueSky Tennessee Institute. Based at BCBST’s Chattanooga headquarters, this groundbreaking collaboration will provide students with the opportunity to earn a bachelor’s degree in computing and a job offer at BlueCross – all in just two years. 

Google Cloud joins the collaboration as an industry and learning partner, providing content and mentorship to prepare BlueSky Institute students for in-demand jobs in technology. Google will provide scholarships to complete the Google IT Support Certificate course and will provide free access to Google Cloud Skills Boost for BlueSky students. BlueSky Institute’s accelerated bachelor’s degree will provide a new pathway into some of the fastest-growing tech careers in the nation. The program will help develop much-needed talent for BlueCross – and prepare the next generation of technology leaders. The program will place a special focus on identifying and recruiting high-potential students from underserved communities in the Chattanooga area.

As BlueSky launches its first cohort of 32 students, I spoke to Mathew Desjardins, site director and senior lecturer at ETSU, Melissa Graham, Student Success manager at BlueSky Tennessee, and Katie Berlent, Public Sector Future Workforce lead at Google Cloud, about how this partnership was developed.

What problem were you trying to solve in establishing the BlueSky Tennessee Institute?

Graham: We found that in our region, there were only 50 computer science graduates over a three-year period. Our goal is to eventually recruit and enroll 50 BlueSky students per year. So we said, how can we not only fill our tech-talent gap, but do something for the community that hasn’t been done before?

Berlent: Yes, the key problem we’re trying to solve here is the talent gap. We hear every day from customers, such as BlueCross and others all over the world, that finding Google Cloud-trained talent is a top priority for their businesses.

How did you come up with this solution?

Desjardins: It was serendipity. BCBST wanted to develop an intensive job training program. ETSU has an amazing computing department. We were already using Google’s Cloud Career Readiness Program, and BCBST was already working with Google Cloud on another project. I’m glad that we were able to collaborate in a way that was best for ETSU, best for BlueCross BlueShield, and ultimately best for the students.

Berlent: We worked closely with the BlueSky Institute team to develop a two-year plan, in which Google Cloud will provide scholarships for BlueSky students to earn the Google IT Support Certificate, access to Google Cloud Skills Boost, and the Google Cloud Career Readiness Program. Google will also provide Google Cloud Teaching Credits to support hands-on learning in the classroom, mentors for the students, and an opportunity to work on Google projects at BCBST as interns.

As you launch the program, what has surprised you the most?

Graham: To see all 32 students show up on day one was a beautiful surprise. We had zero losses in enrollment over the summer, and it really feels good to know that the students trust us with such a major decision in their lives. We are definitely celebrating that as a win for the program.

Where do you expect to see the biggest impact?

Graham: For some of our students, this was their only route to a bachelor’s degree. Some of them were going to stay home and go to a local community college, due to either cost or wanting to still be near family. BlueSky Institute has really been able to eliminate some of those barriers to a bachelor’s degree. That has been very rewarding for us. The multigenerational impact is also very important. Because we’re recruiting from underserved communities, we know that earning these degrees and walking into careers with these certifications is going to trickle down for generations and benefit these communities moving forward.

Desjardins: The students all asked me, “So what’s the cost to us?” And I said, “Nothing.” Google gave us these 500 certificates. I tell them, “I don’t care if you do the Google certificate or not, but I bet ‘future you’ is going to care if you do this Google certificate or not. So I hope you earn as many as you can, because you want to be a better you.” To underline what Melissa said, providing a debt-free education and training certificates–that’s amazing. I, too, feel like this has been the most rewarding part of my career up to this point.

Berlent: Research shows that long-term and wraparound supports make a huge difference in student outcomes. For Google, this is an incredibly exciting partnership because it has all the right components to get these students to the end goal of employment. The students in BlueSky Institute will graduate with industry-standard credentials in addition to that degree. That’s a powerful combination that prepares them for the jobs that are in demand today and in the future.

What do you hope this program will achieve in five years? Or ten years?

Desjardins: Five years from now, I would love for this to be a model that’s being used nationwide. I would love for other businesses to also adopt this model, not just in the education and technology sectors. 

Graham: Our number one goal is student success. We set our goal high for a 100% graduation rate. That would be really powerful and exciting – and then we want our students to choose BlueCross BlueShield of Tennessee as their employer and continue with us long term. Then, like Matt, I would love for BlueSky Institute to reimagine how post-secondary education works, how businesses can partner with universities and others to impact the workforce.

Berlent: In the short term, we want Google content to help and motivate students to succeed. In the long term, I truly hope more employers like BCBST, industry partners like Google, and educational partners like ETSU can come together to support students in preparing for in-demand jobs of the future, especially within under-represented student populations who have the least access to learning this content. This is a model we’d like to see replicated many times over. 

From Ohio to Asia Pacific, Google Cloud continues to support state and local organizations on their mission to equip local talent with in-demand cloud skills. 

To request more information or apply to the BlueSky Tennessee Institute, click here. To learn more about Google’s workforce development initiatives, check out Grow with Google, Google Cloud Skills Boost, and Google Cloud for Faculty Programs.

Blockchain is changing the way the world stores and moves its information. Building on our commitment to help Web3 developers build and deploy new products on blockchain-based platforms, today we’re thrilled to announce Google Cloud’s Blockchain Node Engine. 

Blockchains consist of transaction data that is permanently stored and encrypted, acting as decentralized databases of sorts. Instead of relying on a central entity to validate and store data, the governing infrastructure of a blockchain is a node — a device, such as a computer, laptop, or server, that contains a full copy of the transaction history of the blockchain. Nodes on a blockchain form a peer-to-peer network, constantly exchanging the latest blockchain data so that all nodes stay in sync.

While self-managed nodes are often difficult to deploy and require constant management, Blockchain Node Engine is a fully managed node-hosting service that can minimize the need for node operations. Web3 companies who require dedicated nodes can relay transactions, deploy smart contracts, and read or write blockchain data with the reliability, performance, and security they expect from Google Cloud compute and network infrastructure. 

Ethereum will be the first blockchain supported by Blockchain Node Engine, enabling developers to provision fully managed Ethereum nodes with secure blockchain access.

With Blockchain Node Engine, Web3 organizations enjoy the following benefits:

Streamlined provisioning
Today, manually deploying a node is a time-intensive process that involves provisioning a compute instance, installing an Ethereum client (e.g. geth), and waiting for the node to sync with the network. Syncing a full node from the first block (i.e., “genesis”) can take several days. Google Cloud’s Blockchain Node Engine can make this process faster and easier by allowing developers to deploy a new node with a single operation, and specify the desired region and network (mainnet, testnet). 

Secure development
Customers that wish to secure their blockchain infrastructures currently have a limited set of options in the market. Blockchain Node Engine offers security configurations that can help prevent unauthorized access to the nodes. By placing nodes behind a Virtual Private Cloud firewall, only trusted machines and users are able to communicate with client endpoints. Additionally, other Google Cloud services such as Cloud Armor can help protect nodes from DDoS attacks. 

Fully managed operations
Ensuring that blockchain infrastructure is ready and available, typically means having a DevOps team on-hand to monitor the system and remediate problems during an outage. Blockchain Node Engine is a fully managed service, which can mean you don’t have to worry about availability. Google Cloud actively monitors the nodes and restarts them during outages as needed. By reducing the need for a dedicated DevOps team, and by offering Google Cloud’s  service level agreement (SLA), Blockchain Node Engine can let your team focus on your users instead of your infrastructure.

Scaling Web3 innovation 

We look forward to supporting organizations with a reliable, easy-to-use blockchain node hosting service so they can focus their time on innovating and scaling their Web3 applications. 

To learn more about Blockchain Node Engine, please visit Google Cloud for Web3. And to try Blockchain Node Engine for yourself, reach out to the Google Cloud sales team for access to the Private Preview.