We’re excited to announce our first-ever Google Cloud Startup Summit will be taking place on September 9, 2021.

At Google Cloud, we’re invested in helping maximize your potential using all that Google has to offer. From my experience in venture capital, mentoring founders and working in the technology industry, I know how important it can be to get the right support as a startup while you embark on your journey. 

We hope you will join us as we bring together our startup community, including startup founders, CTOs, VCs and Google experts to provide behind the scenes insights and inspiring stories of innovation. To kick off the event, we’ll be bringing in X’s Captain of Moonshots, Astro Teller, for a keynote focused on innovation. We’ll also have exciting technical and business sessions, with Google leaders, industry experts, venture investors and startup leaders. You can see the full agenda here to get more details on the sessions.

We can’t wait to see you at the Google Cloud Startup Summit at 10am PT on September 9! Register to secure your spot today.

The financial services industry is evolving at a rapid pace, with shifting consumer expectations, new technologies, and developing regulatory requirements. Financial services firms need the right technology to help them stay agile and prepare for the future. 

The cloud is a key point of leverage for firms looking to improve performance across a broad range of activities. Moving to the public cloud can advance operational resiliency, improve staff productivity, increase regulatory compliance and enhance business model innovation. 

However, there are a number of financial services companies that are still hesitant in their cloud journeys. The barriers to adoption vary, from the complexity of legacy systems, to trust and skills gaps, regulatory uncertainty, and fragmentation of compliance requirements. Although many companies have embraced the benefits of cloud technology, more robust cloud adoption—especially around core back-office functions—will require additional facilitation, including through regulatory harmonization and streamlining.

A new comprehensive study on cloud adoption in financial services

To better understand the challenges and opportunities of cloud adoption in financial services Google Cloud, together with the Harris Poll, surveyed more than 1,300 leaders from the financial services industry across the United States, Canada, France, Germany, United Kingdom, Hong Kong, Japan, Singapore and Australia. 

There were five noteworthy takeaways from the study:  

1. A vast majority of financial services companies are already using some form of public cloud. A large number of surveyed financial services companies (83%) report they are deploying cloud technology as part of their primary computing infrastructures. Of those using cloud technology, the most popular architecture of choice is hybrid cloud (38%), followed by single cloud (28%), and multicloud (17%). Notably, of respondents without a multicloud deployment, 88% reported they are considering adopting a multicloud strategy in the next 12 months.

2. Financial services institutions in North America are leading in cloud adoption. Of the financial services companies who are implementing a cloud strategy, the highest levels of cloud workload adoption were reported in North America, with institutions in the U.S. (54%) and Canada (52%) leading the way. The lowest level of cloud adoption was reported in Japan (42%).

3. As financial services companies continue to use the cloud, more core functionalities can and will be migrated. While many financial services companies have migrated substantial workloads to the cloud, the industry is far from full adoption when it comes to core, back-office workloads. Of financial services companies currently using a majority cloud strategy in the United States, for example, only half (54%) of their workloads are fully deployed in the cloud. Data and IT security (74%), regulatory reporting (57%), and fraud detection and prevention (57%) rank among the highest workload adoption. Core underwriting activity (40%) and data reconciliation (48%) ranked lowest. Across Europe, cloud usage for core activities like underwriting also scored low with the UK listing only 30% adoption.

4. Among respondents, there is a very strong positive perception of the potential for cloud technology to assist in business operations and regulatory compliance. Nearly all respondents (>88%) agreed that cloud adoption can:

help adapt to changing customer behaviors and expectations,  

enhance operational resilience, 

support the creation of innovative new products and services, 

enhance financial services institutions’ data security capabilities, and 

better connect siloed legacy software infrastructure within financial services institutions. 

5. Certain regulator-induced challenges, including the complexity of sectorial compliance frameworks and fragmentation, create hurdles to cloud adoption for financial services companies. While 88% of respondents had a positive view of current regulatory efforts to provide guidance and clarity for cloud implementation, the results showed that more needs to be done to facilitate adoption. Most respondents (84%) agree that regulatory reviews and approvals take too long because of regulatory fragmentation across regulatory bodies. And 78% say that regulatory uncertainty over the use of public cloud prevents their organizations from adopting cloud technologies that would otherwise provide benefit to them. Additionally, a third of all on-premises respondents (38%) say that the large investment of resources for the regulatory approval process is a reason why they’re not using cloud services.

“While many banks have already deployed hybrid cloud environments, others are still in various stages of planning and deploying,” said Jerry Silva, research vice president for IDC Financial Insights. “Clearly, hybrid infrastructure is a reality, and financial institutions must focus not only on leveraging the modern infrastructure model to gain efficiencies, resilience and agility, but also on taking the necessary steps to manage such environments, including the security and compliance of cloud services.”

Future recommendations for financial services regulators

Financial services firms should continue to maximize the potential of technology by migrating more core workloads to the cloud, and actively considering multicloud and hybrid-cloud strategies. Such strategies enhance resiliency of existing IT infrastructure and reduce concerns over vendor lock-in. 

The research also points to steps that regulators could take to provide additional clarity and guidance, such as aligning regulatory reviews across agencies to avoid fragmentation; developing regulatory “safe harbors” for cloud adopters based on adherence to accepted standards and best practices; training regulatory staff on emerging tech; and advancing data reporting requirements via cloud and related technologies.

In the past few years, many regulators across the globe have taken a robust approach to rationalizing rules and guidance to cloud adoption in the financial sector, which has helped significantly stimulate adoption. But further assurances and harmonization of best practices around supervision is needed to advance risk-based and secure digital innovation.  

At Google Cloud, we’re committed to working with financial services customers and regulators to provide them with controls and assurances on risk management, data locality, transparency, and compliance. We are constantly engaging with regulators to share information, respond to their considerations and concerns, and address questions in the interest of transparency and building trust. 

To learn more about these findings and more, download our infographic and our full report. 

Research methodology

The survey was conducted online by the Harris Poll on behalf of Google Cloud, from December 7, 2020, to January 4, 2021, among 1,363 senior executives in France (n=113), Germany (n=178), the UK (n=192), Hong Kong (n=99), Indonesia (n=100), Japan (n=142), Singapore (n=71), Australia (n=134), Canada (134), and the United States (n=200) who are employed full-time, part-time, or self-employed whose main functional role is in risk/compliance or IT at a company in the banking, finance, or financial services industry with a title of director level or higher. The data in each country were weighted by the number of employees to bring them into line with actual company size proportions in the population. A global post-weight was applied to ensure equal weight of each country in the global total.

Google provides a set of Dataflow templates that customers commonly use for frequent data tasks, but also as reference data pipelines that developers can extend. But what if you want to customize a Dataflow template without modifying or maintaining the Dataflow template code itself? With user-defined functions (UDFs), customers can extend certain Dataflow templates with their custom logic to transform records on the fly.

A UDF is a JavaScript snippet that implements a simple element processing logic, and is provided as an input parameter to the Dataflow pipeline. This is especially helpful for users who want to customize the pipeline’s output format without having to re-compile or to maintain the template code itself. Example use cases include enriching records with additional fields, redacting some sensitive fields, or filtering out undesired records – we’ll dive into each of those. That means you do not have to be an Apache Beam developer or even have a developer environment setup in order to tweak the output of these Dataflow templates!

As the time of this writing, the following Google-provided Dataflow templates support UDF:

Pub/Sub to BigQuery

Pub/Sub to Datastore

Pub/Sub to Splunk

Pub/Sub to MongoDB

Datastore to GCS Text

Datastore to Pub/Sub

Cloud Spanner to Cloud Storage Text

GCS Text to Datastore

GCS Text to BigQuery (Batch and Stream)

Apache Kafka to BigQuery

Datastore Bulk Delete

Note: While the UDF concepts described here apply to any Dataflow template that supports UDF, the utility UDF samples below are from real-world use cases using the Pub/Sub to Splunk Dataflow template, but you can re-use those as starting point for this or other Dataflow templates. 

How UDF works with templates

When a UDF is provided, the UDF JavaScript code runs in the Nashorn JavaScript engine included in the Dataflow worker’s Java runtime (applicable for Java pipelines such as Google-provided Dataflow templates). The code is invoked locally by a Dataflow worker for each element separately. Element payloads are serialized and passed as JSON strings back and forth.

Here’s the format of a Dataflow UDF function called process which you can reuse and insert your own custom transformation logic into:

Using JavaScript’s standard built-in JSON object, the UDF first parses the stringified element inJson into a variable obj, and, at the end, it must return a stringified version outJson of the modified element obj. Where highlighted, you add your custom element transformation logic depending on your use case. In the next section, we provide you with utility UDF samples from real-world use cases. 

Note: The variable includePubsubMessage is required if the UDF is applied to Pub/Sub to Splunk Dataflow template since it supports two possible element formats: that specific template can be configured to process the full Pub/Sub message payload or only the underlying Pub/Sub message data payload (default behavior). The statement setting data variable is needed to normalize the UDF input payload in order to simplify your subsequent transformation logic in the UDF, consistent with the examples below. For more context, see includePubsubMessage parameter in Pub/Sub to Splunk template documentation.

Common UDF patterns

The following code snippets are example transformation logic to be inserted in the above UDF process function. They are grouped below by common patterns.

Pattern 1: Enrich events

Follow this pattern to enrich events with new fields for more contextual information.

Example 1.1: 

Add a new field as metadata to track pipeline’s input Pub/Sub subscription

Example 1.2*: 

Set Splunk HEC metadata source field to track pipeline’s input Pub/Sub subscription

Example 1.3: 

Add new fields based on a user-defined local function e.g. callerToAppIdLookup() acting as a static mapping or lookup table

Pattern 2: Transform events

Follow this pattern to transform the entire event format depending on what your destination expects.

Example 2.1: 

Revert logs from Cloud Logging log payload (LogEntry) to original raw log string. You may use this pattern with VM application or system logs (e.g. syslog or Windows Event Logs) to send source raw logs (instead of JSON payloads):

Example 2.2*: 

Transform logs from Cloud Logging log payload (LogEntry) to original raw log string by setting Splunk HEC event metadata. Use this pattern with application or VM logs (e.g. syslog or Windows Event Logs) to index original raw logs (instead of JSON payload) for compatibility with downstream analytics. This example also enriches logs by setting HEC fields metadata to incoming resource labels metadata:

Pattern 3: Redact events

Follow this pattern to redact or remove a part of the event.

Example 3.1: 

Delete or redact sensitive SQL query field from BigQuery AuditData data access logs:

Pattern 4: Route events

Follow this pattern to programmatically route events to separate destinations.

Example 4.1*: 

Route event to the correct Splunk index per used-defined local function e.g. splunkIndexLookup() acting as a static mapping or lookup table:

Example 4.2: 

Route unrecognized or unsupported events to Pub/Sub deadletter topic (if configured) in order to avoid invalid data or unnecessary consumption of downstream sinks such as BigQuery or Splunk:

Pattern 5: Filter events

Follow this pattern to filter out undesired or unrecognized events.

Example 5.1: 

Drop events from a particular resource type or log type, e.g. filter out verbose Dataflow operational logs such as worker & system logs:

Example 5.2: 

Drop events from a particular log type, e.g. Cloud Run application stdout:

* Example applicable to Pub/Sub to Splunk Dataflow template only

Testing UDFs

Besides ensuring functional correctness, you must verify your UDF code is syntactically correct JavaScript on Nashorn JavaScript engine which is shipped as part of JDK (8 through 14) pre-installed in Dataflow workers. That’s where your UDF ultimately runs. Before pipeline deployment, it is highly recommended to test your UDF on Nashorn engine: any JavaScript syntax error will throw an exception, potentially on every message. This will cause a pipeline outage as the UDF is unable to process those messages in-flight.

At the time of this writing, Google-provided Dataflow templates run on JDK 11 environment with the corresponding Nashorn engine v11 release. By default, Nashorn engine is only ECMAScript 5.1 (ES5) compliant so a lot of newer ES6 JavaScript keywords like let or const will cause syntax errors. In addition, it’s important to note that Oracle Nashorn engine is a slightly different JavaScript implementation than Node.js. A common pitfall is using console.log() or Number.isNaN() for example, neither of which are defined in the Nashorn engine. For more details, see this introduction to using Oracle Nashorn. That said, using the utility UDFs provided above without major code changes should be sufficient for most use cases.

An easy way to test your UDF on Nashorn engine is by launching Cloud Shell where JDK 11 is pre-installed, including jjs command-line tool to invoke Nashorn engine.

Let’s assume your UDF is saved in dataflow_udf_transform.js JavaScript file and that you’re using UDF example 1.1 above which appends new inputSubscription field.

In Cloud Shell, you can launch Nashorn in interactive mode as follows:

In Nashorn interactive shell, first load your UDF JavaScript file which will load the UDF ‘process’ function in global scope:

To test your UDF, define an arbitrary input JSON object depending on your pipeline’s expected in-flight messages. In this example, we’re using a snippet of a Dataflow job log message to be processed by our pipeline:

You can now invoke your UDF function to process that input object as follows:

Notice how the input object is serialized first before being passed to UDF which expects an input string as noted in the previous section.

Print the UDF output to view the transformed log with the appended inputSubscription field as expected:

Finally exit the interactive shell:

Deploying UDFs

You deploy a UDF when you run a Dataflow job by referencing a GCS file containing the UDF JavaScript file. Here’s an example using gcloud CLI to run a job using the Pub/Sub to Splunk Dataflow template:

The relevant parameters to configure:

gcs-location: GCS location path to the Dataflow template

javascriptTextTransformGcsPath: GCS location path to JavaScript file with your UDF code

javascriptTextTransformFunctionName: Name of JavaScript function to call as your UDF

As a Dataflow user or operator, you simply reference a pre-existing template URL (Google-hosted), and your custom UDF (Customer-hosted) without the requirement to have a Beam developer environment setup or to maintain the template code itself.

Note: The Dataflow worker service account used must have access to the GCS object (JavaScript file) containing your UDF function. Refer to Dataflow user docs to learn more about Dataflow worker service account.

What’s Next?

We hope this helps you get started with customizing some of the off-the-shelf Google-provided Dataflow templates using one of the above utility UDFs or writing your own UDF function. As a technical artifact of your pipeline deployment, the UDF is a component of your infrastructure, and so we recommend you follow Infrastructure-as-Code (IaC) best practices including version-controlling your UDF. If you have questions or suggestions for other utility UDFs, we’d like to hear from you: create an issue directly in GitHub repo, or ask away in our Stack Overflow forum.

In a follow-up blog post, we’ll dive deeper into testing UDFs (unit tests and end-to-end pipeline tests) as well as setting up a CI/CD pipeline (for your pipelines!) including triggering new deployment every time you update your UDFs – all without maintaining any Apache Beam code.

In May we announced Vertex AI, our new unified AI platform which provides options for everything from using pre-trained models to building your models with a variety of frameworks. In this post we’ll do a deep dive on training and deploying a custom model on Vertex AI. There are many different tools provided in Vertex AI, as you can see in the diagram below. In this scenario we’ll be using the products highlighted in green:

AutoML is a great choice if you don’t want to write your model code yourself, but many organizations have scenarios that require building custom models with open-source ML frameworks like TensorFlow, XGBoost, or PyTorch. In this example, we’ll build a custom TensorFlow model (built upon this tutorial) that predicts the fuel efficiency of a vehicle, using the Auto MPG dataset from Kaggle.

If you’d prefer to dive right in, check out the codelab or watch the two minute video below for a quick overview of our demo scenario: 

Environment setup 

There are many options for setting up an environment to run these training and prediction steps. In the lab linked above, we use the IDE in Cloud Shell to build our model training application, and we pass our training code to Vertex AI as a Docker container. You can use whichever IDE you’re most comfortable working with, and if you’d prefer not to containerize your training code, you can create a Python package that runs on one of Vertex AI’s supported pre-built containers.

If you would like to use Pandas or another data science library to do exploratory data analysis, you can use the hosted Jupyter notebooks in Vertex AI as your IDE. For example, here we wanted to inspect the correlation between fuel efficiency and one of our data attributes, cylinders. We used Pandas to plot this relationship directly in our notebook.

To get started, you’ll want to make sure you have a Google Cloud project with the relevant services enabled. You can enable all the products we’ll be using in one command using the gcloud SDK:

Then create a Cloud Storage bucket to store our saved model assets. With that, you’re ready to start developing your model training code.

Containerizing training code

Here we’ll develop our training code as a Docker container, and deploy that container to Google Container Registry (GCR). To do that, create a directory with a Dockerfile at the root, along with a trainer subdirectory containing a train.py file. This is where you’ll write the bulk of your training code:

To train this model, we’ll build a deep neural network using the Keras Sequential Model API:

We won’t include the full model training code here, but you can find it in this step of the codelab. Once your training code is complete, you can build and test your container locally. The IMAGE_URI in the snippet below corresponds to the location where you’ll deploy your container image in GCR. Replace $GOOGLE_CLOUD_PROJECT below with the name of your Cloud project:

All that’s left to do is push your container to GCR by running docker push $IMAGE_URI. In the GCR section of your console, you should see your newly deployed container:

Running the training job 

Now you’re ready to train your model. You can select the container you created above in the models section of the platform. You can also specify key details like the training method, compute preferences (GPUs, RAM, etc.) and hyperparameter tuning if required.

Now you can hand over training your model and let Vertex do the heavy lifting for you.

Deploy to endpoint

Next, let’s get your new model incorporated into your app or service. Once your model is done training you will see an option to create a new endpoint. You can test out your endpoint in the console during your development process. Using the client libraries, you can easily create a reference to your endpoint and get a prediction with a single line of code:

Start building today

Ready to start using Vertex AI? We have you covered for all your use cases spanning from simply using pre-trained models to every step of the lifecycle of a custom model. 

Use Jupyter notebooks for a development experience that combines text, code and data
Fewer lines of code required for custom modeling
Use MLOps to manage your data with confidence and scale

Get started today by trying out this codelab yourself or watching this one hour workshop. 

Data-savvy businesses increasingly rely on images and videos for critical functions, and yet are challenged by the sheer mass of information—more than 3.2 billion images and 720,000 hours of video are created daily. This explosion in visual data has paved the way for the growth of computer vision, a form of artificial intelligence (AI) that enables computers to “see” the world similarly to the way people do, but with unblinking consistency, and greater accuracy. 

The transformational impact and value of computer vision solutions are significant and has been a guiding objective for companies and AI developers. And yet, even as the applications for computer vision increase dramatically, architecting and implementing vision AI solutions remain highly complex. Visual data, such as images and video, are made up of thousands of pixels of information that represent millions of different patterns and meanings, which can make interpreting even a single image overwhelming from a computational perspective.

Many organizations struggle with deployments and fail to operationalize vision AI solutions due to development delays, machine learning and data science hiring challenges, inaccurate output, a lack of integration with existing infrastructure, difficulty of use, and high cost. Plainsight, with the power of Google Cloud resources, is addressing all these challenges and helping businesses by enabling the deployment of vision AI within enterprise private networks that can be managed easily and scaled economically.

Plainsight has announced  availability of its vision AI platform on Google Cloud Marketplace. Businesses can now easily deploy end-to-end vision AI to private clouds to realize the full value of their video and other visual data for accurate, actionable insights across diverse use cases.

Delivering on the Promise of AI: Seeing What’s Hiding In Plain Sight

For organizations to integrate AI and machine learning into their businesses successfully, the technology must be powerful enough to solve real challenges, yet fast, easy, and accessible enough to ensure the innovation potential is realized. Plainsight on Google Cloud delivers the power of enterprise vision AI that’s quick and easy to use with Google Cloud resources that enable global scale, increased security, bolstered privacy, unified billing, and cost savings. 

To streamline vision AI workflows, Plainsight facilitates the entire pipeline, from visual data ingestion and annotation, through continuous model training, deployment, and monitoring for easier innovation and faster time-to-production. Our platform accelerates vision AI development in a manner that is complete, accurate, and accessible to non-technical business leaders. We believe that AI should be available and accessible to anyone and everyone—so that teams across entire organizations can reap the benefits. 

By integrating Plainsight into their private networks, companies worldwide can now leverage one intuitive platform for centralized control of streamlined vision AI model creation and training with optimized visual data handling for diverse enterprise solutions. These use cases include: social distancing monitoring, medical imaging, drug compound screening, defect detection in manufacturing processes, identifying gas leaks, or even livestock counting and crop health monitoring for agriculture, to name a few.

We enable customers so they can create successful solutions that enable them to clearly see their business from all angles and to take advantage of the knowledge visual data can reveal by simply and quickly operationalizing practical vision AI applications.

AI-Powered Dataset Creation, Automated Model Training & Easy Deployment Without A Single Line Of Code 

For vision AI applications, success is inextricably dependent on the quality and quantity of the datasets required to train the relevant models. To aid enterprises in this vital stage, the Plainsight platform provides built-in data annotation for the fast and easy creation of datasets. This includes AI-powered features that accelerate the speed and quality of labeling such as SmartPoly, for the automated polygon masking of objects, TrackForward, to predict and automatically label objects from frame to frame in video annotations, and AutoLabel for automated object recognition and labeling based on pre-trained machine learning models, to highlight a few.

In addition, to ensure the success of AI integration, we significantly reduce time-intensive processes with Plainsight vision AI’s automated machine learning with continuous model training and easy deployment capabilities. In just a few clicks, users can leverage optimizations for the most reliable model training without endless experimentation cycles. And, models are easily deployed at scale all within one, easy-to-manage model operationalization process for the business.

Growing With Google Cloud

Plainsight is a vision AI innovation leader, developing solutions that address unmet needs for challenger brands and Fortune 500s across vertical markets. As a team recognized for succeeding where others have failed, our expanding partnership with Google Cloud provides a powerful combination that helps customers see and activate the value of their visual data with a suite of services in a secure and private manner.

Our vision AI Platform simplifies building and operationalizing AI to solve business problems enterprises are facing every day—and the demand is increasing. To accelerate our  journey to faster, more accessible AI for enterprises, we knew we needed strong support to grow Plainsight and scale our backend tools to match our vision. 

Google Cloud delivered everything, and more, in one program. The Startup Program by Google Cloud provided the technology and services for scale and the support we needed to maximize the value the Program provided us. The Startup Program has been a springboard for architecting  Plainsight vision AI in the cloud, accelerating our goals and optimizing innovation, efficiency, and growth. The team also helped us optimize Google Ads campaigns, fueling adoption of Plainsight. 

After launching the SaaS version of Plainsight Data Annotation in November 2020, we grew our user base by nearly 110x in just three short months. Google Ads has also dramatically increased website traffic, growing new users by nearly 5.75X and page views by over 5X. The Google team helped us identify where Google Cloud offerings could be leveraged instead of developing in-house solutions and offered best practices that enabled us to deliver faster on our initiatives. 

Kubernetes was already the underlying component of our platform and leveraging Google Kubernetes Engine (GKE) as a managed service removed a layer of complexity. By combining GKE and Anthos, we were able to standardize our deployments, aligning to how our customers leverage Anthos for enterprise applications in their own organizations. In addition, as a fast-moving, customer-centric company we use Google Workspace to help us centralize and manage our day-to-day work internally. By leveraging multiple products across Google’s ecosystem, we take advantage of a holistic partnership that has helped our business tremendously as we scale. 

Leveraging Google’s Partners for Strategic Consultation

To facilitate this expansion of our partnership with Google and to maximize our use of Google Cloud services, we are working with DoiT International, a Google Managed Services Provider and 2020 Global Reseller Partner of the Year. DoiT provides us with ongoing technical consultation for cloud-native architecture, Google Cloud Marketplace integration, production-grade Kubernetes support, Google Cloud cost optimization, and technical support. The DoiT team has been invaluable in compiling best practices, tips, and strategies from their vast experience with various cloud customers to ease our Marketplace integration and is providing input for infrastructure strategy to support our continued rapid growth.

Plainsight Delivers Enterprise Vision AI Through The Google Cloud Platform Marketplace

Plainsight vision AI is now available to Google Cloud Customers on Google Cloud Marketplace enabling organizations across industries to deploy private Plainsight instances within their own environments. Marketplace customers will benefit from Google Cloud privacy, security, scalability and unified billing through their Google Cloud account.  

Combining the powerful benefits provided by Google Cloud resources with Plainsight’s vision AI Platform into private networks, enterprises worldwide can now leverage one intuitive platform for centralized control of streamlined vision AI model creation and training with optimized visual data handling for diverse enterprise solutions. 

Through our Google partnership, we’re able to leverage a powerful foundation that allows us to rapidly innovate, scale and accelerate delivery on our vision AI platform capabilities. By executing on our vision to make AI easier, faster and more accessible for all users across entire enterprises, we’re helping businesses see more and by seeing more, they’ll have the power to solve more.  

If you want to learn more about how Google Cloud can help your startup, visit our page here where you can apply for our Startup Program, and sign up for our monthly startup newsletter to get a peek at our community activities, digital events, special offers, and more.

The global-wide interruptions of the coronavirus pandemic provided the opportunity for businesses to take a closer look at how we work, learn, live, and consume. With work stoppages and quarantine orders in place, carbon emissions and pollution levels saw significant reductions, highlighting how business and environmental sustainability are linked. As the world returns to work, businesses need to adjust to the new reality and make decisions that will ensure the continuing sustainability of our planet. 

Understanding the need for sustainability

A recent Google-commissioned study by IDG, based on a global survey of more than 2,000 IT decision-makers, illustrated the importance of sustainability for organizations’ IT departments while also revealing that significant work is needed before businesses meet their sustainability goals. The good news is that ninety percent say that sustainability is a priority of their IT department and as more organizations consider the benefits of digital transformation and cloud migration, 75% report that sustainability is a “must-have” or “major consideration” when evaluating cloud providers.

Businesses face both internal and external pressures to reform their practices and processes to meet new targets around sustainability, and IT leaders have an important role to play in that effort. Specifically, 54% said sustainability initiatives were important to their companies because of corporate social responsibility, 46% said it was because of environmental regulations and mandates, and 40% said it was due to the climate change impacts.

The research also examined the areas where IT leaders are looking to make positive changes for sustainability in their organizations. Not surprisingly, 62% said in their IT infrastructure and data centers, 53% their supply chain, and 46% in their facilities and building operations.

Facing the obstacles to progress

However, for many organizations, behaviors lag intent. Only 67% actually have environmental sustainability targets in place. The gap between behavior and intent widens when looking at digitally forward vs. digitally conservative organizations. For the purposes of the study, digitally forward companies were defined as either digital natives or those with a digital strategy and implementation in place. Digitally conservative organizations either have no strategy in place, or are in the process of considering or planning digital transformations, but haven’t reached the stage of implementation. 

The study revealed that digital natives embracing cloud solutions were more likely than more conservative organizations to demonstrate a commitment to sustainability. Twenty percent of digital natives are implementing IT initiatives to reduce emissions to become more environmentally sustainable, while only 10% of digitally conservative organizations have done so. 

When considering the effects of the COVID-10 pandemic, 20% of respondents from digital conservative organizations delayed or cancelled initiatives around reducing emissions to become more environmentally sustainable compared to only 12% for digitally forward respondents. 

Sustainability is in our DNA

Sustainability has been a core value for Google since our inception. We were the first major company to become carbon neutral in 2007 and we were the first major company to match our energy use with 100 percent renewable energy in 2017 — and have continued to do so every year since. We are proud to operate the cleanest global cloud in the industry, and we’re the world’s largest corporate purchaser of renewable energy. It is this commitment that makes Google the leading choice for organizations with sustainability initiatives. 

To learn more about the IDG findings and how IT leaders are implementing sustainable solutions, download the full report.

Interested in how Google Cloud’s commitment to providing sustainable solutions can help IT leaders and their organizations meet their own sustainability targets?

Google is focused on enhancing our products to help billions of people take action to reduce their environmental footprint.

Google creates tools and invests in technology that will foster a carbon-free future for everyone and reduce carbon footprints.

Google Cloud is the only major cloud provider to purchase enough renewable energy to cover our entire operations, enabling us to operate the cleanest cloud in the industry.

When running on Google Cloud, an organization’s usage is net carbon neutral, and the electricity used to power workloads is matched 100% with renewable energy.

Looking for more? Check out our blog that introduces the Google Cloud Region Picker, which helps organizations choose a Google Cloud region based on carbon footprint, price, and latency.

The Opportunity for Streamed AR/VR Content 

What if you could get a high quality AR/VR experience without a dedicated physical computer—or even without a physical tether?  

In the past, interacting with VR required a dedicated, high-end workstation and, depending on the headset, wall-mounted sensors and a dedicated physical space. Complex tasks in VR can push the limits of sensor reach, cable-length, and spatial boundaries, entangling the artist and restricting their movement. This solution was not scalable beyond a handful of advanced use cases.

Recently, tetherless VR headsets from manufacturers such as HTC and Oculus have emerged that free the user from these odious restrictions, enabling a new freedom to experience VR and AR from just about anywhere. The enhanced portability and reduced cost has led to substantially increased adoption among consumers and opened up many new exploratory use cases in enterprise.

However, while these headsets are more accessible and portable, a tradeoff in compute power was required in order to achieve these goals. The limited on-device compute power of this new generation of Head Mounted Displays (HMDs) is acceptable for many consumer applications such as casual gaming. However, advanced enterprise workloads with heavy graphics, compute, or memory requirements can be difficult or even impossible to run on these lightweight devices.

By combining NVIDIA CloudXR with their powerful GPUs running NVIDIA RTX Virtual Workstations (vWS) in Google Cloud data centers, you can experience high-fidelity VR and AR applications from just about anywhere with a good internet connection. The heavy computations are performed in the cloud on a GPU-attached VM and the content streams to any CloudXR compatible headset. 

The combination of Google Cloud’s private fiber optic network — the same network we built for global delivery of YouTube content — and CloudXR’s QoS technologies provides the user with the highest possible quality of service. In fact, the streaming experience is comparable to that of a headset tethered to a powerful physical workstation, but without the friction of hardware and cables. 

This combination of power and portability sets the stage to unlock the potential of high-quality gaming and enterprise AR/VR experiences anywhere on Earth.

Case Study: Creating a Masterpiece

Digital character creation is a core skill for many 3D artists today. One of the preferred methods of 3D character creation is digital sculpting, which lets artists create both hard-surface and organic shapes with high levels of accuracy and control.

Sculpting is one of the many tasks in the character designer’s skillset. Artists must also master the texturing, rigging, and posing of characters as part of the digital character creation pipeline. 

Mastering all these tasks can be challenging, and often, the technology gets in the way; working in multiple applications requires switching contexts and workflows, breaking the artist’s creative flow. The traditional user experience can also be unintuitive, forcing the creator to translate what they want to do with their hands and head into mouse movement and keyboard presses.

Masterpiece Studio Pro revolutionizes this character creation workflow by giving artists the first fully immersive 3D creation pipeline. Artists work in VR, giving them a far more intuitive and seamless way to work which combines the best of the digital and physical worlds.

In Masterpiece Studio Pro, the artist can work on the character or object at any scale, using familiar tools and hand gestures to sculpt a model, much as they would a real clay figure. Performing other tasks such as skeleton creation is simple, allowing the artist to work directly with the limbs of a character to place and adjust the joints.

Bringing It All Together

NVIDIA CloudXR streaming from the cloud provides tremendous opportunities for new creative use cases within gaming and across the enterprise. This solution joins Virtual Studio for Gaming as the latest in our series to help developers build better games.

The Masterpiece Studio use case is a powerful demonstration of new modalities for content creation and collaboration. To try CloudXR with NVIDIA RTX vWS on Google Cloud for yourself, see this tutorial. Masterpiece Studio Pro also offers extensive learning materials and a free trial.

When confronted with challenges, resilient organizations adapt and innovate to address changing customer needs. Through its partnership with Google Cloud, the Wisconsin Department of Workforce Development exemplifies this evolutionary approach. 

 Just a few months ago, challenges created by the antiquated unemployment insurance (UI) system threatened the agency’s ability to serve constituents who lost their jobs during the pandemic. Now, and in the months ahead, that continuing evolution will transform how the DWD serves its constituents with the modernization of their UI system, which includes customer experience workflows, predictive analytics, and streamlining and expediting paper applications.

Like many workforce development agencies nationwide, last year, DWD experienced a deluge of claims filed in response to the COVID-19 economic fallout. Unfortunately, DWD’s legacy UI infrastructure — largely written in the COBOL computing language and hosted on a mainframe server – was no match for the volume and complexity of cases. Wisconsin’s multiple technology systems required a large amount of manual processing, with staff using spreadsheets to manually calculate benefit adjustments. As a result, the state just couldn’t keep up with the surge of claims, and it needed to pivot quickly to a new solution to keep up with demand. 

DWD initially responded by staffing up call centers, hiring UI application adjudicators, and deploying other personnel–all told, hiring, contracting with, or reassigning more than 1,300 individuals. However, while this enabled DWD to respond to approximately 7 million calls per month, the massive number of incoming claims surged to create a  backlog of more than 750,000 claims. Legacy technology issues continued to significantly slow the query-response time. 

DWD leaders recognized that staffing alone was not the solution and that innovation was needed to overcome past, inadequate IT investment. To address the inherited issues, Wisconsin turned to Google Cloud. Working together, we were able to scale the state’s response to claims and speed up overall response time. We were also successful in screening out fraudulent claims so that the UI program could be administered–with integrity–to Wisconsinites who needed financial assistance.   

Year-to-date, Wisconsin has now disbursed $2 billion in unemployment benefits, in addition to successfully clearing its entire 2020 UI backlog. As a result of our partnership, the state is now processing an average of 157,000 claims each week and releasing most payments to citizens within two to three business days. Before the new system was in place, the response time could be weeks or even months.

Here is how Wisconsin and Google Cloud are modernizing the state’s current legacy system through a modular approach:

Artificial intelligence (AI)/ machine learning (ML) for predictive analytics: Through the use of Google AI/ML, the state is creating predictive analytics based on historical data to shorten adjudication decision-making for UI claims, enabling it to release payments to eligible claimants faster. Comprehensive data models and confidence scores analyze the backlog data to determine the shortest route to approval and payments, with a high level of confidence and accuracy. DWD is using Google Cloud technology to identify where in the process the claimant was getting stuck in one of the “hold buckets” for processing a UI claim. Using Google Cloud’s data and AI/ML tools, Wisconsin is able to identify problem areas and quickly resolve those issues. This informed DWD’s rewrite of the UI claim application process. And it also helped DWD identify fraudulent claims.

Document AI (DocAI) for streamlining paper applications: DWD is also partnering with Google Cloud to streamline paper applications and fax documents as part of UI claims processing–enabling documents to be submitted online instead of by fax or hard-copy mail. Our DocAI solution helps Wisconsin staff make faster decisions by rapidly extracting critical data from documents, saving time, and removing manual processes, which allows employees to focus on high-priority activities. 

These modernization steps in Wisconsin are resulting in a bold new vision for state unemployment systems across the United States. Through a combination of design thinking, deep partnership with state officials, and modern technology, DWD’s solutions are tailored to maximize benefits to the constituents they’re designed to serve. By joining forces with Google Cloud and utilizing modernizing technology to make informed, data-driven decisions, Wisconsin DWD is helping residents have a better experience that is easier to understand and navigate–all while better serving the community overall.

With everyone and their dog shifting to containers, and away from virtual machines (VMs), we realized that running vendor-provided software on VMs at Google was slowing us down. 

So we moved. 

Enter Anthos, Google Cloud’s managed application platform, and its associated developer tools. Today we’ll take you through our process of moving Confluence and Acrolinx from VMs running in our private data center environment over to a fully managed, containerized deployment for Google. Both Confluence and Acrolinx were deployed before on the Google Compute Engine platform and have been used within Google for content management. 

In the past, Google used internal systems for allocating application resources, automating replication and providing high availability for enterprise applications, but these systems relied on customized infrastructure and they were often incompatible with enterprise software. 

The many frustrations that came with running enterprise applications on VMs included:

Service turnup times in the order of days

Hard-to-manage infrastructure and workloads programmatically

Challenges with VM monoliths management (as compared to microservices)

Reliable rollback of application install/upgrade failures

Challenges with enforcing security policies at scale

… and many others

To mitigate these frustrations, we made the shift to an industry-standard, universally available managed platform: Kubernetes.

Kubernetes and Anthos

Deploying Kubernetes gave us the ability to configure, manage, and extend workloads running on containers rather than VMs. The good news was that it could handle the scale of our deployments with ease.

Anthos is Google Cloud’s platform of tools and technologies designed to ease the management of containerized workloads, whether running on Google Cloud, other clouds, or on-premises. It brings configuration management, service management, telemetry, logging and cluster management tooling. In addition, it saves operational overhead for our application teams.

As our vendor-provided software became compatible with containerization, we could build on 15 years of experience running containerized workloads and enjoy the perks of using a fully managed cloud service for our applications.

Adopting Anthos gave us some big benefits right away:

Automated resource provisioning

Application lifecycle management

Security Policies Management

Config-as-code for workload state

This removed substantial manual toil from our team, freeing them up for more productive work. Using Anthos Config Connector we could express the compute, networking and storage needs through code, allowing Anthos to allocate them without manual interaction. We also relied on Anthos to administer creating Kubernetes clusters and manage a single admin cluster that would host the Config Connector. This gave us simpler orchestration when we needed to create new Kubernetes clusters to run our applications.

How we modernized operations

Our continuous integration and continuous deployment process benefitted from Anthos as well. By using Anthos Config Management (Config Sync), a multi-repository configuration sync utility, we can automate the process of applying our desired configuration to the Kubernetes clusters that we would otherwise have applied manually before via kubectl. The multi-repo Config Sync provides a consistent experience when managing both the common security policies across clusters and the workload specific configs that are namespace-scoped.

Config Sync is a Kubernetes Custom Resource Definition (CRD) resource which is installed on a user cluster by GKE Hub.

GKE Hub provides networking assistance within Anthos, and lets you logically group together similar GKE clusters. Once the clusters are registered with a GKE Hub, the same security policies can be administered on all the registered clusters. Onboarding a new application then wouldn’t incur any additional overhead, because the same security policies would be applied automatically.

The resulting clusters and administration of these applications looks like this:

Our updated deployment process

We’ve deployed a variety of third-party applications on Anthos. Today, we’ll walk you through how we set up Confluence and Acrolinx.

To provision and deploy, we need to:

Ensure that all the configs (both security policies and workload configs) are stored in a single source of truth (i.e Git repos). Any changes must be reviewed and approved by multiple parties to prevent unilateral changes.

Deploy and enforce our required security policies.

Express the desired state of workload configs in a Git repo.

Deploy a Continuous Integration and Continuous Deployment pipeline to ensure that changes to the configs are tested before committing them to a Git repo. Such configs will then be applied to the target clusters to ensure the desired state of both the applications.

Even though we’re running multiple segmented workloads, we can apply common security policies to them all. We also delegate application deployment to the developers while maintaining security guardrails to prevent mistakes.

How we set up Anthos clusters

We know what we want to deploy, and how to protect them. Let’s dig into how we can set up those clusters with Terraform and then how to make sure all our security policies are applied. Once that is complete we can let the developer or operator manage any future changes to the application, while the cluster admin retains control of any cluster policy changes.

We’ll register the cluster with the right GKE Hub, then apply our desired configuration to that cluster, and finally deploy the applications to their namespaces.

Let’s start with the prod GKE cluster. We can create it using these Terraform templates,then cluster with GKE Hub using:

Next, we will enable the ACM/Config Sync feature for the GKE Hub, hub-prod, using the gcloud command-line:

Here, a ConfigManagement resource configures Config Sync on the prod GKE cluster with the relevant root Git repo (root-prod).

After creating the GKE clusters, we will set up cluster namespaces to deploy Confluence and Acrolinx:

Here’s one way the root and namespace repos can be organized in a root-prod structured repo.

All the cluster-scoped resources will be kept in the cluster directory while all the namespace scoped resources for the given applications will be kept in each of the namespaces sub-directories. This separation allows us to define the common cluster scoped security policies at a higher level while still defining application configs at each application namespace level. The cluster admins can own the security policies while delegating namespace ownership to the developers. 

We now have a GKE cluster prod that is registered with a GKE Hub. Since the cluster is registered with the GKE Hub with Config Sync enabled, the security policies now can be applied on this cluster.

Deploying changes to applications

In order for Config Sync to apply config changes to resources of Confluence and Acrolinx applications, each of the Namespace resources and Namespace repos must first be configured.

Looking at an example of a root-prod Git repo as shown above and the respective Namespaces repos, RepoSync resources and how Confluence and Acrolinx application resources will be managed by Config Sync in the prod GKE cluster.

The following is an example of a Namespace and RepoSync resource in the confluence-prod directory.

Config Sync will read the Namespace config file and create confluence-prod Namespace in the same prod GKE cluster. 

The RepoSync resource sets up a process to connect to the Git repo to find configuration information that will be used by the Confluence application.

We are now ready to create Kubernetes resources for Confluence from its namespace Git repo.

Next, we can deploy a StatefulSet resource that defines the container’s spec (CPU, RAM, etc.) for running Confluence app in the confluence-prod namespace repo: 

After submission to the repo, Config Sync will read the StatefulSet and deploy the image based on the resources listed.

Our security practice

Every organization has a requirement to ensure that the workloads are made secure without any additional efforts from the developers and that there is a central governing process that enforces such security policies across all the workloads. Thisensures that everyone follows best practices when deploying workloads.  It also reduces much of the burden and cognitive load from the developers when ensuring that workloads follow such security principles and policies.

Historically, when running applications on VMs, it has been traditionally difficult to micro-segment applications, apply a different set of policies to the micro-segmented applications and/or based on workload identities. Some examples of such policies are: whether an application is built and deployed in a verifiable manner; preventing privilege escalation (e.g setuid binaries) and applying that config for a group of workloads etc.

With the advent of Kubernetes and standards such as OPA (Open Policy Agent), it is possible now to micro-segment workloads, define a set of policies that can enforce certain constraints and rules at the workload identity level for a group of similar workload resources. This is one such library of OPA Constraints that can be used to enforce policy across Cluster workloads.

Policy Controller enables the enforcement of fully programmable policies. You can use these policies to actively block non-compliant API requests, or simply to audit the configuration of your clusters and report violations. Policy Controller is based on the open source Open Policy Agent Gatekeeper project and comes with a full library of pre-built policies for common security and compliance controls.

Conclusion

In the end we got to a much better place by deploying our applications with Anthos, backed by Kubernetes.

Our security policies were enforced automatically, we scale up and down with demand, and new versions could be deployed smoothly. Our developers enjoyed faster workflows, whether spinning up a new environment or testing out an update for stability. Provisioning got easier too, with less overhead for the team, especially as deployments grew to service the whole of Google.

Overall we’re quite happy with how we improved developer productivity with faster application turnup times, going from days to just hours for a new application. At the same time we’re better able to reliably enforce policies that ensure that applications are hosted in a secure and reliable environment. 

We’re glad we can share some of our journey with you; if you want to try it out yourself, get started with Anthos today.

A reverse proxy stands in front of your data, services, or virtual machines, catching requests from anywhere in the world and carefully checking each one to see if it is allowed.

Who are you (the individual making the request)? What is your role? Do you have access permission (authorization)?

What device are you using to make the request? How healthy is your device right now? Where are you located? 

At what time are you making the request?

This issue of GCP Comics presents an example of accessing some rather confidential data from an airplane, and uses that airplane as a metaphor to explain what the proxy is doing.

Reverse proxies work as part of the load balancing step when requests are made to web apps or services, and they can be thought of as another element of the network infrastructure that helps route requests to the right place. No one can access your resources unless they meet certain rules and conditions.

If a request is invalid or doesn’t meet the necessary criteria set by your administrators, either because it is from an unauthorized person or an unsafe device, then the proxy will deny the request.

Why might the proxy say no to my request? When assessing the user making the request, denial of access could be due to reasons such as:

I’m in Engineering, but I am trying to access Finance data.
I’m not even a part of the company.
My job changed, and I lost access.

Looking at the device originating the request, the proxy could deny access due to a number of factors, such as:

Device operating system out of date
Malware detected
Device is not reporting in
Disk encryption missing
Device doesn’t have screen lock

Leveraging identity and device information to secure access to your organization’s resources improves your security posture.

Resources

To learn more about proxies and Zero Trust, check out the following resources:

Overview of Identity-Aware Proxy

Extending Zero Trust models to the web
Creating a device-based access level
How to set up a proxy for on-premises apps
BeyondCorp Enterprise Quickstart Guide

Want more GCP Comics? Visit gcpcomics.com & follow us on medium pvergadia & max-saltonstall, and on Twitter at @pvergadia and @maxsaltonstall. Be sure not to miss the next issue!

When you’re troubleshooting an application on Google Kubernetes Engine (GKE), the more context that you have on the issue, the faster you can resolve it. For example, did the pod exceed it’s memory allocation? Was there a permissions error reserving the storage volume? Did a rogue regex in the app pin the CPU? All of these questions require developers and operators to build a lot of troubleshooting context. 

Cloud Monitoring data for GKE in Cloud Logging

To make it easier to troubleshoot GKE apps, we’ve added contextual Cloud Monitoring data accessible right from Cloud Logging. With this new feature, you can easily see the relevant pod, node and cluster events, metrics, alerts, and SLOs right from the log line itself. Additionally, the data loaded for a specific log entry is scoped to the Kubernetes resource, which saves you valuable time while investigating an app error.

Today’s announcement builds on other recent integrations including the addition of a logs tab nested in the details page of each of your GKE resources and combining metrics and logs in the GKE Dashboard in Monitoring. Now, wherever you start your troubleshooting journey – in Monitoring, Logging or GKE – you have the observability data at your fingertips. 

For example, if you’re troubleshooting a GKE app error in Cloud Logging and looking at the app logs, you can now view the metric charts for container restarts, uptime, memory, CPU and storage without leaving the log entry. Active alerts are highlighted on the alerts tab, which can provide helpful context for troubleshooting. This unique and integrated experience brings together critical log and metric data for the specific Kubernetes resource where your app is running.

Viewing Monitoring data for GKE from a log line

From a k8s_container, k8s_pod, k8s_node, or k8s_cluster log, select the blue chip with the resource.labels resource name and then select “View Monitoring details” to access an integrated metrics panel directly from the Logs Explorer. Selecting “View in GKE” opens the detailed view of the GKE resource in the Cloud Console on a new tab.

The metrics panel provides a lot of contextual data including alerts, Kubernetes events and metrics related to the GKE resource.

Alerts 

Alerts triggered by the GKE resource are displayed under the alerts tab. The color-coded alert status provides an easy way to see ongoing, acknowledged and closed incidents. Selecting “VIEW INCIDENT” opens the incident details in Cloud Monitoring. If you want to create a new alert, use the link to create a brand new alert policy.

Kubernetes events for clusters and pods

The metrics panel provides select events for clusters and pods.  For each event, the name, associated resource and a link to view/copy the log message are displayed. Kubernetes events can provide important information to help determine the root cause of an issue. For example, if a FailedScheduling event is displayed, this can quickly guide troubleshooting to check the resources available to the Kubernetes resource.

Metrics for containers, pods and nodes

The metrics tab contains metrics bundles for container (default), pod and node metrics collected from the GKE cluster and reported in Cloud Monitoring. Each metric bundle offers pre-built charts that can be selected to view the CPU, memory, storage and container restarts. For example, by looking at the CPU or memory, you can determine whether there were any spikes in the metrics for the Kubernetes resources.

More to come

We’re committed to making Google Cloud’s operations suite the best place to troubleshoot your GKE apps. We’ve integrated logs directly into GKE resource details pages and built a specialized integrated GKE Dashboard, all to make it easier to troubleshoot GKE apps. However, there is still more coming and we’re already working hard to add new features to the metrics panels to surface even more context for troubleshooting GKE apps.

Get started today

If you haven’t already, to get started with Cloud Logging and Cloud Monitoring on GKE, viewdocumentation, watch a quick video on troubleshooting services on GKE and join the discussion in our new Cloud Operations page on the Google Cloud Community site.

ML pipelines are great at automating end-to-end ML workflows, but what if you want to go one step further and automate the execution of your pipeline? In this post I’ll show you how to do exactly that. You’ll learn how to trigger your Vertex Pipelines runs in response to data added to a BigQuery table.

I’ll focus on automating pipeline executions rather than building pipelines from scratch. If you want to learn how to build ML pipelines, check out this codelab or this blog post.

What are ML pipelines? A quick refresher

If the term ML pipeline is throwing you for a loop, you’re not alone! Let’s first understand what that means and the tools we’ll be using to implement it. ML pipelines are part of the larger practice of MLOps, which is concerned with productionizing ML workflows in a reproducible, reliable way. 

When you’re building out an ML system and have established steps for gathering and preprocessing data, and model training, deployment, and evaluation, you might start by building out these steps as ad-hoc, disparate processes. You may want to share the workflow you’ve developed with another team and ensure they get the same results as you when they go through the steps. This will be tricky if your ML steps aren’t connected, and that’s where pipelines can help. With pipelines, you define your ML workflow as a series of steps or components. Each step in a pipeline is embodied by a container, and the output of each step will be fed as input to the next step. 

How do you build a pipeline? There are open source libraries that do a lot of this heavy lifting by providing tooling for expressing and connecting pipeline steps and converting them to containers. Here I’ll be using Vertex Pipelines, a serverless tool for building, monitoring, and running ML pipelines. The best part? It supports pipelines built with two popular open source frameworks: Kubeflow Pipelines (which I’ll use here) and Tensorflow Extended (TFX).

Compiling Vertex Pipelines with the Kubeflow Pipelines SDK

This post assumes you’ve already defined a pipeline that you’d like to automate. Let’s imagine you’ve done this using the Kubeflow Pipelines SDK. Once you’ve defined your pipeline, the next step is to compile it. This will generate a JSON file with your pipeline definition that you’ll use when running the pipeline.

With your pipeline compiled, you’re ready to run it. If you’re curious what a pipeline definition looks like, check out this tutorial.

Triggering a pipeline run from data added to BigQuery

In MLOps, it’s common to retrain your model when new data is available. Here, we’ll look specifically at how to trigger a pipeline run when more data is added to a table in BigQuery. This assumes your pipeline is using data from BigQuery to train a model, but you could use the same approach outlined below and replace BigQuery with a different data source. Here’s a diagram of what we’ll build:

To implement this, we’ll use a Cloud Function to check for new data, and if there is we’ll execute our pipeline. The first step here is to determine how many new rows of data should trigger model retraining. In this example we’ll use 1000 as the threshold, but you can customize this value based on your use case. Inside the Cloud Function, we’ll compare the number of rows in our BigQuery table to the amount of data last used to train our model. If it exceeds our threshold, we’ll kick off a new pipeline run to retrain our model.

There are a few types of Cloud Functions to choose from. For this we’ll use an HTTP function so that we can trigger it with Cloud Scheduler. The function will take two parameters: the name of the BigQuery dataset where you’re storing model training data, along with the table containing that data. The function then creates a table called count in that dataset, and uses it to keep a snapshot of the number of rows used last time you ran your retraining pipeline:

If the current number of rows in the table exceeds the latest value in the count table by your predetermined threshold, it’ll kick off a pipeline run and update count to the new number of rows with the Kubeflow Pipelines SDK method create_run_from_job_spec:

The resulting count table will show a running log of the size of your data table each time the function kicked off a pipeline run:

You can see the full function code in this gist, where the check_table_size function is the Cloud Functions entrypoint. Note that you’ll want to add error handling based on your use case to catch scenarios where the pipeline run fails.

When you deploy your function, include a requirements.txt file with both the kfp and google-cloud-bigquery libraries. You’ll also need your compiled pipeline JSON file. Once your function is deployed, it’s time to create a Cloud Scheduler job that will run this function on a recurring basis. 

You can do this right from the Cloud Scheduler section of your console. First, click Create job and give the job a name, frequency, and time zone. The frequency will largely depend on how often new data is added in your application. Setting this won’t necessarily run your pipeline with the frequency you specify, it’ll only be checking for new data in BigQuery[1]. 

In this example we’ll run this function weekly, on Mondays at 9:00am EST:

Next, set HTTP as the target type and add the URL of the Cloud Function you deployed. In the body, add the JSON with the two parameters this function takes: your BigQuery dataset and table name:

Then create a service account that has the Cloud Functions Invoker role. Under Auth header, select Add OIDC token and add the service account you just created:

With that, you can create the Scheduler job, sit back, and relax with the comforting thought that your retraining pipeline will run when enough new data becomes available. Want to see everything working? Go to the Cloud Scheduler in your console to see the last time your job ran:

You can also click the Run Now button on the right if you don’t want to wait for the next scheduled time. To test out the function directly, you can go to the Functions section of your console and test it right in the browser, passing in the two parameters the function expects:

Finally, you can see the pipeline running in Vertex AI. Here’s an example of what a completed pipeline run looks like:

What’s next?

In this post I showed you how to trigger your pipeline when new data is added to a BigQuery table. To learn more about the different products I covered here, check out these resources:

Vertex Pipelines intro codelab
Cloud Scheduler quickstart
Cloud Functions quickstart

Do you have comments on this post or ideas for more ML content you’d like to see? Let me know on Twitter at @SRobTweets.

[1] Note that if you’d like to run your pipeline on a recurring schedule you can use the create_schedule_from_job_spec method as described in the docs. This will create a Cloud Scheduler job that runs your pipeline at the specified frequency, rather than running it only in response to changes in your Cloud environment.

Thank you to Marc Cohen and Polong Lin for their feedback on this post.

Logs are an essential part of troubleshooting applications and services. However, ensuring your developers, DevOps, ITOps, and SRE teams have access to the logs they need, while accounting for operational tasks such as scaling up, access control, updates, and keeping your data compliant, can be challenging. To help you offload these operational tasks associated with running your own logging stack, we offer Cloud Logging. If you don’t need to worry about data residency, Cloud Logging will pick a region to store and process your logs. 

If you do have data governance and compliance requirements, we’re excited to share that Cloud Logging now offers even more flexibility and control by providing you a choice of which region to store and process your logging data. In addition to the information below, we recently published a whitepaper that details compliance best practices for logs data.

Choose from 23 regions to help keep your logs data compliant

Log entries from apps and services running on Google Cloud will automatically be received by Cloud Logging within the region where the resource is running. From there, logs will be stored in log buckets. Log buckets have many attributes in common with Cloud Storage buckets, including the ability to:

Set retention from 1 day to 10 years

Lock a log bucket to prevent anyone from deleting logs or reducing the retention period of the bucket

Choose a region for your log bucket. We recently introduced support for 23 regions to host your log buckets:

Europe – europe-central2, europe-north1, europe-west1, europe-west2, europe-west3, europe-west4, europe-west6

Americas – us-central1, us-east1, us-east4, us-west1, us-west2, us-west3, northamerica-northeast1, southamerica-east1

Asia Pacific – asia-east1, asia-east2, asia-northeast1, asia-northeast2, asia-northeast3, asia-south1, asia-southeast1, australia-southeast1    

How to create a log bucket

You can get started with regionalized log storage in less than five minutes.

Go to the Cloud Console and go to Logging

Navigate to Logs Storage and click on “Create logs bucket”

Name the log bucket and choose the desired region. Note that the region cannot be changed later. 

Set the retention period and then click Create Bucket.

Once you have created the bucket, you need to point the incoming logs to that bucket. To complete this:

Go to the Logs Router section of the Cloud Console and click on the dots to the right of the _Default sink. 

Select “Edit Sink”

Under Sink Destination, change the log bucket selected from “projects/…/_Default” to “projects/…/ (name of newly created bucket)”. 

Scroll to the bottom and select “Update sink” to save the changes

If you need more detailed information on this topic, please see our step by step getting started guide for overcoming common logs data compliance challenges. 

More about data residency in Cloud Logging

We have covered a lot of information about logs in this blog. For more on this topic and other best practices for compliance with logs data, please download this whitepaper. We hope this helps you focus on managing your apps rather than your operations. If you would like to pose a question or join the conversation about Google Cloud operations with other professionals, please visit our new community page. Happy Logging!

Introduction

The Oracle database is a popular RDBMS in the database industry,  but for a highly scalable environments we see customers choosing to leverage Cloud Spanner with its multi-region, unlimited scalability and up to 99.999% availability. Migration of Oracle to Spanner can be addressed using many approaches but one of the most important stages of the migration is Data Validation. One tool that helps simplify the data validation process is the Data Validation Tool (DVT).

Data Validation Tool is an open sourced python CLI based tool that compares heterogeneous data source tables and provides multi-level validation functions. DVT can connect to different types of source and target databases and create validation reports based on the criteria defined. In this blog, we will see how to set up the Data Validation Tool to compare data between Oracle and Spanner tables and create validation reports in BigQuery.

Installing Data Validation Tool

To install and run a data validation tool (DVT), we need a machine with python 3.6+, pip, gcc installed.  DVT natively supports BigQuery and Spanner connections with Google Cloud SDK installed. DVT has the capability to write the output of data validation into BigQuery tables directly (console output is default).  

BigQuery output tables need to be created beforehand for this and we can use terraform to create them easily. This will need a terraform cli to be installed on your machine. In addition to that, make sure git packages are installed to clone the repository.

Clone the repository onto your machine.

Inside the repository, navigate to the terraform folder. Perform the below steps to create the BigQuery tables necessary for data validation reports.

Delete the testenv.tf file inside the terraform folder

View variables.tf inside the terraform folder and replace default “pso-kokoro-resources” with default = “YOUR_PROJECT_ID”

Run the following commands from terraform directory

Alternatively, we can use the bq command line tool as a part of Google Cloud SDK to create the tables.

Before installing the dvt tool, create a virtual environment in python to sandbox the tool.

Now we can install the tool in the selected environment using pip.

To configure the Oracle connection, install cw_Oracle package using pip. 

This will also require an Oracle instant client library installed on the local machine.  Follow the steps mentioned here.

Creating Connections:

Before running the validation, connection to the Oracle system needs to be created. 

 Fill in the below details before executing the command

<oracle-host-name>

Oracle hostname

<oracle-port-number>

Oracle portname

<user-name>

Oracle login username

<password>

Oracle login password

<database-name>

Oracle database name

Next step, create a connection to the Spanner instance.

 Fill in the below details before executing the command:

<gcp-project-id>

GCP Project ID where Spanner instance is located

<spanner-instance-id>

Spanner Instance ID

<spanner-database-id>

Spanner Database ID

Use `gcloud auth application-default login` to use the default user to connect to GCP services. In case if you want to use a service account to create a connection, you can provide a service account json key file in the custom parameter `google_service_account_key_path`.

Validation Types:

DVT supports the following validation types:

Table level

Table row count

Group by row count

Column aggregation – SUM(), MIN(), MAX(), AVG()

Filters and limits

Column level

Full column data type

Based on the scenario, different validation types can be used. For this tutorial, we will be doing each column aggregation based validation using count, min and max functions. To visualize and customize the validation, provide the config-file option to write the validation configuration as a yaml file.

For example, we are considering validation of employee table which has 11 columns. We will be adding count based aggregation on all columns and min and max aggregation on the salary column.

Fill in the below details before executing the command

<gcp-project-id>

GCP Project ID where Spanner instance is located

Config.yaml is created with the validation criteria.

Running Validation:

After the configuration file is created, we can run the validation based on the confg.yaml.

This command will run the validation tool and store the results in the BigQuery table.

Analyzing Validation Report

The validation report shows aggregation values and differences for each column defined in the config file. Status column automatically fills the value based on the threshold provided (default threshold is 0.0). 

Further Information

Introducing the Data Validation Tool for EDW migrations

Migrating from Oracle to Cloud Spanner

Data Validation Tool is under active development. Please give them a try and help out by giving feedback, filing issues and sending PRs for fixes and enhancements. These open source tools are part of the user community and are not officially supported by Google Cloud. 

Since the start of the COVID-19 pandemic, there’s been a rapid acceleration of digital transformation across the entire healthcare industry. Telehealth has become a more mainstream and safe way for patients and caregivers to connect. Machine learning modeling has helped speed up innovation and drug discovery. And new levels of integration and data portability have helped enable greater vaccine availability and equitable access to those who need it.

Data has been at the crux of this digital transformation — helping people stay healthy, accelerating life sciences research and delivering more personalized and equitable care. We recently unveiled partial results from our research with The Harris Poll, which revealed that nearly all physicians (95%) believe increased data interoperability will ultimately help improve patient outcomes. Today, we’re unveiling the second part of that research. 

In February 2020, we commissioned The Harris Poll to survey 300 physicians in the U.S. about their biggest pain points — this was just before the COVID-19 pandemic strained the entire healthcare system and made us all hyper-aware of the risks we take in going to the hospital. In June 2021, we followed-up with those same questions and more. What it unveiled was just how much COVID-19 reshaped technology’s role in the healthcare field and how it’s changing day-to-day operations for physicians. 

Here are some of the highlights: 

Healthcare organizations accelerated technological upgrades over the course of the pandemic. After a year shaped primarily by the COVID-19 pandemic, use of telehealth saw substantial YOY growth, jumping nearly threefold from 32% in February 2020 to 90% this year. Forty-five percent of physicians say the COVID-19 pandemic accelerated the pace of their organization’s adoption of technology. In fact, more than 3 in 5 physicians (62%) say the pandemic has forced their healthcare organization to make technology upgrades that normally would have taken years. For example, 48% of physicians would like to have access to telehealth capabilities in the next five years. Before the COVID-19 pandemic, about half of physicians (53%) say their healthcare organization’s approach to the adoption of technology would best be described as “neutral” (i.e., willing to try new technologies only if they have been in the market for awhile or others have tried and recommended them). 

Despite the technological leaps this year, most physicians still believe the industry lags behind in technology adoption but recognize the opportunity for technological support and advancement. The majority of physicians don’t view the healthcare industry as a leader when it comes to digital adoption. More than half of physicians describe the healthcare industry as lagging behind the gaming (64%), telecommunications (56%), and financial services industries (53%). However, the healthcare industry is not seen to be trailing as much as it was last year behind retail (54% in 2020; 44% in 2021); hospitality and travel (53% in 2020; 43% in 2021); and the public sector (39% in 2020; 26% in 2021). 

Better interoperability alleviates physician burnout, improves health outcomes and speeds up diagnoses. The majority of physicians say increased data interoperability will cut the time to diagnosis for patients significantly (86%) and will ultimately help improve patient outcomes (95%.) In addition to better patient experiences and outcomes, more than half of physicians (54%) believe increased access to data via technology has had a positive impact on their healthcare organization overall. A majority believe that technology can alleviate the likelihood of physician “burn-out” (57%) and that efficient tools help decrease friction and stress (84%). And, as a result, 6 in 10 physicians say access to better technology and clinical data systems would allow them to have better work/life balance (60%) and that better access to/more complete patient data would reduce administrative burdens (61%). It is therefore not surprising that nearly 9 in 10 physicians (89%) say they are increasingly looking for ways to bring together all patient data into a single place for a more complete view of health. 

Familiarity with new Department of Health and Human Services (DHHS) interoperability rules grows, and many physicians are in favor. Most physicians (74%) say they have at least heard of the new DHHS rules (launched in 2019) to improve the interoperability of electronic health information. This is a clear rise from 2020 (64%), but deeper knowledge is fairly low. Only 30% of physicians say they are somewhat or very familiar with the new rules (though, again, this is a rise from 2020, when only 18% said they were very/somewhat familiar). Similar to in 2020, among those who have heard of the new rules, nearly half are in favor (48% in 2021; 45% in 2020) but a similar proportion remain unsure (46% in 2021; 50% in 2020). And like in 2020, by far the top potential benefit of the rules is thought to be forcing EHRs to be more interoperable with other systems (70%).

Google was founded on the idea that bringing more information to more people improves lives on a vast scale. In healthcare, that means creating tools and solutions that make data available in real time to help streamline operations and improve quality of care and patient outcomes. For example, our recently announced Healthcare Data Engine makes it easier for healthcare and life sciences leaders to make smart real-time decisions through clinical, operational, & groundbreaking scientific insights. To find out more about the Healthcare Data Engine, click here.

Survey methodology: The 2021 survey was conducted online within the United States by The Harris Poll on behalf of Google Cloud from June 9 – 29, 2021 among 303 physicians who specialize in Family Practice, General Practice, or Internal Medicine, who treat patients, and are duly licensed in the state they practice. The 2020 survey was conducted from February 18 – 25, 2020 among 300 physicians who specialize in Family Practice, General Practice, or Internal Medicine, who treat patients, and are duly licensed in the state they practice. Physicians practicing in Vermont were excluded from the research. This online survey is not based on a probability sample and therefore no estimate of theoretical sampling error can be calculated. For complete survey methodology, including weighting variables and subgroup sample sizes, please contact press@google.com.

When it comes to learning how to implement some technology, we all have our own version of what I call the “tab game”—that is, your setup for all the tabs and windows you need open at once. You may have several monitors so you can see documentation, your IDE, and terminal windows at the same time. You may have several guides and references open at once in one window to get all the information you need.

Personally, I like to work just from my laptop because I like to move around and work from various comfy spots. I think my tab game would probably enrage most devs because it involves a lot of swiping back and forth between windows *and* toggling tabs. It’s not pretty. That is, it wasn’t pretty until I discovered tutorials in the Google Cloud Console!

Jen really didn’t know about tutorials in the Google Cloud Console?

Yes, I honestly didn’t know about them! I’m sharing about it because if I can work for Google and not know, then I can’t be the only one, and it would be a shame to miss out on this because it’s a brilliant idea. Also I wrote some pretty sweet tutorials for the console, but I swear that the main reason I’m telling you is because it’s a cool thing!

There are several reasons that these tutorials are great:

You can view the instructions and the console at the same time. No more playing the tab game!
The tutorials include links and highlights, making it easy to find the screens and buttons you’re looking for
You can run code from Cloud Shell, so you don’t need a separate window for an IDE
You can use the demo data provided to try things out, or you can apply the steps to your existing projects using data that suits your app’s needs

Firestore tutorials

I’m developing a series of tutorials in the Google Cloud Console designed to take you through everything you need to know about Firestore–from manually adding data in the Google Cloud Console to triggering Cloud Functions to make changes for you. Below are links and summaries for the currently available tutorials. Check back regularly to find the latest additions as they’re released!

Add Data to Firestore

Enable Firestore on a project
Learn about the Firestore data model
Add a collection of documents
Add fields to a document
Delete documents and collections

Updating Data in Firestore using Node.js or using Python

Add a collection of documents
Explore available data types
Replace the data of document
Replace fields in a document
Handle special cases: incrementing, timestamps, and arrays

Reading Data from Firestore using Node.js or using Python

Add a collection of documents
Explore available data types
Read a collection
Read a single document
Order documents
Query documents

Transactions in Firestore using Node.js

Add a collection of documents
Update data without a transaction to observe issue
Complete a transaction
Complete a batched write

Batched Writes in Firestore using Node.js or using Python

Use Cloud Shell and Cloud Shell Editor to write a Node.js or Python app
Complete a batched write

Firestore triggers for Cloud Functions

Initialize Cloud Functions using the Firebase CLI
Write a Cloud Function triggered by a new document write to Firestore

Offline Data in Firestore

Add data to Firestore in the Cloud console Firestore dashboard
Create a web app that uses Firestore using the Firebase SDK
Deploy Firestore security rules that enable access to the required data
Enable data persistence in the web app
Observe app behavior with and without network connection

Chime in

Is there a particular action or concept in Firestore that you’d like to see a tutorial for? Is there another Google Cloud product that you want to learn more about? Tweet @ThatJenPerson and you may just see your suggestion come to life in the Google Cloud Console!

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com”organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?

Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

API activity (e.g. service accounts with authentication activity, API calls consumed)

Networking activity (ingress and egress)

Billing activity (e.g. services with billable usage)

User activity (e.g. active project owners)

Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.

As the Olympics kicked off in Tokyo at the end of July, we found ourselves reflecting on the beauty of diverse countries and cultures coming together to celebrate greatness and sportsmanship. For this month’s blog, we’d like to highlight some key data and analytics performances that should help inspire you to reach new heights in your data journey.

Let’s review the highlights!

BigQuery ML Anomaly Detection: A perfect 10 for augmented analytics

Identifying anomalous behavior at scale is a critical component of any analytics strategy. Whether you want to work with a single frame of data or a time series progression, BigQuery ML allows you to bring the power of machine learning to your data warehouse. 

In this blog released at the beginning of last month, our team walked through both non-time series and time-series approaches to anomaly detection in BigQuery ML:

Non-time series anomaly detection

Autoencoder model (now in Public Preview)

K-means model (already in GA)

Time-series anomaly detection

ARIMA_PLUS time series model (already in GA)

These approaches make it easy for your team to quickly experiment with data stored in BigQuery to identify what works best for your particular anomaly detection needs. Once a model has been identified as the right fit, you can easily port that model into the Vertex AI platform for real-time analysis or schedule it in BigQuery for continued batch processing.

App Analytics: Winning the team event

Google provides a broad ecosystem of technologies and services aimed at solving modern day challenges. Some of the best solutions come when those technologies are combined with our data analytics offerings to surface additional insights and provide new opportunities. 

Firebase has deep adoption in the app development community and provides the technology backbone for many organization’s app strategy. This month we launched a design pattern that shows Firebase customers how to use Crashlytics data, CRM, issue tracking, and support data in BigQuery and Looker to identify opportunities to improve app quality and enhance customer experiences.

Crux on BigQuery: Taking gold in the all-around data competition

Crux Informatics provides data services to many large companies to help their customers make smarter business decisions. While they were already operating on a modern stack and not on the hunt for a modern data warehouse, BigQuery became an enticing option due to performance and a more optimal pricing model. Crux also found advantages with lower-cost ingestion and processing engines like Dataflow that allow for streaming analytics.

Technology is a team sport, and Crux found our support team responsive and ready to help. This decision to more deeply adopt Google Cloud’s data analytics offerings provides Crux with the flexibility to manage a constantly evolving data ecosystem and stay competitive.

You can read more about Crux’s decision to adopt BigQuery in this blog.

Google Trends: A classic emerges a champion

Following up on the launch of our Google Trends dataset in June, we delivered some examples of how to use that data to augment your decision making. 

As a quick recap of that dataset, Google Cloud, and in particular BigQuery, provide access to the top 25 trending terms by Nielsen’s Designated Market Area® (DMA) with a weekly granularity. These trending terms are based on search patterns and have historically only been available on the Google Trends website.

The Google Trends design pattern addresses some common business needs, such as identifying what’s trending geographically near your stores and how to match trending terms to products to identify potential campaigns. 

Dataflow GPU: More power than ever for those streaming sprints

Dataflow is our fully-managed data processing platform that supports both batch and streaming workloads. The ability of Dataflow to scale and easily manage unbounded data has made it the streaming solution of choice for large workloads with high-speed needs in Google Cloud. 

But what if we could take that speed and provide even more processing power for advanced use cases? Our team, in partnership with NVIDIA, did just that by adding GPU support to Dataflow. This allows our customers to easily accelerate compute-intensive processing like image analysis and predictive forecasting with amazing increases in efficiency and speed. 

Take a look at the times below:

Data Fusion: A play-by-play for data integration’s winning performance

Data Fusion provides Google Cloud customers with a single place to perform all kinds of data integration activities. Whether it’s ETL, ELT, or simply integrating with a cloud application, Data Fusion provides a clean UI and streamlined experience with deep integrations to other Google Cloud data systems. Check out our team’s review of this tool and the capabilities it can bring to your organization.

As hopefully you all know by now, migrating your SQL database got easier this past November when Google’s Database Migration Service went GA for MySQL. If you didn’t know, migrating your SQL database to Cloud SQL is now easier! Something you might have noticed, or I’ll happily point out if you haven’t noticed, is that Database Migration Service lets you pull from a number of different sources, including on premises, AWS, and even Cloud SQL for MySQL.

Side plug – we just announced DMS supports Cloud SQL for PostgreSQL as a source, so you can use DMS for your PostgreSQL major version upgrades too. But THIS post is about MySQL…

Because something ELSE I wanted to call out… notice this, it appears on the “Create a destination” page of the DMS process:

So if you’re following along here, I’m telling you that you can ALSO use DMS to do major version upgrades on your MySQL database. We’ll all still wait with baited breath for in-place upgrades of course, but while we’re waiting this is a great option.

There’s a catch, because of course there is. The key here is that because we’re using a migration tool to upgrade, there’s two dimensions of complications to deal with. We have the migration piece and the upgrade piece. DMS helps manage a good amount of this, but there’s still some things you need to consider. In this post, I’m going to pull together all the pieces you need to think about and link to everything you need to do a major version upgrade for MySQL using Database Migration Service.

Why are we here?

Just to cover why you might want to upgrade MySQL in the first place. Between performance upgrades and feature updates, there’s plenty of reasons to do it. If you need convincing, here’s a nice list of the enhancements in MySQL 8.0. There’s also the elephant in the room: officially 5.6 was deprecated this past February. You might have been on 5.6 for the last 8 years, saw it’s officially end-of-lifed this year and are panicking a little bit. The good news here is that Cloud SQL will support 5.6 for a while longer, but that doesn’t mean now isn’t a good time to upgrade.

Version compatibility

The first thing to look at is what versions you’re upgrading between. So 5.6 to 5.7 or 5.7 to 8.0. 5.6 to 8.0 is right out. MySQL has significant changes between major versions that are likely to break compatibility, so you need to triple check your database for some of these incompatible changes.

For example, between 5.6 and 5.7 you need to keep an eye out for any system or status variables in the INFORMATION_SCHEMA tables in your 5.6 database. Those were all replaced by the Performance Schema tables in 5.7.6. There are also a lot of little things–like if you have a column with the YEAR(2) data type, you’ll need to update all those values to a 4-digit YEAR column before you’ll be able to use those columns again. If you’re going 5.6 to 5.7, you can go over the full list of changes here.

Between 5.7 and 8.0 of course there are yet more changes to watch out for. Big ones for me are that default flags were changed quite a bit between the two. While a lot of them might not break things with a segmentation fault, they could lead to some unintended behavior in your application. Also, you might want to take a peek at your AUTO_INCREMENT columns. It’s been deprecated for FLOAT and DOUBLE types. For a full list of what changed between these two versions, check out here.

Connecting the dots

If you’re planning on doing the major version upgrade from one Cloud SQL instance to another, you can skip this section because you’ve already done what I’m going to talk about. If, however, you’re changing from an on premises database, or in some small edge cases, a GCE (virtual machine) instance to Cloud SQL, there’s a few extra things to watch out for.

#1 thing is latency. If you’ve gotten comfortable having a near-instantaneous response from your database because it lived right next to your application, be prepared for a bit of a wakeup call. Unless your application also lives in the Cloud, there’s now some extra time that will pass between your application and your database. This might be totally fine! Or, it could introduce some really difficult-to-debug re-entrancy bugs into your application. A colleague of mine wrote a great post that covers all the things you need to watch out and potentially change for when moving your database to Cloud SQL from another source here.

#2 is how do I even now get to my database?! Network connectivity can be really difficult. Firewalls, routing and DNS can all throw a wrench in the works. You had everything all nicely connected before you did this, and now all of a sudden your database lives (potentially) in an entirely different part of the internet with different requirements to connect to it. This particular rabbit hole I’ve gone very deep on already. I wrote an in-depth blog post on connecting source to destination instances with Database Migration Service here. If you run into problems with connectivity, that post should cover most situations.

More migrating!

DMS makes the moving your data piece of the process easy, but since we rely on the database engine’s native migration mechanisms, we’re also subject to its limitations. This means there’s some steps to complete once you’re done migrating your database regardless of where it came from. For example, you know, your IP address will change. Because we’re using a migration method to upgrade our version, we’re instantiating a new instance. Once we get in-place upgrades this is a non-issue, but in the meantime we have to remember to update any applications, load-balancers, etc that are pointing at the old IP address.

Another thing is, don’t forget, your users are stored in system tables that aren’t migrated either. Those will need to be brought over once the DMS portion is complete. There’s a few more bits and bobs, and there’s a complete post here that lists the things you’ll need to finish off before your new instance is ready for use.

What’s next?

Now that you’ve got your newly upgraded instance all set, it’s time to get it ready for production. There are a few things that you’ll likely want that can’t be set up until the Cloud SQL instance is fully online.

By default, for example, SSL-only connections are off by default and we know many organizations enforce SSL-only connections. This will need to be enabled on the server and certificates will need to be set up. This can be done after the data is migrated but before the DMS target instance is promoted.

Disaster recovery is another must-have for production instances. It’s also a requirement for our SLA which guarantees uptime. Absolutely need to set this up. If you edit your instance it’ll be on the main Overview page:

Setting to “Multiple zones” enables this feature.

Also falling into this category are read replicas. Optimizing your data infrastructure to support your application is key and this is an integral part of that. If you have a read-heavy application, you’ll want to set up your read replicas at this stage of the game as well.

Setting up high availability and the read replicas can only be done after you’ve promoted the instance at the end of the DMS process.

One last bit before I leave you. You’ve done all this work to upgrade your instance to the next version. You’ve set up all the extra pieces you need. You’ve connected your application and it can totally talk to your database. Now is NOT the time to hit the push to prod button. This may absolutely come across as obvious to many of you reading this. But I’m going to say it anyway. TEST! Test all the things. Test all the edge cases you can think of. Test adding even more latency into your system arbitrarily to be sure that you can handle random internet lag spikes between application and database. The high availability feature once it’s on allows you to test automatic failover to simulate your primary instance going down and the fallback instance taking over. So test. Test all these scenarios to ensure that you have a smooth launch of your new major version SQL instance.

Hopefully this all helped to guide you to successfully using Database Migration Service to upgrade your MySQL instance to a new version! If you have questions or comments, please feel free to reach out to me on Twitter, my DMs are open! Thanks for reading.

Mainframe modernization has been a hot topic over the past decade or so. Many people have predicted the demise of mainframe times. But the mainframe is standing tall, strong and growing. Over time, the term “modernization” itself is manifested in many ways. So to even begin the modernization conversation, we need to define what modernization is. 

As Henry Ford once said, “If I had asked people what they wanted, they would have said faster horses.” Instead, Ford invented an automobile for the masses to improve their lives. So modernization is akin to evolution, which in hindsight reflects the progress humanity made in lifestyle, resilience, knowledge, communication, and, in general, wisdom.

Similarly, modernization in mobile phones meant lighter weight, wider screen, faster speed, and more storage. And then the iPhone was released in 2007, which showed innovation and changed the way humans interact with digital devices. 

Deciding on mainframe modernization

The decision to embark on a mainframe modernization journey typically lies with three stakeholders: infrastructure owners, application owners, and business owners. For a long time, mainframe modernization has focused on solving infrastructure and application problems like higher cost, lack of skillset availability, locked-in data, and poor integration with modern tools and products.In other words, mainframe modernization has been only focused on the infrastructure and application layer. 

In this paper, we try to understand the importance of ‘business owner’, who is the most critical stakeholder and sponsor.

Journey depends on disposition for data and applications

For a long time, the mainframe modernization journey has been defined as just moving the workload off the mainframe, with little or no focus on business amplification. As a result, there has not been much success in either commencing projects like these or in completing them. 

But mainframe modernization should be looked at from a business imperative lens. You need to define a goal before embarking on the transformation path. This will help identify the right partner and trusted advisor, optimize the overall effort, and minimize the number of hops to get to that goal.

Google’s approach to mainframe modernization revolves around bringing business needs to the forefront of modernization initiatives. Let’s look at these one by one:

One Google: As part of the larger Google ecosystem, it’s quite easy and seamless for Google Cloud services to integrate with products like Maps, Search, and Ads and hence be able to provide a very personalized and enhanced user experience. Most enterprise customers are existing subscribers to one or more of these services, so it’s also easy for them to integrate Google Cloud services.

Data-first modernization approach: Google offers a data-first approach to mainframe modernization that involves moving ‘system of records’ or ‘system of engagement’ from a mainframe to Google Cloud datastores. This approach enables you to leverage and unlock the data in many ways:

Enabling business users, aka citizen developers, to build apps leveraging data and tools like AppSheet.

Reducing mainframe consumption by routing web and mobile apps to data on Google Cloud.

Including the data that was moved from the mainframe to Google Cloud in advanced analytics to understand customer behavior and come up with market-relevant products.

AI/ML on Google Cloud: When it comes to extracting info and wisdom from raw data using artificial intelligence and machine learning, there’s no parallel to Google. And thereby our customers get a significant competitive advantage in understanding market trends and behavioral patterns. This enables them to meet and exceed customer requirements while providing a very personalized experience.  

Industry-specific solutions: Google has been investing heavily in business-specific industry solutions. With innovation deep rooted in its DNA and culture, Google has always been looking at things, starting from search, outside the box. One example is AlphaFold, which is an artificial intelligence program created by Google’s DeepMind that performs predictions of protein structure. Another example is Verily, whose mission is to make the world’s health data useful so that people enjoy longer and healthier lives. Similarly, Waymo is focused on fully autonomous driving, Wing is focused on drone delivery, and Route4Me provides easy-to-use driving route optimization for businesses. In financial services, there are Google Pay, Lending DocAI, Procurement DocAI, Real-time credit card fraud detection, and so on.

All these business amplifications and enhancements come with state-of-the-art cloud infrastructure:

Hardware security: Google OpenTitan is the first open source project that builds a transparent, high-quality reference design and integration guidelines for silicon root of trust (RoT) chips.

Data security: The user owns the data. They set the rules and decide who needs to access what. Google Cloud encrypts data not only at rest and in motion but also during processing.

Latency: With an in-house network of optical fibers and undersea cables, there’s significantly less latency compared to others. Google search runs on the same platform.

Scalability: Alphabet has 9 products—Android, Chrome, Gmail, Drive, Maps, Search, the Play Store, YouTube and Photos—with more than a billion users each. These run on Google Cloud providing an uninterrupted experience to all users. 

100% carbon neutral footprint: Helps customers fulfill their obligations towards carbon neutrality and help save the planet earth.

Many of the large deals like Sabre, Deutsche Bank, Albertsons Companies and Telus are testament of Google’s focus on amplifying and redefining business with our customers. With the advent of Google Cloud’s focus on mainframe modernization, business stakeholders are finding value in business transformation. Instead of looking at the modernization from an ‘IT lens’ and decommissioning the mainframe, they found a partner in Google who strives to not only solve the current problems but also define a ‘North Star’ for the enterprise and collaborate to achieve those goals. 

Conclusion

Wrapping the thought with automobiles—Tesla showed the possibility that an automobile can be software driven, independent of fossil fuel, and contribute to reducing carbon footprint at the same time. Having said that, this has only been possible with the revolutionary or evolutionary journey started by Mr. Ford.