Google Cloud

Editor’s note: This post is part of an ongoing series on IT predictions from Google Cloud experts. Check out the full list of our predictions on how IT will change in the coming years.

Prediction: By 2025, over half of cloud infrastructure decisions will be automated by AI and ML

Google’s infrastructure is designed with scale-out capabilities to support billions of people, powering services like Search, YouTube, and Gmail every day. To do that, we’ve had to pioneer global-scale computing and storage systems and shorten network latency and distance limitations with new innovations. Along the way, we’ve come to see cloud infrastructure as more than a simple commodity — it’s a source of inspiration and new capabilities.

But even as the demand on the industry’s cloud infrastructure continues to increase, there are simultaneously plateaus in the efficiency available from the underlying hardware. In the past, we saw annual performance gains of 30-40%, levels that often enabled a single infrastructure configuration to meet the needs of the vast majority of workloads. As these improvements have slowed and new workloads such as AI/ML and analytics have emerged, we have seen a corresponding explosion in the variety and capability of infrastructure. While empowering, the burden of picking the right combination of infrastructure components for a given workload still falls on an organization’s cloud architects. 

But by 2025, we predict that the burden and complexity of infrastructure decision making will disappear through the power of AI and ML automation, which will automatically combine purpose-built infrastructure, prescriptive architectures, and an ecosystem to deliver a workload-optimized, ultra-reliable infrastructure. The focus for cloud architects will therefore be on enabling business logic and innovation, rather than how that logic maps to underlying infrastructure.

Already, we are making investments to turn this vision into reality, building custom silicon like the Infrastructure Processing Unit (IPU) for our new C3 VMs or a liquid-cooled board for the new tensor processing unit. The latter, the TPU v4 platform, is likely the world’s fastest, largest, and most efficient machine learning supercomputer. It can train large-scale workloads up to 80% faster and 50% cheaper than alternatives. Put another way, TPU v4 will nearly double the performance of critical ML and AI services at half the cost, unlocking new possibilities for what organizations can achieve when leveraging large-scale learning and inference for business services.

These same IPUs and TPUs represent the foundation that will make it possible to automate cloud infrastructure decisions. They’ll be able to support the telemetry data and ML-based analytics for proactive infrastructure recommendations that will increase the performance and reliability of workloads. 

Instead of determining hardware specifications and building the right infrastructure, you’ll only need to specify a workload. AI and ML will take over the burden and recommend, configure, and identify the best options based on your budgetary, performance, and scaling requirements. What is most exciting for us is how this will enable a much more rapid pace of service innovation, which is the primary end goal of great cloud infrastructure.

Global life science supply chains are lengthy and complex with many moving parts. One small disruption can create serious delays and affect your ability to deliver therapeutics for patients. 

Supply chain disruptors

Over the last few years, healthcare organizations have encountered a range of obstacles, from both internal and external factors, that have resulted in supply networks failing to get drugs and medical devices to where they need to be on time. These obstacles include:

Labor and supply shortages 

Rising material costs 

Raw material constraints

Geo-political events

Unpredictable weather 

How do you overcome supply chain disruptors that are out of your control?

The intelligent healthcare supply chain

While many organizations have already implemented data-driven supply chains, organizations are still faced with the challenges of static, siloed, and different functional supply chain applications; limited data exchange with key trading partners across upstream and downstream operations; and the inability to effectively leverage relevant external data. 

At Google Cloud, we believe the key to meaningful and effective change is a data-driven supply chain that allows you to achieve visibility, flexibility, and innovation.

Our solutions help you prepare for the unpredictable and enhance the value of your data. By unlocking AI-driven insights, you can strengthen distribution networks and optimize your workflows and supply chains to become more reliable, intelligent, and sustainable. Some of the business challenges we address include:

Predicting demand with Vertex AI Forecast

Visual inspection for quality and predictive maintenance with pre-built ML models

Automating and optimizing pickup and delivery operations with Cloud Fleet Routing API

Real time and holistic inventory visibility with Supply Chain Twin

Make sure you’re prepared for the unpredictable with real-time visibility over your distribution networks. Learn how you can harness the power of AI and analytics and gain actionable insights that enhance your supply chain.

Editor’s Note: A native of Recife, Brazil, Patricia Florissi is a Technical Director in OCTO, Google Cloud’s Office of the Chief Technology Officer. OCTO is a global team of senior technologists with experience at large corporations, institutions, and successful startups, who work alongside customers to identify solutions for tough, but transformational challenges. 

How has your personal story led you to what you are doing today?

It’s not immediately evident how my personal story led me here, but I am very passionate about what I do at OCTO. My specialty is large computing systems, in particular, federated computations, a superset of federated learning and federated analytics. For me, federated computations are about orchestrating computing across a set of locations, computing on data in place, while preserving the privacy of each participating entity, and sharing only aggregated results with the orchestrator

How does that relate to growing up in Recife in the 1970’s? 

I believe that life creates the paths you follow. I also believe that good technology always reflects human experience. My parents were older when they had me, so they were worried they wouldn’t always be around to take care of me. Because of that, I was raised to be independent, mindful, curious and resourceful. I’m driven by my passion for family and learning, and am proud of the role I play in uniting generations. 

Nothing about my education was simple. The university I attended in Recife didn’t have a lot of resources, and only a few of us were able to study computer programming. In fact, I had to share a computing textbook with two other students—one of whom became my husband. We learned to program on one of the country’s only personal computers.

Maybe my passion for federation comes from a passion for combining the power of the individual with societal benefit. When people come together, they can achieve bigger, more complex, and more ambitious goals. I’ve seen how the contributions of few, can create something that couldn’t have been achieved otherwise.  

Why did you ultimately decide to emigrate to the U.S.?

It was kind of an accident actually. Growing up, everybody in Brazil who wanted to do graduate work in computer science went abroad—so when funding opened up, my husband and I enrolled at Columbia University. After graduating, we went back to Brazil where our twin daughters were born. But a year later, my husband got sick and sought treatment in the States. At that point, I started working for computer companies in the U.S. and joined Google in 2020.

What drew you to Google Cloud?

I love working at Google because the work OCTO does is very complex—complex technology, complex implications for businesses, and complex to execute. We have a great culture for challenging ourselves, learning, and creating by thinking outside the box. As soon as we understand something, we use that as the basis and starting point for learning something else. At OCTO, there is a healthy combination of curiosity and humility.

How is that reflected in your work with customers?

At OCTO, we work with customers to identify their problems and help tackle them. We don’t show up with the answers; we ask questions to understand how they operate, their vision, their challenges and the technology trends that matter most to them. This helps us identify solutions. There’s a lot of joy in the “a-ha!” moment, but it also takes a lot of time and commitment to get there. There are many ups and downs, and it demands a tremendous amount of resilience. But the work is extremely impactful.

How does your specialty—federated computations—impact the world?

Federated computation is a way to process data that is scattered all over the world, without moving that data to a central location. The data stays in place and is computed locally. 

A few examples come to mind, such as:

Big retailers who have data in computers across their stores generating images of their shelf that could be collectively used for training ML models  to predict “out of the shelf” events before they occur. Federated computations enable the collective analysis without moving data from the store, enhancing privacy while utilizing the compute already dispersed in the stores.

Healthcare providers who collect patient data that could be collectively analyzed but need to be protected because it’s personal information. By analyzing the data in place, federated computations enhance data protection; and 

Vehicles, which have become like mobile devices with wheels, collect telemetry data that could be collectively analyzed and used for training predictive maintenance models. Federated computations allow the analysis of the data in-vehicle, enhancing data privacy.

Federated computation is also a positive trend in regards to technology and societal benefit. The world may be headed to increased computing capacity and more data available for analysis. At the same time, the growing concerns with data privacy and data residency, among others, have been leading towards increasing data isolation. It’s an opposition that I’d love to unite and Google gives me the opportunity to work with companies to figure out how to address this. Every day, I wake up and think, “What haven’t I tried yet?,” then I get started.

We’re pleased to announce that Confidential Space, our new solution that allows you to control access to your sensitive data and securely collaborate in ways not previously possible, is now available in public Preview. First announced at Google Cloud Next, Confidential Space can offer many benefits to securely manage data from financial institutions, healthcare and pharmaceutical companies, and Web3 assets. Today, we will explore some security properties of the Confidential Space system that makes these solutions possible. 

Confidential Space uses a trusted execution environment (TEE), which allows data contributors to have control over how their data is used and which workloads are authorized to act on the data. An attestation process and hardened operating system image helps to protect the workload and the data that the workload processes from an untrusted operator.

The Confidential Space system has three core components:

The workload is a containerized image with a hardened OS that runs in a cloud-based TEE. You can use Confidential Computing as the TEE that offers hardware isolation and remote attestation capabilities.

The attestation service, which is an OpenID Connect (OIDC) token provider. This service verifies the attestation quotes for TEE and releases authentication tokens. The tokens contain identification attributes for the workload.

A managed cloud protected resource, such as a Cloud Key Management Service key or Cloud Storage bucket. The resource is protected by an allow policy that grants access to authorized federated identity tokens. 

The system can help ensure that access to protected resources is granted only to authorized workloads. Confidential Space also can help protect the workload from inspection and tampering, before and after attestation.

In our published Confidential Space Security Overview research paper, we explore several potential attack vectors against a Confidential Space system and how it can mitigate those threats. Notably, the research notes how Confidential Space can protect against malicious workload operators and administrators, and malicious outside adversaries, who are attempting to create rogue workload attestations.

Through these protections, Confidential Space establishes confidence that only the agreed upon workloads will be able to access sensitive data. The research also highlights some of the extensive security reviews and tests executed to identify potential weak points in the system, including domain expert reviews, meticulous security audits, and functional and fuzz testing.

We asked the NCC Group for an independent security assessment of Confidential Space to analyze its architecture and implementation. NCC Group leveraged their experience reviewing other Google Cloud products to dig deep into Confidential Space. 

The NCC Group’s  extensive review, which included penetration testing and automated security scanning, found zero security vulnerabilities. In their report, the architecture review highlights how the security properties are achieved through the coordination of measured boot with vTPM attestation, reduced attack surface with constricted administrator controls and access, workload measurement and enforced launch policy, and resource protection policy based on attested workload runtime properties.

The combination of these attributes creates powerful security properties, gating release of data on runtime measurements of the actual workload code and environment instead of just user and service account credentials. Confidential Space provides a platform that includes:

A dependable workload attestation, including workload code measurement, arguments and environment, and operating environment claims

A fully-managed attestation verification service that validates expected environmental attestation claims

A policy engine allowing for arbitrarily complex (or extremely simple) policy to be created around those claims

A mechanism to attach those policies to Google Cloud resources

Together, the platform provides a mechanism where one can ensure that their data is only ever released into trusted workloads that will not abuse that data.

Take a look at our documentation and codelab and take it for a spin. We hope that Confidential Space can inspire organizations to solve their use cases around multi-party collaboration with sensitive data; please contact your Google Cloud sales representative if you have any questions.

Nowadays, many companies have to ask themselves: Do we want to wait until a startup disrupts our business model or will we just do it on our own? Klöckner, a German steel and metal distributor, chose the second option, leveraging the power of Google Cloud to  bring a more customer-focused, agile approach to an industry often mired in costly, antiquated, and time-consuming processes.

In this blog post, you will learn how Klöckner used Google Cloud Vertex AI services such as AutoML and Vertex AI Pipelines to improve internal processes through machine learning (ML), including increasing velocity in ML model building, reducing time to production, and providing solutions for production-level ML challenges such as versioning, lineage, and reproducibility. To put it in the words of the customer: “Vertex provided solutions for problems we were not aware of yet,” said Matthias Berkenkamp, the engineer who spearheaded Klöckner’s use of the AI service.  

Klöckner implemented a webshop, offering a solution for their customers to purchase products through a digital channel. Although the shop was very well perceived, Klöckner discovered that a significant portion of their customers still ordered through phone calls, emails and submitted PDFs. The order then required manual processing to get the data into their order systems—taking up valuable time and creating both inefficiencies and frequent errors. Klöckner recognized that ML could help improve and automate the purchase entry process.

From the first ML model to the complexity of ML in production

Klöckner’s data science team came up with different ideas and developed models for various use cases, such as automated extraction of text from PDF files that were attached to purchase order emails. However, they didn’t have experience on how to implement the model into their first ML application, which was called IEPO (Information Extraction from Purchase Orders). Moreover, Klöckner experienced communication challenges between data scientists and the DevOps team. There was neither a software build pipeline nor documentation about the model training, which was an absolute no-go for a team living the GitOps approach: if it’s not in Git, it cannot go to production. After many delays, the model was finally deployed but couldn’t handle heavy loads such as orders with 20+ PDFs. No one tested these edge cases before as there were so many other problems to solve first. This neglected part between creating a model and finally running it in production was an underestimated beast.

How Vertex AI provided the right tools

AutoML was the eye-opening entry point

Klöckner identified a new use case to be tested: a model that matches mail content (e.g. product numbers, descriptions or specific parameters such as “80/60/4mm, Alu-Rechteckrohr”) to specific internal products. They adopted Vertex AI Auto ML for this task. Because AutoML enabled faster model development and easier collaboration, it simple for everyone to start: no internal processes to get a budget (the initial $300 credit is often enough to train first models), staffing of data scientists or anything like that was not needed for a first try, and considerably less data was already sufficient to test it out. Surprisingly, for the little effort it took, the results were decent and accessible for everyone. This experience turned the switch for many people involved (from data science through product owners to DevOps), which led to assessing Vertex AI to solve the challenges of bringing ML models into production.

ML in production requires special tooling

After struggling with ML in production initially, the DevOps team started to look for and compare different tools that could help in handling the complexity. That was when they came across the open-source software Kubeflow Pipelines (KFP), an ML workload orchestrator. At first glance it seemed to be a good fit, as Klöckner already used Kubernetes clusters. There was still too much time consuming overhead operating clusters manually, however. Additionally, knowledge about Docker containers and Kubernetes is required to use Kubeflow—which is overkill for most data scientists and areas in which the company’s team was inexperienced.

After having a closer look into Vertex AI, Klöckner encountered Kubeflow Pipelines again, but this time they were able to run their ML workloads on the fully-managed service, Vertex AI Pipelines. The provided software development kit (SDK) by Google made it easy to use Kubeflow Pipelines, even for data scientists with little experience. This let the data science team and the DevOps team focus on the pipelines themselves, instead of managing Kubernetes clusters or the overall infrastructure underneath. Further, the DevOps team felt better prepared this time, as they collectively had finished Coursera’s ML Engineering for Production course. Hence, they better understood ML and MLOps concepts and practices. This helped them to realize what went wrong the first time and what they could do to improve operating a project of this size better across teams. It was critical to not reproduce issues from prior projects and Vertex AI offered everything needed in an integrated bundle. From managed datasets over pipelining to metadata versioning, governance and so much more. Let alone the helping hand of other Google Cloud services such as Cloud Build and Artifact Registry for building, storing, and retrieving Docker container images.

The initial workflow: a good start but imperfect

In their first attempt for the IEPO solution, each GitLab build pipeline started with the commit of a fully trained model. Data scientists manually did all training steps before on their manually managed and long running VM instances in Google Cloud. There were some scripts (e.g., for creating the train/test/validation data split or downloading PDFs to the respective machine), but each scientist needed to have the workflow in mind or consult the different places where it was hopefully correctly written down. The contributors to the software build pipeline in GitLab cared about building a Docker image and the serving infrastructure around the model. In fear of breaking the model build and training, the data scientists did not upgrade their machines in the cloud. A simple change in environment (e.g a simple library update) might have broken their VMs and the model training for several days. They felt that having mostly mathematical backgrounds and more interest in preparing data and improving machine learning models, the maintenance of a VM should be none of their duties. They realized that most packages on used VMs were outdated for two years or more and partly publicly accessible—a real security nightmare for every administrator. Hence, the goal was to go away from everyone developing/hosting on their own, resulting in different software versions etc. There was a clear need for some harmonization across all users and environments.

The result: more standardized & automated workflows for end-to-end ML in production

Trust and patience paid out. The new workflow using Vertex AI Pipelines addresses their previous problems. Everything that goes to production needs to go via Git, otherwise it won’t be deployed. Data scientists needed to understand and learn how to work more like DevOps. A handful of workshops and trainings later, the data science team was convinced and understood the benefits of working through Git-based processes, such as working better collaboratively, faster, consistent, etc. 

Now, each commit to the Git repository is triggering a GitLab pipeline. There, the model code gets dockerized and uploaded to the Google Artifact Registry to which the Kubeflow Pipelines components have easy access. The subsequent build step defines the ML pipeline and creates it in Vertex AI. Data cleanup and splits, model training and validation are all is written in python code with the help of the Vertex AI SDK for Python. Each Kubeflow Pipeline component can request access to a GPU for faster training or more memory for preprocessing of the data. When the model training is finished, the GitLab pipeline continues and deploys the model onto a Kubernetes cluster for serving.

The data science team learned to use Vertex AI Pipelines with the KFP SDK. It took a bit of training on how to use it most efficiently, such as having several pipelines in parallel instead of one big one. It is being done with splitting pipelines automatically depending which language the data has (i.e. split by label that represent language/country) as seen below:

To overcome the security issues with virtual machines, data scientists only use short-termed Jupyterlab instances on Vertex AI Workbench. By convention, the lifetime of such an instance is limited to one ticket or task (whenever someone creates a new ticket, they will receive the latest version). Google manages the machine images (as well as security updates) and data scientists can concentrate on their actual work again.

The improvements that Vertex AI brought to Klökner’s ML workflows have been significant. Experiments that are formalized into components are easier to share and reproduce. Also, parallel training is straightforward to set up using the predefined components and pipelines that they build with the help of their partner, dida. Moreover, the impact gets even more tangible looking at concrete process improvements. With Vertex AI, the time required for developing and training models was significantly reduced. Whereas the initial IEPO model, developed on machines of individual ML experts, could take many days or weeks, it now takes hours to get through first iterations. And with a defined workflow and standard components that have transparent inputs and outputs, human errors are significantly reduced. There are fewer hidden black boxes and more sharing and collaborations. All of this helped to have a purchase order fully processed into their ERP system in several minutes instead of half a day on average.

What’s next?

The teams at Klöckner don’t see their work done yet. Now that they have declared workflows including Vertex AI as their “gold standard,” the plan is to port other ML workloads and previously trained models to Vertex AI. And of course, there is plenty of room for further improvement and optimization in their ML pipelines, such as automatically triggering continuous training pipelines when new data arrives. Similarly, on the serving side, model registration and live deployment of an improved model if committed to master (after metrics comparison and acceptance test) could be brought fully onto Vertex AI and automated as well (e.g., via Model Registry & Vertex AI Endpoints). Additionally, runtime metrics can be tracked to know how a model behaves and data drift detection can provide better understanding of when it’s time for retraining.

And last but not least, Klöckner has many ideas already for new ML use cases to simplify and accelerate their processes (e.g., a model that understands when a customer says “Hey, please execute the same order as last time”). With whatever comes next, one thing is for sure: with Google Cloud and its Vertex AI platform they have powerful technologies and resources to continue being a front runner of their industry.

Further Links / Material:

Coursera course – ML Engineering for Production

More MLOps related articles / best practices…

Best Practices for managing Vertex AI Pipelines code

Building reusable ML workflows with Pipeline Templates

Digitec Galaxus: Reinforcement Learning using TFX

The MLOps Playbook: Best Practices for Ensuring Reliability of ML Systems

MLOps in Glassdoor: an à la carte approach

MLOps Whitepaper

We would like to thank Matthias Berkenkamp (former Sr. DevOps employee of Klöckner), Martin Schneider (Klöckner), Kenny Casagrande (Klöckner) and Kai Hachenberg (Google Cloud) for their help in creating this blog post.

This blog follows our guide to Designing TIC 3.0 compliant solutions on Google Cloud which provides extended guidance for telemetry reporting requirements. 

The National Cybersecurity Protection System (NCPS) provides capabilities to enable the Cybersecurity and Infrastructure Security Agency (CISA) to secure and defend federal civilian agencies’ IT infrastructure against advanced cyber threats. As agencies are moving their IT infrastructure to the cloud, the NCPS program has evolved to ensure that CISA’s analysts can continue to provide situational awareness and support to agencies’ workloads in the cloud.

In this blog, we will provide guidance for how agencies can collect, enrich, and report logs to CISA in alignment with the telemetry cycles described in the NCPS Cloud Interface Reference Architecture (NCIRA) program documentation*. 

Reporting Patterns: 

The NCPS telemetry cycles has three stages for passing information from agency cloud resources to CISA: 

Stage A: Cloud Sensing – Generate the telemetry 

Stage B:  Agency Processing – Prepares the telemetry for communication

Stage C: Reporting to CISA – Includes the transmission of information and transition from the agency to CISA infrastructure and control 

Each stage has attributes to capture the functions that take place within the stage. In the next section, we will discuss each of these stages and how to configure them in Google Cloud. 

A. Cloud Sensing 

The NCPS program identified different sensor positions (Gateway, Subnet, Interface, or Service) and different telemetry types to generate (Network Flow logs, Packet Captures, Application Logs, or Transaction logs). The cloud service model (IaaS, PaaS, or SaaS) will have an impact on the proper attributes and options available to configure when logging is enabled for the service.  

We have gathered configuration guidance for how to enable logging for different Google Cloud services, sensor positions, and telemetry types and made them available in the NCPS for Google Cloud – Cloud Sensing document to achieve the requirements of this stage.  

B. Agency Processing 

In this stage, log data is filtered, enriched, aggregated, and transformed into an appropriate format for ingestion into CISA’s Cloud Log Aggregation Warehouse (CLAW).  On Google Cloud, this is achieved using a Log Router to route desired logs from Cloud Logging to PubSub, and then using a Dataflow pipeline to process the logs from PubSub to further filter, aggregate, enrich, transform, and finally send the finalized logs to CLAW. An example diagram of this Agency Processing is shown below.

To design for scale, agencies can also deploy the PubSub and Dataflow pipeline in a separate project, and configure Log Routers in multiple other projects to export logs in a centralized model as shown below. 

Agencies have options for operations to perform on the logs before sending to CISA in the next stage.

Data Filtering:  Agency log data can be filtered at three levels:

Log Router – The Log Router collects logs from Google Cloud logging and routes them to a destination sink such as BigQuery or PubSub.  The Log Router can specify inclusion or exclusion filters to control which logs are routed to this sink. Here is an example inclusion filter for logs from an external HTTPS load balancer with Cloud Armor for Web Application Firewall (WAF) protection: 

“resource.type:(http_load_balancer) AND jsonPayload.enforcedSecurityPolicy.name:(ca-policy)”

PubSub Filter – When a Log Router sends logs to a PubSub topic, these logs can be further filtered by leveraging a filter on the PubSub subscription.Dataflow Pipeline – Once logs flow to the Dataflow Pipeline, the pipeline can also filter out (exclude) log entries or specific attributes of a given log entry by leveraging a custom Beam ParDo transformation in the pipeline.

Data Enrichment:  Log data can also be enriched as the logs flow through Dataflow.  An example of this would be using a Google Cloud API to get the load balancer host and port in order to add those items to each log line for that load balancer.

Data Aggregation:  Log data can be aggregated as the logs flow through Dataflow.  An example of this would bejoining two sources of data within a fixed window of time (such as every 5 minutes) to produce a single log file that is a logical concatenation of other logs within that window.

Data Transformation: Log data can also be transformed as the logs flow through Dataflow.  An example of this would be converting log dates to specific formats and timezones, splitting log line properties to multiple properties, or conjoining log line properties to a single property.

C. Reporting to CISA

This is the final stage of agency processing. Though data can be transferred in either a push or pull model, CISA recommends the push model to enable agencies’ control of ingestion rate. Agencies will need to work with CISA to get the needed information about the receiving end credentials.  With those credentials, the Dataflow pipeline can create a sink that sends the final logs to CISA CLAW on a desired window interval (generally 15 minutes). 

Optionally, Agencies can add an additional sink to their pipeline to archive the logs exactly as they are sent to CISA CLAW.  In most cases, this would be accomplished by creating a BigQuery sink in the Dataflow pipeline that stores the same Dataflow PCollection that is being sent to CISA CLAW.

Sample Pipeline

A full sample of what an agency’s Dataflow pipeline might look like is available here.  A screenshot of that sample which shows the specific Dataflow code that reads from logs from PubSub, enhances the logs, and sends the logs to CLAW is shown below:

Conclusion 

Google has a history of supporting government agency requirements such as FedRAMP, ITAR, NIST SP 800-53, DoD IL4 and IL5, and many others.  Google is continuing this pattern to help Federal Civilian agencies comply with CISA’s guidance regarding TIC 3.0.  

If you would like any additional information or help complying with this CISA guidance in your Google Cloud environment, please reach out to your Google Account Team, or to the Google TIC 3.0 technical team at tic-initiative@google.com. 

TL;DR – Budgets can do more than just track costs! You can set up automated cost controls using programmatic budget notifications, and we have an interactive walkthrough with sample architecture to help get you started.

There’s a few blog posts on what Google Cloud Budgets are and how to use them for more than just sending emails by using programmatic budget notifications. These are important steps to take when using Google Cloud, so you can accurately ask and answer questions about your costs and get meaningful answers in the systems you already use. As your cloud usage grows and matures, you may also need to be more proactive in dealing with your costs.

More than just a budget

To recap: budgets let you create a dynamic way of being alerted about your costs, such as getting emails when you’ve spent or are forecasted to spend a certain amount. When creating a budget, you can provide a fixed amount or you can have the amount based on the previous period, so you could set up a budget that alerts you if your spending has changed significantly in a monthly cadence. In addition, you can have budgets send data to Pub/Sub on a regular basis (programmatic budget notifications) that can be used however you’d like, such as sending messages to Slack.

Budgets that send out notifications are flexible enough to do just about anything, but that’s also where things can become a bit tricky to set up. If you’re monitoring the costs for a large company with a lot of cloud usage, that could involve multiple environments with lots of products being used in different ways. Being informed about the costs is a good starting point, but you’ll likely want to set up automated cost controls to protect yourself and your cloud spending.

In essence, setting up automated cost controls is the same as using programmatic budget notifications: the budget occasionally sends out a Pub/Sub message, and you create a Cloud Function (or similar) subscriber that receives that message and runs some code. Of course, the specifics of that code might be anything and will heavily depend on your business logic needs, ranging from sending a text message all the way to shutting down cloud resources. While the specifics are up to you, we made a few things to make getting started easier!

Show me the way

We’ve created an interactive walkthrough to help you with all of the steps needed in getting programmatic budget notifications up and running.

Following the walkthrough, you’ll set up a budget, Pub/Sub topic, and Cloud Function that work together to respond to programmatic notifications. Not only will you get a sense of all the pieces involved, you can easily modify the code from the function for your specific purposes, so it serves as a great starting point. That also leads to a question I’ve heard often: “This is great, but what code am I supposed to use?” And that is why we’ve expanded our walkthrough to include a full, one-click architecture deployment!

It’s like a sentry, but for your cloud costs

Cost Sentry, powered by DeployStack, takes the next step in programmatic budget notifications and sets up all the pieces needed to create basic automated cost-enforcement, as well as some example architecture to test it on! In fact, the overall architecture isn’t much more than just setting up the programmatic budget notifications alone, but it gives a good example of how that could work in a full environment. 

This architecture will get deployed for you, along with the working code to handle a programmatic budget notification and interact with Compute Engine and Cloud Run.

Both the walkthrough and deploying the Cost Sentry stack can be used as the starting point for a full automated cost-enforcement solution. With these samples, you’ll want to take a look at the Cloud Function code that receives data from your budget, and how it interacts with the Google Cloud APIs to shut down resources. In this example, any Compute Engine instances or Cloud Run deployments that have been labeled with ‘costsentry’ will be shut-down/disabled when your budget exceeds the configured amount.

While this is a great solution for getting an automated cost-enforcement solution started, the hard part is probably in the next questions you’ll need to answer for your use case. Questions like “What do I actually want to have happen when I hit my budget?” and “Will stopping all of these instances automatically have ramifications?” (spoiler alert: probably) are important ones to figure out when looking at the full scope of a cost-enforcement solution.

Setting up a full automated cost enforcement solution gives you the flexibility to customize your response to budget updates, such as sending higher-priority messaging as you get closer to your budget total, and taking action by shutting down services when you greatly exceed your budget. Any way that you want to build a solution, this is a great starting point!

Go forth, and do

This may seem like a lot, and I’m a big fan of the “crawl, walk, run” philosophy. If you’re new to Google Cloud, get started by just setting up a budget for all of your costs. From there, you can work with programmatic budget notifications to start expanding how you use budgets. As you get more familiar with Google Cloud, you’ll likely need to customize your cost controls and start with Cost Sentry to set up your automated cost-enforcement solution.

Check out the interactive walkthrough and Cost Sentry architecture to get started!

Academic institutions are becoming more susceptible to security breaches in the ever-expanding ecosystem of IT services. While facing challenges such as re-training staff on new skills, and expanding teams to support the increasing pace of work, many institutions are seeking tools and solutions to alleviate some burden on their IT security teams.  

One of those tools is Security Command Center, which is available in Google Cloud to help protect institutions. Security Command Center is Google Cloud’s native security and risk management platform that helps educational institutions manage and improve their cloud security and risk posture. Security Command Center also helps institutions to protect against threats by providing security alerts against common risks such as crypto-mining or leaked credentials.

For example, faculty, staff and researchers commonly use Google Cloud to perform computations on data they’ve collected. A researcher can develop a new project in the institution’s Google Cloud organization and create the necessary virtual machines they need to run their tests, and then pass the remaining tasks to their students who work in the lab. As more students and staff access these virtual machines, the Institution’s security teams need to ensure their cloud environment remains secure. 

For workloads created by the institution, security teams are interested in the following key aspects for keeping their environment secure: 

Visibility: understand the number of projects and resources that are deployed. The researchers or graduate students working in the lab could succumb to a phishing email, putting their Google Cloud credentials at risk. With the Google Cloud credentials, hackers can perform unauthorized operations, spin up workloads for malicious activity, potentially causing brand damage and financial implications. 

Detection: uncover threats targeting the institution’s resources; If the researchers are dealing with sensitive data like protected health information (PHI), it’s possible this data can be exfiltrated and cause significant damages if security teams are not alerted.

Misconfigurations: identify security misconfigurations and compliance violations in and resolve them by following actionable recommendations. When any institution faces a breach that compromises data, the brand can become tarnished. This can cause organizations to lose funding from external sources or students who lose trust in the security of their data.

Security Command Center (SCC) Premium helps institutions remain secure and understand their compliance posture through features such as:

Cloud Asset Inventory: A tightly integrated product with Security Command Center enables you to discover, monitor, and analyze your assets in one place. Institutions can verify what is running in their cloud environment and IT staff are alerted if a  resource is associated with any security findings.

Security Health Analytics: A service that can identify misconfigurations in an institution’s Google Cloud resources. In addition, all findings are tied to the industry standard and compliance benchmarks. If any resource misconfigurations are identified or any vulnerabilities are detected, IT security staff can be alerted.

Web Security Scanner: A service that automatically detects web applications running in Google Cloud and starts scanning them for vulnerabilities. By identifying vulnerabilities, institution’s can help prevent attacks.

Event Threat Detection: A service that analyzes platform logs for identifying malicious activity, and can help an institution be alerted to a potential threat.

Container Threat Detection: A built-in service that detects the most common container runtime attacks and alerts security teams to any suspicious activity.

Virtual Machine Threat Detection: A service that provides an agentless memory scanning to help detect and alert an institution to threats like crypto mining malware inside the virtual machines running in Google Cloud.

Compliance Monitoring: Identify and remediate compliance violations among your assets in easily accessible reports

Security Command Center Premium can help provide educational institutions with the visibility of their security posture and surfaces threats, misconfigurations, and vulnerabilities in their Google Cloud environment before an attack can be successful. To learn more, check out our session “Actionable security for institutions with Google Cloud tools” from the Student Success series or chat with our team.

Everyone who has experienced a loved one’s cancer diagnosis knows how important it is to help patients get the best treatment, faster. Cancer is a leading cause of death, killing nearly 10 million people globally each year. But if caught early, and treated appropriately, many people can recover. We’re proud to have worked in this area for many years, applying our technology and tools to fight cancer, in partnership with leading public and private sector organizations.

In 2018, we published a peer-reviewed paper in the Archives of Pathology & Laboratory Medicine about how we applied deep learning to improve breast cancer diagnostic accuracy, based on gigapixel-sized pathology slides of lymph nodes from de-identified patients. In 2019, we published a paper in Nature Medicine about using advances in 3D volumetric modeling and artificial intelligence (AI) to better predict lung cancer. And, in 2021, we announced a new clinical research study exploring whether AI models can reduce the time to diagnosis for breast cancer patients, narrowing the assessment gap and improving the patient experience.

This is hard, important work, and due to the complexity and diversity of cancer, building accurate AI models requires large, diverse, and high-quality datasets, such as images of de-identified pathology slides. Digitized pathology slides can play a critical role in building machine learning (ML) and AI models, making it possible for researchers to test and compare the power of different diagnostic approaches. More importantly, they allow researchers to build models that reflect diverse patient populations and include multiple types of cancer.

Google’s work on government pathology datasets

We’re honored to have partnered with Naval Medical Center San Diego (NMCSD), Marine Corps Base Camp Pendleton, and U.S. Navy Medicine Readiness and Training Command Guam (U.S. NMRTC Guam) to manually clean, catalog, and scan pathology slides, effectively digitizing them to make them more helpful in supporting cancer research. All of this was done through a CRADA, or Cooperative Research and Development Agreement, a common framework for research collaboration between government agencies and companies.

Google paid for digitizing these slides and making them available for research, and the CRADA for NMCSD (and other Department of Defense facilities) provided the government joint ownership of any inventions made collaboratively. Once the data was digitized, NMCSD had the option of paying cloud storage fees to Google (or any other cloud provider). In other words, the government and the research community at large were (and still are) the primary beneficiaries of this critical effort.

It’s also important to note that all of our work with NMCSD—and any project involving data in our cloud—is subject to strict controls to maintain patient privacy and security. We work with de-identified patient data and use strong encryption. Our customers own their data, and we cannot, and do not, use it for any purpose other than explicitly agreed upon by them. (These principles are outlined in our Privacy Resource Center.)

Google’s involvement with the Joint Pathology Center biorepository

Established in 1862, the Armed Forces Institute of Pathology (AFIP) has amassed the world’s largest collection of human pathologic specimens. Samples from the AFIP were instrumental in solving many public health challenges, like sequencing the genome of the 1918 influenza virus. In 2005, AFIP’s biorepository was transferred to the newly created Joint Pathology Center (JPC). During the transition, the DoD asked the U.S. Institute of Medicine (now called the National Academy of Medicine) to provide advice on operating the biorepository. The resulting study recommended digitizing the archives immediately to ensure that the repository did not continue to degrade and could be made accessible to researchers for years to come.

In March 2020, the Defense Innovation Board, a group of industry experts that serves the Secretary of Defense, wrote a report outlining the urgency of creating a fully digitized repository of slides for the JPC and making recommendations on how to achieve this goal. Two other organizations—the DoD Joint AI Center and the Defense Innovation Unit—then released a solicitation seeking industry help. They canvassed the country, using an open buying process, to identify and contract with vendors who could build algorithms and digitize slides. Google was ultimately selected to develop algorithms to detect four types of cancer and digitize at scale, using the proven techniques we had developed previously with NMCSD and NMRTC Guam.

The JPC, however, pursued its own process. This started with the organization putting forth an open call to vendors who could assist the digitization. But then, in August 2020, JPC went forward with a non-competitive, non-publicly-sourced contract. We’re obviously disappointed we weren’t given the opportunity to compete for this important project, but we continue to this day to offer our help and collaboration for this and other digitization initiatives.

Helping billions of people live healthier lives

We remain optimistic that if the repository can be digitized and leveraged to its full potential—and used by researchers and clinicians—it would advance diagnosis and treatment for thousands of illnesses (including cancer), saving American lives, such as those of our service members. And, if digitized quickly and effectively, the repository would be a highly effective use of taxpayer dollars, providing enormous downstream benefits for the U.S. healthcare system. We stand ready to assist. 

Google supports a number of projects designed to help people live healthier lives. We are proud of the work we have done in partnership with many public and private organizations, to democratize access to early diagnostics and treatments for service members, their families, and communities around the world.

Savvy leaders at organizations around the world know that digital transformations can create a virtuous flywheel of more and faster innovation, driven by the power of software integration. APIs can facilitate the necessary software integration and communication, and that requires serious consideration of the organization’s API security posture to better protect its data and digital systems.

The proliferation of APIs has led to expanded attack surfaces and greater inherent risk. A quick scan of the digital landscape shows that traffic generated by APIs now dominates the internet. Google Cloud’s Apigee reports that for their customers alone, API traffic increased 46% between 2019 and 2020.

At the same time, Google Cloud’s 2022 report on API security insights and trends notes that more than 50% of organizations faced an API-related security threat at least once in 2021, confirming that APIs have become a favorite target for threat actors.

This month, we identified five categories of API attacks in a new report, “The Importance of API Security,” that takes a deeper look at how to build better API security systems. These threats include:

Data scraping

Denial of service (DoS) 

Injections or malware 

Account takeover (ATO) 

Scalping and bots 

While DoS, injections, and ATO are well-known attacks that came to the API world from web applications, abuse and bots are unique threats for APIs that are by their nature different from security issues. Security leaders should be concerned with how prepared their organizations are for API security threats. 

The current state of API security strategy

Our 2022 report on API security insights and trends found that most organizations don’t have a robust API security strategy in place, and that 60% say that their API strategy needs improvement. 

Organizations face two primary impediments when establishing and maturing their API security capabilities: A lack of resources, and a lack of experience. Even leaders who are committed to securing APIs may end up deploying fragmented solutions across their organization, which could inadvertently create a false sense of security. 

So what’s the best way forward when it comes to protecting APIs?

Taking a defense-in-depth approach

We believe that layering API defense mechanisms that support each other but are otherwise independent is an effective way to stop adversaries. Along with our additional insights into a rapidly-changing API threat landscape in our “Importance of API Security” report, we also provide recommendations on launching an API-first security strategy that builds on four pillars: 

Essential API security controls and protections enforced for all APIs through a common API management platform, creating a no-exception approach that leaves no space for uncontrolled API exposure

Protection against DDoS and exploits with an adaptive cloud protection suite that includes a WAF, machine-learning-based DDoS protection, and a threat intelligence capability

Anti-bot protection to keep APIs and exposed resources safe from fraudulent activity, spam, and abuse

Adherence to safe API coding principles to prevent the most common classes of security issues upfront

You can read the report here, and reach out to the Google Cybersecurity Action Team to learn more.

Getting into the details

We wrote previously about how we used clustering to connect requests for support (in text form) to the best tech support articles so we could answer questions faster and more efficiently. In a constantly changing environment (and in a very oddball couple of years) we wanted to make sure we’re focused on preserving our people’s productivity by isolating, understanding and responding to new support trends as fast as we can.

Now we’d like to get into a bit more detail about how we did all that and what went on behind the scenes of our process: 

Extraction

Google’s historical support ticket data and metadata are stored in BigQuery, as are the analysis results we generate from that data. We read and write that content using the BigQuery API. However, much of these tickets contain information that is not useful to the ML pipeline and should not be included in the preprocessing and text modeling phases. For example, boilerplate generated from our case management tools must be stripped out using regex and other technologies in order to isolate the IT interaction between the tech and users. 

Furthermore, once all boilerplate has been removed, we use part-of-speech tagging to isolate only the nouns within the interaction, since nouns themselves proved to be the best features for modeling an interaction and differentiating a topic. Any one interaction could have 100+ nouns depending on the complexity.  Using these nouns, we take one more step and use stemming and lemmatization to remove any suffix that may be placed on the noun (e.g., “computers” becomes “computer”). This allows for any modification of the root words to be modeled as the same feature and reduces noise in our clustering results.

Once each interaction is transformed into a set of nouns (and unique identifier), we can then move on to more advanced preprocessing techniques.

Text Modeling

To cluster the ticket set, it must first be converted into a robust feature space. The core technology underlying our featurization process is TensorFlow transformers, which can be invoked using the TFX API. TensorFlow parses and annotates the tickets’ natural-language contents and these annotations, once normalized and filtered, form a sparse feature space. The Cloud Data Loss Prevention (DLP) API redacts several categories of sensitive information — e.g., person names — from the tickets’ contents, which both mitigates privacy leakage and prunes low-relevance tokens from the feature space.

Although clustering can be performed against a sparse space, it is typically more effective if the space is densified to prune excessive dimensionality. We accomplish this using the term frequency-inverse document frequency (TF-IDF) statistical technique with a predefined maximum feature count – we also investigated more heavy-duty densification strategies using trained embedding models, but found that the quality improvements over TF-IDF were marginal for our use case, at the cost of a substantial reduction in human interpretability.

Clustering

The generated ticket feature set is partitioned into clusters using ClustOn. As this is an unsupervised learning problem, we arrived at the clustering process’s hyper-parameterization values via experimentation and human expert analysis. The trained parameters produced by the algorithm are persisted between subsequent runs of the pipeline in order to maintain consistent cluster IDs; this allows later operational systems to directly track and evaluate a cluster’s evolution over real time. 

The resulting cluster set is sanity-checked by some basic heuristic measures, such a silhouette score, and then rejoined with the initial ticket data for analysis. Moreover, for privacy purposes, each cluster whose ticket cohort size falls below a predefined threshold is omitted from the data set; this ensures that cluster metadata in the output, such as feature data used to characterize the cluster, cannot be traced with high confidence back to individual tickets.

Scoring & Anomaly Detection

Once a cluster has been identified, we need a way to automatically estimate how likely it is that the cluster has recently undergone a state change which might indicate an incipient event, as opposed to remaining in a steady state. “Anomalous” clusters — i.e. those which exhibit a sufficiently high likelihood of an event — can be flagged for later operational investigation, while the rest can be disregarded.

Modeling a cluster’s behavior over time is done by distributing its tickets into a histogram according to their time of creation — using 24-hour buckets, reflecting the daily business cycle — and fitting a zero-inflated Poisson regression to the bucket counts using statsmodel1. However, our goal is not just to characterize a cluster’s state, but to detect a discrete change in that state. This is accomplished by developing two models of the same cluster: one of its long-term behavior, and the other of its short-term behavior. The distinction between “long-term” and “short-term” can be as simple as partitioning the histogram’s buckets at some age threshold. But we chose a slightly more nuanced approach: both models are fitted to the entire histogram, but under two different weighting schemata; both decay exponentially by age, but at different rates, so that recent buckets are weighted relatively more heavily in the short-term model than the long-term one.

Both models are “optimized,” in that each achieves the maximum log-likelihood in its respective context. But if the long-term model is evaluated in the short-term context instead, its log-likelihood will show some amount of loss relative to the maximum achieved by the short-term model in the same context. This loss reflects the degree to which the long-term model fails to accurately predict the cluster’s short-term behavior — in other words, the degree to which the cluster’s short-term behavior deviates from the expectation established by its short-term behavior — and thus we refer to it as the deviation score. This score serves as our key measure of anomaly; if it surpasses a defined threshold, the cluster is deemed anomalous.

Operationalize

Using the IssueTracker API, bugs are auto-generated each time an anomalous cluster is detected. These bugs contain some summary of the tokens found within the cluster itself as well as a parameterized link to the DataStudio dashboard. These dashboards show the size of the cluster over time, the deviation score and the underlying tickets. 

These bugs are picked up by Techstop operations engineers and investigated to determine the root causes, allowing for quicker boots on the ground for any outages that may be occurring, as well as a more harmonious flow of data between support operations and change and incident management teams.

Staying within the IssueTracker product, operations engineers create Problem Records in a separate queue detailing the problem, stakeholders and any solution content. These problem records are shared widely with frontline operations to help address any ongoing issues or outages.

However, the secret sauce does not stop there. Techstop then uses Google’s Cloud AutoML engine to train a supervised model to classify any incoming support requests against known Problem Records (IssueTracker bugs). This model acts as a service for two critical functions:

The model is called by our Chrome extension (see this handy guide) to recommend Problem Records to frontline techs based on the current ongoing chat. For a company like Google that has a global IT team, this recommendation engine allows for coverage and visibility of issues in near real time

The model answers the “how big” question: Many stakeholders want to know how big the problem was, how many end users did this problem affect and so on. By training an AutoML model we can now give good estimators about impact and more importantly we can measure impact of project work that addresses these problems.

Resampling & User Journey Mapping

Going beyond incident response, we then semi-automatically extracts user journeys from these trends by sampling each cluster to discover the proportion of user intents. These intents are then used to map user pitfalls and generate a sense of topic for each emerging cluster.

Since operations are constrained by tech evaluation time, a solution to limit the number of reviews necessary that each agent would need to inspect, while still maintaining the accuracy of analysis, was derived. 

User intents are defined as user “Goals” an employee may have when engaging with IT support. For example, “I want my cell phone to boot” or “I lost access to an internal tool” are good examples. Therefore, we propose a two-step procedure (to be applied for each cluster).

First, we sample chats until the probability that we discover a new intent is small (say <5% or whatever number we want). We can evaluate this probability at each step through the Good-Turing method.
A simple Good-Turing estimate of this probability can be found as E(1) / N, where N is the number of sampled chats so far and E(1) is approximately the number of intents that have only been seen once so far. This number should be lightly smoothed for better accuracy; it’s easy to implement this smoothing on our own2 or call a library.

Once we have finished, we take the intents that we consider representative (say there are k of them) and create one additional category for “other intents.” Then, we estimate the sample size for multinomial estimation (with k+1 categories) that we still need to reach, given composition accuracy (say, that each intent fraction is within e.g., 0.1 or 0.2 of the actual fraction). To do so, we consider Thompson’s procedure3, but take advantage of the data collected so far to be used as a plugin estimate for the possible values of the parameters, plus we should also consider a grid of parameter values within a confidence interval of the current plugin estimate, to be sufficiently conservative. The procedure is described on page 43 in this article, steps (1) and (2). The procedure is easy to implement and under our current setup, it will be a few lines of code.

The procedure gives us the target sample size. If we have already reached this sample size in step 1, we are done. Otherwise, we sample a few more chats to reach this sample size.

This work along with the AutoML model allows Google to understand not only the problem impact size, but also key information about user experiences and where the CUJ users are struggling the most. In many cases a problem record will contain multiple CUJs (user intents) with separate personas and root causes. 

Helping the business

Once we can make good estimators for different user goals we can work with domain experts to map clear user journeys, i.e., we can now use the data that this pipeline has generated to construct a user journey in a bottoms-up approach. This same amount of work, sifting through data, aggregating similar cases and estimating proportions of user goals would take an entire team of engineers and case scrubbers. With this ML solution we can now get the same (if not better) results with much lower operational costs.

These user journeys then can be fed to internal dashboards for key decision makers to understand the health of their products and service areas. It allows for automated incident management and acts as a safeguard against unplanned changes or user-affecting changes that did not go through the proper change management processes. 

Furthermore, it is critical for problem management and other core functions within our IT service. By having a small team of operational engineers reviewing the output of this ML pipeline, we can create healthy problem records and keep track of our team’s top user issues.

How do I do this too?

Want to make your own system for insights into your support pipeline? Here’s a recipe to follow that will help you build all the parts you need

Load your data into BigQuery – Cloud BigQuery

Vectorize it with TF-IDF – TensorFlow Vectorizer

Perform clustering – TensorFlow Clustering

Score Clusters – Statsmodels Poisson Regression

Automate with Dataflow – Cloud DataFlow

Operationalize – IssueTracker API

1. When modeling a cluster, that cluster’s histogram serves as the regression’s endogenous variable. Additionally, the analogous histogram of the entire ticket set, across all clusters, serves as an exogenous variable. The latter histogram captures the overall ebb and flow in ticket generation rates due to cluster-agnostic business cycles (e.g. rates tend to be higher on weekdays than weekends), and its inclusion mitigates the impact of such cycles on each cluster’s individual model.

2. Gale, William A., and Geoffrey Sampson. “Good‐turing frequency estimation without tears.” Journal of quantitative linguistics 2.3 (1995): 217-237.

3. Thompson, Steven K. “Sample size for estimating multinomial proportions.” The American Statistician 41.1 (1987): 42-46.

Editor’s note: This post is part of an ongoing series on IT predictions from Google Cloud experts. Check out the full list of our predictions on how IT will change in the coming years.

Prediction #3: By 2025, 90% of security operations workflows will be automated and managed as code

There is not enough funding, resourcing, skills, or broadly applicable solutions to help manage security risk effectively across modern technology environments. Organizations are struggling to identify which alerts and security areas to prioritize while moving quickly through their digital transformation. This challenge is compounded by an exponential increase in data volume, alert fatigue, financial costs, and overall complexity. To combat this, organizations are looking to drive better developer hygiene, leverage more managed services and cloud-native capabilities, use products and solutions that provide greater security-by-default, and shift to security engineering over operations, to manage risk at scale.

Security operations — traditional detection and response workflows — are notoriously overburdened with toil. There are quite simply too many events and not enough people to scale them. Legacy tools coupled with a high bar for security engineering have made it very difficult for organizations to build effective, scalable solutions to manage threats in modern technology environments. As a result, there’s a cybersecurity talent shortage of over 700,000 jobs, which will likely increase and never be filled.1

This new 90/10 split predicted between automated and manual detection and response events can allow security operations teams to focus on their critical security work: threat research and operationalizing threat intelligence, proactive hunting, solving for visibility challenges, maturing alert triage and response automation capabilities, and more importantly, shifting security operations knowledge “left.” This last point can drive a deeper relationship with developers and improve the preventive security of the overall infrastructure.

To achieve this vision, we’ve developed the Autonomic Security Operations (ASO) framework, a holistic and novel approach to modernizing people, processes, and technologies – enabling organizations to adopt Google’s cloud-scale engineering approach to threat management. This framework underpins our substantial technology investments in Chronicle Security Operations, VirusTotal, Mandiant, and beyond.

At the core of ASO is Continuous Detection, Continuous Response (CD/CR), a model we’ve developed for traditional security operations teams to help shift away from the assembly-line approach to managing threats and adopt an agile operating model centered around establishing continuous feedback loops across the core areas of detection and response, in order to objectively and iteratively improve an organization’s security capabilities. It’s heavily grounded in our own approach to security as well as other methodologies, such as DevOps, SRE, Detection Engineering, and Agile.

Some examples of the CD/CR model include: 

Taking an API-first approach to security operations. We’ve heavily invested in developing APIs for most aspects of Chronicle Security Operations, allowing organizations to codify their approach to threat management from instantiating visibility, developing and deploying security analytics, creating response automation playbooks, and deploying dashboards, to tracking KPIs.

Deploying security analytics as-code. While we’re developing curated built-in detections in Chronicle Security Operations and native threat-detection capabilities through Security Command Center, we’re also fostering community collaboration on developing security analytics in our Community Security Analytics repository.These analytics can be deployed as-code across Chronicle and other analytics tools in Google Cloud.

In order for security operations teams to become an autonomic function of their organizations and scale across the threats their businesses face, they will need to adopt modern, developer-friendly workflows like CD/CR, which can free them to prioritize the most important threats to their organizations.

If you’d like to learn more about Google Cloud’s approach to automating security operations, start with the white paper Autonomic Security Operations: 10X transformation of Security Operations Center and watch our latest ASO webinar.

1. Announcement of White House National Cyber Workforce and Education Summit

StreamNative, a company founded by the original developers of Apache Pulsar and Apache BookKeeper, is partnering Google Cloud to build a streaming platform on open source technologies. We are dedicated to helping businesses generate maximum value from their enterprise data by offering effortless ways to realize real-time data streaming. Following the release of StreamNative Cloud in August 2020, which provides scalable and reliable Pulsar-Cluster-as-a-Service, we introduced StreamNative Cloud for Kafka. This is to enable a seamless switch between Kafka API and Pulsar. We then launched StreamNative Platform to support global event streaming data platforms in multi-cloud and hybrid-cloud environments.

By leveraging our fully-managed Pulsar infrastructure services, our enterprise customers can easily build their event-driven applications with Apache Pulsar and get real-time value from their data. There are solid reasons why Apache Pulsar has become one of the most popular messaging platforms in modern cloud environments, and we have strong beliefs in its capabilities of simplifying building complex event-driven applications. The most prominent benefits of using Apache Pulsar to manage real-time events include:

Single API: When building a complex event-driven application, it traditionally requires linking multiple systems to support queuing, streaming and table semantics. Apache Pulsar frees developers from the headache of managing multiple APIs by offering one single API that supports all messaging-related workloads.

Multi-tenancy: With the built-in multi-tenancy feature, Apache Pulsar enables secure data sharing across different departments with one global cluster. This architecture not only helps reduce infrastructure costs, but also avoids data silos.

Simplified application architecture: Pulsar clusters can scale to millions of topics while delivering consistent performance, which means that developers don’t have to restructure their applications when the number of topic-partitions surpasses hundreds. The application architecture can therefore be simplified.

Geo-replication: Apache Pulsar supports both synchronous and asynchronous geo-replication out-of-the-box, which makes building event-driven applications in multi-cloud and hybrid-cloud environments very easy.

Facilitating integration between Apache Pulsar and Google Cloud

To allow our customers to fully enjoy the benefits of Apache Pulsar, we’ve been working on expanding the Apache Pulsar ecosystem by improving the integration between Apache Pulsar and powerful cloud platforms like Google Cloud. In mid-2022, we added Google Cloud Pub/Sub Connector for Apache Pulsar, which enables seamless data replication between Pub/Sub and Apache Pulsar, and Google Cloud BigQuery Sink Connector for Apache Pulsar, which synchronizes Pulsar data to BigQuery in real time, to the Apache Pulsar ecosystem.

Google Cloud Pub/Sub Connector for Apache Pulsar uses Pulsar IO components to realize fully-featured messaging and streaming between Pub/Sub and Apache Pulsar, which has its own distinctive features. Using Pub/Sub and Apache Pulsar at the same time enables developers to realize comprehensive data streaming features on their applications. However, it requires significant development effort to establish seamless integration between the two tools, because data synchronization between different messaging systems depends on the functioning of applications. When applications stop working, the message data cannot be passed on to the other system.

Our connector solves this problem by fully integrating with Pulsar’s system. There are two ways to import and export data between Pub/Sub and Pulsar. The first, is the Google Cloud Pub/Sub source that feeds data from Pub/Sub topics and writes data to Pulsar topics. Alternatively, the Google Cloud Pub/Sub sink can pull data from Pulsar topics and persist data to Pub/Sub topics. Using Google Cloud Pub/Sub Connector for Apache Pulsar brings three key advantages:

Code-free integration: No code-writing is needed to move data between Apache Pulsar and Pub/Sub.

High scalability: The connector can be run on both standalone and distributed nodes, which allows developers to build reactive data pipelines in real time to meet operational needs.

Less DevOps resources required: The DevOps workloads of setting up data synchronization are greatly reduced, which translates into more resources to be invested in unleashing the value of data.

By using the BigQuery Sink Connector for Apache Pulsar, organizations can write data from Pulsar directly to BigQuery. This is unlike before, where developers could only use Cloud Storage Sink Connector for Pulsar to move data to Cloud Storage, and then query the imported data with external tables in BigQuery which had many limitations,  including low query performance and no support for clustered tables.

Pulling data from Pulsar topics and persisting data to BigQuery tables, our BigQuery sink connector supports real-time data synchronization between Apache Pulsar and BigQuery. Just like our Pub/Sub connector, Google Cloud BigQuery Sink Connector for Apache Pulsar is a low-code solution that supports high scalability and greatly reduces DevOps workloads. Furthermore, our BigQuery connector possesses the Auto Schema feature, which automatically creates and updates BigQuery table structures based on the Pulsar topic schemas to ensure smooth and continuous data synchronization.

Simplifying Pulsar resource management on Kubernetes

All the products of StreamNative are built on Kubernetes, and we’ve been developing tools that can simplify resource management on Kubernetes platforms like Google Cloud Kubernetes (GKE). In August 2022, we introduced Pulsar Resources Operator for Kubernetes, which is an independent controller that provides automatic full lifecycle management for Pulsar resources on Kubernetes.

Pulsar Resources Operator uses manifest files to manage Pulsar resources, which allows developers to get and edit resource policies through the Topic Custom Resources that render the full field information of Pulsar policies. It enables easier Pulsar resource management compared with using command line interface (CLI) tools, because developers no longer need to remember numerous commands and flags to retrieve policy information. Key advantages of using Pulsar Resources Operator for Kubernetes include:

Easy creation of Pulsar resources: By applying manifest files, developers can swiftly initialize basic Pulsar resources in their continuous integration (CI) workflows when creating a new Pulsar cluster.

Full integration with Helm: Helm is widely used as a package management tool in cloud-native environments. Pulsar Resource Operator can seamlessly integrate with Helm, which allows developers to manage their Pulsar resources through Helm templates.

How you can contribute

With the release of Google Cloud Pub/Sub Connector for Apache Pulsar, Google Cloud BigQuery Sink Connector for Apache Pulsar, and Pulsar Resources Operator for Kubernetes, we have unlocked the application potential of open tools like Apache Pulsar by making them simpler to build, easier to manage, and extended their capabilities. Now, developers can build and run Pulsar clusters more efficiently and maximize the value of their enterprise data. 

These three tools are community-driven services and have their source codes hosted in the StreamNative GitHub repository. Our team welcomes all types of contributions for the evolution of our tools. We’re always keen to receive feature requests, bug reports and documentation inquiry through GitHub, emails or Twitter.

Financial institutions have vast amounts of data about their customers. However, many of them struggle to leverage data to their advantage. Data may be sitting in silos or trapped on costly mainframes. Customers may only have access to a limited quantity of data, or service providers may need to search through multiple systems of record to handle a simple customer inquiry. This creates a hazard for providers and a headache for customers. 

Elastic and Google Cloud enable institutions to manage this information. Powerful search tools allow data to be surfaced faster than ever – Whether it’s card payments, ACH (Automated Clearing House), wires, bank transfers, real-time payments, or another payment method. This information can be correlated to customer profiles, cash balances, merchant info, purchase history, and  other relevant information to enable the customer or business objective. 

This reference architecture enables these use cases:

1. Offering a great customer experience: Customers expect immediate access to their entire payment history, with the ability to recognize anomalies. Not just through digital channels, but through omnichannel experiences (e.g. customer service interactions).

2. Customer 360: Real-time dashboards which correlates transaction information across multiple variables, offering the business a better view into their customer base, and driving efforts for sales, marketing, and product innovation.

3. Partnership management: Merchant acceptance is key for payment providers. Having better access to present and historical merchant transactions can enhance relationships or provide leverage in negotiations. With that, banks can create and monetize new services.

4. Cost optimization: Mainframes are not designed for internet-scale access. Along-side with technological limitation, the cost becomes a prohibitive factor. While Mainframes will not be replaced any time sooner, this architecture will help to avoid costly access to data to serve new applications.

5. Risk reduction: By standardizing on the Elastic Stack, banks are  longer limited in the number of data sources they can ingest. With this, banks can better respond to call center delays and potential customer-facing impacts like natural disasters. By deploying machine learning and alerting features, banks can detect and stamp out financial fraud before it impacts member accounts.

Architecture

The following diagram shows the steps to move data from Mainframe to Google Cloud, process and enrich the data in BigQuery, then provide comprehensive search capabilities through Elastic Cloud.

This architecture includes the following components:

Move Data from Mainframe to Google Cloud

Moving data from IBM z/OS to Google Cloud is straightforward with the Mainframe Connector, by following simple steps and defining configurations. The connector runs in z/OS batch job steps and includes a shell interpreter and JVM-based implementations of gsutil, bq and gcloud command-line utilities. This makes it possible to create and run a complete ELT pipeline from JCL, both for the initial batch data migration and ongoing delta updates.

A typical flow of the connector includes:

Reading the mainframe dataset

Transcoding the dataset to ORC

Uploading ORC file to Cloud Storage

Register ORC file as an external table or load as a native table

Submit a Query job containing a MERGE DML statement to upsert incremental data into a target table or a SELECT statement to append to or replace an existing table

Here are the steps to install the BQ MainFrame Connector:

copy mainframe connector jar to unix filesystem on z/OS

copy BQSH JCL procedure to a PDS on z/OS

edit BQSH JCL to set site specific environment variables

Please refer to the BQ Mainframe connector blog for example configuration and commands.

Process and Enrich Data in BigQuery

BigQuery is a completely serverless and cost-effective enterprise data warehouse. Its serverless architecture lets you use SQL language to query and enrich Enterprise scale data. And its scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes. An integrated BQML and BI Engine enables you to analyze the data and gain business insights. 

Ingest Data from BQ to Elastic Cloud

Dataflow is used here to ingest data from BQ to Elastic Cloud. It’s a serverless, fast, and cost-effective stream and batch data processing service. Dataflow provides an Elasticsearch Flex Template which can be easily configured to create the streaming pipeline. This blog from Elastic shows an example on how to configure the template.

Cloud Orchestration from Mainframe

It’s possible to load both BigQuery and Elastic Cloud entirely from a mainframe job, with no need for an external job scheduler.

To launch the Dataflow flex template directly, you can invoke the gcloud dataflow flex-template run command in a z/OS batch job step.

If you require additional actions beyond simply launching the template, you can instead invoke the gcloud pubsub topics publish command in a batch job step after your BigQuery ELT steps are completed, using the –attribute option to include your BigQuery table name and any other template parameters. The pubsub message can be used to trigger any additional actions within your cloud environment.

To take action in response to the pubsub message sent from your mainframe job, create a Cloud Build Pipeline with a pubsub trigger and include a Cloud Build Pipeline step that uses the gcloud builder to invoke gcloud dataflow flex-template run and launch the template using the parameters copied from the pubsub message. If you need to use a custom dataflow template rather than the public template, you can use the git builder to checkout your code followed by the maven builder to compile and launch a custom dataflow pipeline. Additional pipeline steps can be added for any other actions you require.

The pubsub messages sent from your batch job can also be used to trigger a Cloud Run service or a GKE service via Eventarc and may also be consumed directly by a Dataflow pipeline or any other application.

Mainframe Capacity Planning

CPU consumption is a major factor in mainframe workload cost. In the basic architecture design above, the Mainframe Connector runs on the JVM and runs on zIIP processor. Relative to simply uploading data to cloud storage, ORC encoding consumes much more CPU time. When processing large amounts of data it’s possible to exhaust zIIP capacity and spill workloads onto GP processors. You may apply the following advanced architecture to reduce CPU consumption and avoid increased z/OS processing costs.

Remote Dataset Transcoding on Compute Engine VM

To reduce mainframe CPU consumption, ORC file transcoding can be delegated to a GCE instance. A gRPC service is included with the mainframe connector specifically for this purpose. Instructions for setup can be found in the mainframe connector documentation. Using remote ORC transcoding will significantly reduce CPU usage of the Mainframe Connector batch jobs and is recommended for all production level BigQuery workloads. Multiple instances of the gRPC service can be deployed behind a load balancer and shared by all Mainframe Connector batch jobs.

Transfer Data via FICON and Interconnect

Google Cloud technology partners offer products to enable transfer of mainframe datasets via FICON and 10G ethernet to Cloud Storage. Obtaining a hardware FICON appliance and Interconnect is a practical requirement for workloads that transfer in excess of 500GB daily. This architecture is ideal for integration of z/OS and Google Cloud because it largely eliminates data transfer related CPU utilization concerns.

We really appreciate Jason Mar from Google Cloud who provided rich context and technical guidance regarding the Mainframe Connector, and Eric Lowry from Elastic for his suggestions and recommendations, and the Google Cloud and Elastic team members who contributed to this collaboration.

As a network of social businesses, NTUC Enterprise is on a mission to harness the capabilities of its multiple units to meet pressing social needs in areas like healthcare, childcare, daily essentials, cooked food, and financial services. Serving over two million customers annually, we seek to enable and empower everyone in Singapore to live better and more meaningful lives.

With so many lines of business, each running on different computing architectures, we found ourselves struggling to integrate data across our enterprise ecosystem and enable internal stakeholders to access the data. We deemed this essential to our mission of empowering our staff to collaborate on digital enterprise transformation in ways that enable tailor-made solutions for customers. 

The central issue was that our five main business lines, including retail, health, food, supply chain, and finance, were operating on different combinations of Google Cloud, on-premises, and Amazon Web Services (AWS) infrastructure. The complex setup drove us to create a unified data portal that would integrate data from across our ecosystem, so business units could create inter-platform data solutions and analytics, and democratize data access for more than 1,400 NTUC data citizens. In essence, we sought to create a one-stop platform where internal stakeholders can easily access any assets they require from over 25,000 BigQuery tables and more than 10,000 Looker Studio dashboards.

Here is a step-by-step summary of how we deployed DataHub, an open-source metadata platform alongside Google Cloud solutions to establish a unified Data Portal that allows seamless access for NTUC employees across business lines, while enabling secure data ingestion and robust data quality.

DataHub’s built-in data discovery function provides basic functionality to locate specific data assets from BigQuery tables and Looker Studio dashboards for storage on DataHub. However, we needed a more seamless way to ingest the metadata of all data assets automatically and systematically.

We therefore carried out customizations and enhancements on Cloud Composer, a fully managed workflow orchestration service built on Apache Airflow, and Google Kubernetes Engine (GKE) Autopilot, which helps us scale out easily and efficiently based on our dynamic needs.

Next, we built data lineage, which enables the end-to-end flow of data across our tech stack, drawing data from Cloud SQL into Cloud Storage, then channeling the data back through BigQuery into Looker Studio dashboards for easy visibility. This was instrumental in enabling users across NTUC’s business lines to access data securely and intuitively on Looker Studio. 

Having set up the basic platform architecture, our next task was to enable secure data ingestion. Sensitive data needed to be encrypted and stored in Cloud Storage before populating BigQuery tables. The system needed to be flexible enough to securely ingest data in a multi-cloud environment, including Google Cloud, AWS, and our on-premises infrastructure.

Our solution was to build an in-house framework to fit requirements of Python and YML, as well as GKE and Cloud Composer. We created the equivalent of a Collibra data management platform to suit NTUC’s data flow (from Cloud Storage to BigQuery). The system also needed to conform to NTUC data principles, which are as follows: 

All data in our Cloud Storage data lake must be stored in a compressed form like Avro, a data security service

Sensitive columns must be hashed using Secure Hash Algorithm 256-bit (SHA-256)

The solution must be flexible for customization depending on needs

Connection must be made by username and password

Connection must be made with certificates (public key and private key), including override functions in code

Connections require one logical table from hundreds of physical tables (MSSQL sharding tables)

Our next task for the Data Portal was creating an automated Data Quality Control service to enable us to check data in real-time whenever a BigQuery table is updated or modified. This liberates our data engineers, who were previously building BigQuery tables by manually monitoring hundreds of table columns for changes or anomalies. This was a task that used to take an entire day, but is now reduced to just five minutes. We enable seamless data quality in the following way: 

Activity in BigQuery tables is automatically written into Cloud Logging, a fully managed, real-time log management service with storage, search, analysis, and alerts 

The logging service can then filter out events from BigQuery into Pub/Sub for datastreams that are then channeled into Looker Studio, where users can easily access the specific data they need

In addition, the Data Quality Control service sends notifications to users whenever someone updates BigQuery tables incorrectly or against set rules, whether that is deleting, changing or adding data to columns. This enables automated data discovery, without engineers needing to go intoBigQuery to look up tables

These steps enable NTUC to create a flexible, dynamic, and user-friendly Data Portal that democratizes data access across business lines for more than 1,400 platform users, opening up vast potential for creative collaboration and digital solution development. In the future, we plan to look at how we can integrate even more data services into the Data Portal, and leverage Google Cloud to help develop more in-house solutions.

Today, the American Legion Department of California announced a new partnership with Google Cloud to connect 500 California veterans with no-cost access to Google Cloud Skills Boost. This new collaborative effort will help train, upskill, and provide credentials to veterans in cloud computing skills – creating pathways to technical, in-demand jobs. 

As companies, nonprofits, and governments increasingly leverage cloud computing technology to strengthen their organizations, skilled cloud talent is in high demand nationally, with particularly high demand in California. According to Lightcast, there were 27,180 unique cloud roles open in the U.S. as of October 2022. Cloud skills are the most in-demand skill set by IT departments: more than one-third of tech learners said professionals with cloud skills are the most challenging to find.

Veterans can tap into the demand for cloud talent by building skills on Google Cloud Skills Boost. Google Cloud Skills Boost combines award-winning learning experiences with the ability to earn credentials to validate new skills. Learners can prepare for certifications like Cloud Digital Leader, which assesses general cloud knowledge, and Associate Cloud Engineer, which assesses the ability to implement cloud solutions based on technical requirements. Among Google Cloud certified individuals, an estimated 70% who applied for a new job received at least one offer (2020 Google Cloud Certification Impact Report).

“As service members we face an immediate four-year educational disadvantage when we join the military; and with this program we will be filling the gap and helping to ensure that veterans have a fair chance in this industry,” shared Roberto Arnold, Charles P. Rowe Post 30 commander of American Legion. 

“Google is proud to offer no-cost access to Google Cloud Skills Boost for California veterans in partnership with American Legion,” said Alice Kamens, Google Cloud Workforce Development program manager. “Through this partnership, we’re advancing our work to ensure everyone has access to the technical training needed to pursue cloud computing careers.”

Those interested in learning more about the program or applying for a Google Cloud Skills Boost scholarship can learn more here. Scholarship recipients need access to a computer, and the internet. 

In September, we launched the Google Maps Platform hackathon on Devpost. We were blown away by the creativity of the projects submitted. Over 2,600 participants spanning 100+ countries joined to build something they’re passionate about and created unique experiences using Google Maps Platform. Thanks to all the participants and judges that made this hackathon one to remember. 

And without further ado, here are the winning projects.

Best Map Customization: BeeDrone Delivery Simulator

The scene is New York city, where a ground station (in the form of a space needle) houses several “bee” drones for on-demand deployment. The demo uses a custom vector map style and 3D models of drones to simulate a package pickup-and-delivery use case. Drones leave the ground station, pick up and deliver packages, and return to recharge when needed. The drones follow a kinematic model that lets them fly at a specific altitude above ground level, and orient themselves correctly when flying. The demo shows a floating dashboard above the map to keep track of the on-time delivery percentage (the aim is to be at 100%!) and the amount of demand for pickups and deliveries.

Best Data Visualization: Kicky – Soccer Analytics on a Map

Kicky uses freely accessible soccer analytics data sources. It visualizes soccer moments on Google maps in real-world coordinates. It enables users to analyze data using some machine learning and statistics libraries in the background. Users can see distinct player movements on the 3D map. hey can even see a snapshot of selected events, including player positions and the direction of the shot.

Best Mobile App: DayTrip

DayTrip uses the Places API for recommendations and reviews, and allows the users to choose four different activities (breakfast, activity, lunch, and final attraction) around their location. After choosing, users can see a short summary of their expected day trip and start navigating via Google Maps. It gives the user a single place that gathers the best recommendations and plans the best, time saving day trip.

Most Creative User Experience: Save the Rain! 

The app looks up how much rain you get in a year, then calculates how much water you could save in a year based on the size of your roof. You can also toggle on additional layers of information including: mean annual precipitation worldwide, mean annual precipitation precipitation change, forecasting for the year 2100, US Weekly Drought Monitor (real-time), and 24 Rain Forecast from NOAA (real-time).

Honorable Mentions

Crop Progress Tracker

Recycling Trashcan AR Map

TaxiFlex

Mapple – Neighborhood Delivered

Congratulations to the winners and thank you to all the participants! Check out all the amazing projects submitted. We can’t wait to see you at the next hackathon. 

For more information on Google Maps Platform, visit our website.

Every day, teams across Google Cloud come together to help address complex and pressing compliance, risk, and privacy requirements that enable customers to accelerate their digital transformation and innovate faster. Let’s take a look at some of our top trust and compliance-related efforts from the last few months: 

Combining data processing terms. In September, we updated and consolidated our data processing terms for Google Cloud, Google Workspace (including Workspace for Education), and Cloud Identity (when purchased separately). The combined terms are now one Cloud Data Processing Addendum (the “CDPA”). The updated terms maintain the previous benefits while strengthening our data processing commitments. We’ve also incorporated the new international data transfer addendum issued by the U.K. Information Commissioner. Learn more in our updated whitepaper. 

Seven Trusted Cloud announcements from Google Cloud Next ‘22. In October, we hosted Google Cloud Next ‘22, a global, 24-hour livestream and in-person event. At Next ‘22, we had seven new Trusted Cloud announcements, including Confidential Space, the next solution in our Confidential Computing portfolio. Confidential Space allows multiple parties to securely collaborate on joint tasks such as data analysis and machine learning (ML) model training. Read all of our Trusted Cloud announcements here. 

Advancing Digital Sovereignty. Also announced at Next ‘22 was our updated portfolio of Sovereign Solutions that help customers address their digital sovereignty concerns. These Solutions align with our 2021 “Cloud. On Europe’s Terms” announcement, and include Sovereign Controls, Supervised Cloud, and Hosted Cloud options. 

Expanding our cloud regions. In recent months, we announced five new Cloud regions, including Swedenand Norway, to support our customers’ digital transformation and sustainability efforts. These new regions will help our customers maintain security, data residency, and compliance standards, including region-specific data storage requirements. Keep up to date with our latest Cloud locations here. 

Validating records retention in Japan. In January 2022, the Japanese government’s revisions of the Electronic Records Retention Law came into effect. To address these changes, the Japan Image and Information Management Association (JIIMA) created a certification for software which complies with the new legal requirements. JIIMA has certified Google Workspace as compliant with this certification that will help our customers meet their regulatory requirements for digital record-keeping in Japan. 

Meeting additional requirements. In other compliance news, our latest U.S. Public Sector compliance authorization, StateRAMP, enables us to support government customers with enhanced data residency and support capabilities via Assured Workloads. In Germany, we updated our Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin) offerings, and with support from our trust partner, microfin, we mapped our compliance posture to German requirements for IT (BAIT/VAIT) in banking and insurance. 

The completed Korea Financial Security Institute (K-FSI) audit. To help support a group of leading South Korean financial institution customers, we worked with auditors from K-FSI to complete an audit based on their guidelines. Our customers joined this “pooled audit,” which is an excellent example of industry collaboration that helped increase their confidence in Google Cloud.  

Hot off the press: two new papers. For telecommunications organizations in the U.S., our paper Insights into the U.S. Telecommunications Industry addresses key industry trends, regulatory themes, influences, and how companies can use Google Cloud services to tackle their top-of-mind challenges. In support of the European Commission’s digital transformation vision, Google published a paper and blog outlining a set of security recommendations. The recommendations include driving resilience through “open security” and prioritizing strong encryption. 

Using our compliance developments

Many Google Cloud customers already use our trust resources to facilitate internal and external conversations with their key customers, business partners, and regulators. Visit our Compliance Resource Center or continue the conversation with our sales team.

When running Apache Hadoop and Spark, it is important to tune the configs, perform cluster planning, and right-size compute. Thorough benchmarking is required to make sure the utilization and performance are optimized. In Dataproc, you can run Spark jobs in a semi-long-running cluster or ephemeral Cloud Dataproc on Google Compute Engine (DPGCE) cluster or via Dataproc Serverless Spark. Dataproc Serverless for Spark runs a workload on an ephemeral cluster. An ephemeral cluster means the cluster’s lifecycle is tied to the job. A cluster is started, used to run the job, and then destroyed after completion. Ephemeral clusters are easier to configure, since they run a single workload or a few workloads in a sequence. You can leverage Dataproc Workflow Templates to orchestrate this. Ephemeral clusters can be sized to match the job’s requirements. This job-scoped cluster model is effective for batch processing. You can create an ephemeral cluster and configure it to run specific Hive workloads, Apache Pig scripts, Presto queries, etc., and then delete the cluster when the job is completed.  

Ephemeral clusters have some compelling advantages:

They reduce unnecessary storage and service costs from idle clusters and worker machines.

Every set of jobs runs on a job-scoped ephemeral cluster with job-scoped cluster specifications, image version, and operating system. 

Since each job gets a dedicated cluster, the performance of one job does not impact other jobs.

Persistent History Server (PHS)

The challenge with ephemeral clusters and Dataproc Serverless for Spark is that you will lose the application logs when the cluster machines are deleted after the job. Persistent History Server (PHS) enables access to the completed Hadoop and Spark application details for the jobs executed on different ephemeral clusters or serverless Spark. It can list running and completed applications. PHS keeps the history (event logs) of all completed applications and its runtime information in the GCS bucket, and it allows you to review metrics and monitor the application at a later time. PHS is nothing but a standalone cluster. It reads the Spark events from GCS, then parses and presents application details, scheduler stages, and task level details, as well as environment and executor information, in the Spark UI. These metrics are helpful for improving the performance of the application. Both the application event logs and the YARN container logs of the ephemeral clusters are collected in the GCS bucket. These log files are important for engaging Google Cloud Technical Support to troubleshoot and explore. If PHS is not set up, you have to re-run the workload, which adds to support cycle time. If you have set up PHS, you can provide the logs directly to Technical Support.

The following diagram depicts the flow of events logged from ephemeral clusters to the PHS server:

In this blog, we will focus on Dataproc PHS best practices. To set up PHS to access web interfaces of MapReduce and Spark job history files, please refer to Dataproc documentation.

PHS Best Practices

Cluster Planning and Maintenance

It’s common to have a single PHS for a given GCP project. If needed, you can create two or more PHSs pointing to different GCS buckets in a project. This allows you to isolate and monitor specific business applications that run multiple ephemeral jobs and require a dedicated PHS. 

For disaster recovery, you can quickly spin up a new PHS in another region in the event of a zonal or regional failure. 

If you require High Availability (HA), you can spin up two or more PHS instances across zones or regions. All instances can be backed by the dual-regional or multi-regional GCS bucket.

You can run PHS on a single-node Dataproc cluster, as it is not running large-scale parallel processing jobs. For the PHS machine type:

N2 are the most cost-effective and performant machines for Dataproc. We also recommend 500-1000GB pd-standard disks.

For <1000 apps and if there are apps with 50K-100K tasks, we suggest n2-highmem-8.

For <1000 apps and there are apps with 50K-100K tasks, we suggest n2-highmem-8.

For >10000, we suggest n2-highmem16.

We recommend you benchmark with your Spark applications in the testing environment before configuring PHS in production. Once in production, we recommend monitoring your GCE backed PHS instance for memory and CPU utilization and tweaking machine shape as required.

In the event of significant performance degradation within the Spark UI due to a large amount of applications or large jobs generating large event logs, you can recreate the PHS with increased machine size with higher memory. 

As Dataproc releases new sub-minor versions on a bi-weekly cadence or greater, we recommend recreating your PHS instance so it has access to the latest Dataproc binaries and OS security patches. 

As PHS services (e.g. Spark UI, MapReduce History Server) are backwards compatible, it’s suggested to create a Dataproc 2.0+ based PHS cluster for all instances.

Logs Storage 

Configure spark:spark.history.fs.logDirectory to specify where to store event log history written by ephemeral clusters or serverless Spark. You need to create the GCS bucket in advance. 

Event logs are critical for PHS servers. As the event logs are stored in a GCS bucket, it is recommended to use a multi-Region GCS bucket for high availability. Objects inside the multi-region bucket are stored redundantly in at least two separate geographic places separated by at least 100 miles.

Configuration

PHS is stateless and it constructs the Spark UI of the applications by reading the application’s event logs from the GCS bucket. SPARK_DAEMON_MEMORY is the memory to allocate to the history server and has a default of 3840m. If too many users are trying to access the Spark UI and access job application details, or if there are long-running Spark jobs (iterated through several stages with 50K or 100K tasks), the heap size is probably too small. Since there is no way to limit the number of tasks stored on the heap, try increasing the heap size to 8g or 16g until you find a number that works for your scenario.

If you’ve increased the heap size and still see performance issues, you can configure spark.history.retainedApplications and lower the number of retained applications in the PHS.   

Configure mapred:mapreduce.jobhistory.read-only.dir-pattern to access MapReduce job history logs written by ephemeral clusters. 

By default, spark:spark.history.fs.gs.outputstream.type is set to BASIC. The job cluster will send data to GCS after job completion. Set this to FLUSHABLE_COMPOSITE to copy data to GCS at regular intervals while the job is running. 

Configure spark:spark.history.fs.gs.outputstream.sync.min.interval.ms to control the frequency at which the job cluster transfers data to GCS. 

To enable the executor logs in PHS, specify the custom Spark executor log URL for supporting external log service. Configure the following properties:

Lifecycle Management of Logs

Using GCS Object Lifecycle Management, configure a 30d lifecycle policy to periodically clean up the MapReduce job history logs and Spark event logs from the GCS bucket. This will improve the performance of the PHS UI considerably.

Note: Before doing the cleanup, you can  back up the logs to a separate GCS bucket for long-term storage. 

PHS Setup Sample Codes

The following code block creates a Persistent History server with the best practices suggested above.

The following code block creates an ephemeral cluster that logs events to the GCS bucket.

With the Persistent History server, you can monitor and analyze all completed applications. You will also be able to use the logs and metrics to optimize performance and to troubleshoot issues related to strangled tasks, scheduler delays, and out of memory errors. 

Additional Resources

Dataproc PHS Documentation

PHS with Terraform

Spark Monitoring and Instrumentation

Run a Spark Workload

Spark History Web Interface

Monitoring Compute Engine VMs

Today, we published a new Google research report on software supply chain security because we’ve seen a sharp rise in software supply chain attacks across almost every sector —and expect these trends to continue for the foreseeable future. We urge all organizations to act now to improve their software supply chain security.

Among the report’s conclusions, there are two key findings we want to highlight. First, the lessons we’ve learned from various security events call for a more holistic approach to strengthen defenses against software supply chain attacks. Second, we have worked with the security  community to develop and deploy a common Supply-chain Levels for Software Artifacts (SLSA) framework that can mitigate threats across the entire software supply chain ecosystem. These frameworks can help organizations securely build and verify the integrity of software. You can find more information on the report’s conclusions here. 

We know that modern day software supply chains continue to grow deeper, wider, and more complex. That complexity can make it challenging for customers to even know where to begin analyzing their supply chains for security issues. Our research shows that organizations must deal with these same complex issues regardless of which environments they operate in.

At Google Cloud, we’re deeply committed to working with our customers to help ensure that they have the support they need to evaluate their security posture, resiliency, and hygiene. Below, we suggest five steps to protect software across processes and systems, and tap into relevant Google Cloud products and services. These recommendations can enable customers to benefit from Google’s extensive security experience and reduce their need to develop, maintain, and operate complex processes to secure their open source dependencies.

Implementing best practices with Google Cloud

Customers who are interested in improving their software supply chain security can take immediate steps to implement best practices.  

1. Enhance your existing Google Cloud security features with the Google Cloud security foundation guide. The guide can help you weigh important considerations including organizational structure, authentication and authorization, resource hierarchy, networking, logging, and detective controls. You can further engage Mandiant experts to assess your readiness. 

You can also view centralized information about vulnerabilities and possible risks using Google Cloud services like Security Command Center, and get information about your service usage with Recommender, including recommendations that can help you to reduce risk. For example, you can identify IAM principals with excess permissions or unattended Google Cloud projects. You can also find additional resources from the Google Cybersecurity Action Team (GCAT), our premier security advisory team, here.

2. Explore fast software delivery and reliable and secure software with Google Cloud’s DevOps capabilities. You also should review foundational practices for designing, developing, and testing code that apply to most programming languages. 

We strongly recommend you evaluate how you distribute software and the terms of software licenses in all of your dependencies. For more information on Google’s approach to helping organizations address vulnerabilities in open source software, see Appendix B in the research report. 

3. Document the policies for your organization and incorporate validation of policies into your development, build, and deployment processes as you implement best practices. For example, your organization’s policies might include criteria for deployment that you implement with Binary Authorization. GCAT has published additional information on security policies and other cloud security transformation tips for CISOs here. 

You can also explore Minimum Viable Secure Product, a security checklist of controls to establish a baseline security posture for a product. You can use the checklist to establish your minimum security control requirements and to evaluate software by third-party vendors.

Tapping into new Google product and service offerings

At Google Cloud, we continue to focus on delivering new and innovative security capabilities to help customers address the latest security threats. From the attack on SolarWinds to the community response to open source vulnerabilities such as Log4j, we’re seeing a spike in demand from customers on what we can do to help them manage software supply chain risk. We’ve made several recent announcements on that front that can help customers get started with Google Cloud today. 

4. Use Google Cloud’s Software Delivery Shield. It provides a fully managed software supply chain security solution that offers a modular set of capabilities to help equip developers, DevOps, and security teams with the tools they need to build secure cloud applications. Software Delivery Shield spans across a family of Google Cloud services from developer tooling to runtimes including GKE, Cloud Code, Cloud Build, Cloud Deploy, Artifact Registry, and Binary Authorization. To learn more about Software Delivery Shield, check out the solution page, or watch this Google Cloud Next session to get a quick overview of Software Delivery Shield.

5. Enable our Assured Open Source Software (OSS) service, which can help enterprise and public sector open source software users to easily incorporate the same OSS packages that we use at Google into their own developer workflows. Packages curated by the Assured OSS service: 

are regularly scanned, analyzed, and fuzz-tested for vulnerabilities; 

have corresponding enriched metadata incorporating Container/Artifact Analysis data; 

are built with Cloud Build including evidence of verifiable Supply chain Levels for Software Artifacts (SLSA)-compliance;

are verifiably signed by Google; 

and are distributed from an Artifact Registry that is secured and protected by Google. 

If you are interested in learning more about software supply chain security in general, please contact usor reach out to your sales representative to schedule a software supply chain security workshop.