Is Cloud an option for Canadian Healthcare healthcare and medical research organizations?

Yes, Canadian healthcare and medical research organizations are moving to the cloud. The cloud market is expected to grow in Canada significantly through 2027.

There are several reasons why Canadian healthcare and medical research organizations are moving to the cloud. 

Reduce costs by eliminating the need to invest in and maintain on-premises infrastructure. 

Enable the healthcare research community to drive their research more expediently to clinical outcomes

Improve patient satisfaction by making it easier for patients to access their health information and communicate with their providers.

Improve the quality of care by providing access to patient data and records from anywhere in the country. 

Overall, the transition to the cloud is a positive development for Canadian healthcare and medical research organizations. 

Canadian healthcare providers face many challenges before they can move to the cloud, such as addressing security and privacy concerns, data sovereignty issues, and ensuring interoperability. To help them overcome these challenges, it is important to provide Healthcare Data Custodians, Infrastructure Architects, and Research Leads with clear guidance on how the cloud can align with Canadian Healthcare Regulations. This will allow them to have a practical understanding of what is required to enhance their cloud journey and facilitate a smoother transition to the cloud.

iSecurity and MD+A Health are actively assisting Canadian healthcare and medical research organizations in comprehending the risks and exploring pathways to embrace the cloud. Through extensive research and analysis, iSecurity and MD+A Health have evaluated Google Cloud as a suitable platform for healthcare. Their diligent efforts have resulted in the production of comprehensive documents that detail their findings via a Threat Risk Assessment (TRA) and a Privacy Impact Report (PIA).

Why a Threat Risk Assessment? 

A threat risk assessment is a process of identifying and evaluating threats to an organization and then determining the likelihood and impact of those threats. The goal of a threat risk assessment is to identify the most serious threats and develop mitigation strategies to reduce the likelihood and impact of those threats.

A threat risk assessment typically involves the following steps:

Identify threats: The first step is to identify all potential threats to the organization. This can be done by brainstorming, interviewing experts, or reviewing historical data.

Evaluate threats: Once the threats have been identified, they need to be evaluated in terms of their likelihood and impact. The likelihood of a threat is the probability that it will occur, while the impact of a threat is the severity of the consequences if it does occur.

Prioritize threats: The threats need to be prioritized based on their likelihood and impact. The most serious threats should be addressed first.

Develop mitigation strategies: Once the threats have been prioritized, mitigation strategies need to be developed to reduce the likelihood and impact of those threats. Mitigation strategies can include things like implementing security controls, training employees, and developing contingency plans.

Implement mitigation strategies: The mitigation strategies need to be implemented and tested to ensure that they are effective.

Monitor and review: The threat risk assessment should be monitored and reviewed regularly to ensure that it is still effective.

Why a Privacy Impact Assessment?

A Privacy Impact Assessment (PIA) is a process that organizations use to identify and assess the privacy risks associated with a new or changed information technology (IT) system or project. The goal of a PIA is to help organizations protect the privacy of individuals whose personal information is collected, used, or disclosed by the IT system or project.

PIAs typically include the following steps:

Identifying the purpose of the IT system or project and the types of personal information that will be collected, used, or disclosed.

Identifying the privacy risks associated with the IT system or project.

Assessing the likelihood and severity of the risks.

Developing and implementing controls to mitigate the risks.

Monitoring the effectiveness of the controls.

PIAs are an important tool for organizations to help them follow privacy laws and regulations. They can also help organizations build trust with their patients, employees and the research community by demonstrating their commitment to protecting privacy.

The benefits of conducting a PIA:

Helps organizations identify and assess privacy risks

Helps organizations develop and implement controls to mitigate privacy risks

Helps organizations comply with privacy laws and regulations

Helps organizations build trust with customers and employees

Why Google Cloud?

Google Cloud is committed to providing Canadian healthcare organisations with an environment to expand both their clinical and research environments. Google Cloud has invested significant resources into building out a cloud environment based on best practices coming from Google’s experience running some of the world’s largest platforms.  

Some highlights include:

Built-in security features that help protect your data and applications from unauthorized access, use, disclosure, disruption, modification, or destruction.

A comprehensive security management platform that helps you assess, prioritize, and address security risks across your organization.

A team of security experts who can help you design, implement, and manage your security solutions.

A wide range of security training and resources to help you learn about and stay up-to-date on the latest security threats and best practices.

In today’s ecommerce marketplace, consumers want customized experiences, but the costs of acquisition, personalization, fulfillment, and operations are exploding. Data is one way for ecommerce and retail providers to delight consumers cost-effectively. Unfortunately, this valuable data is often spread across marketing channels, in-house IT systems, third-party fulfillment systems, and bespoke operational systems.

Inspired by Google’s BigTable, unified analytics provider Isima’s flagship product, bi(OS), is a lean modern data stack that accelerates analytics outcomes for organizations by onboarding, processing, consuming, and operating on data using a self-service UI (Clicks), Python, and SQL. Together, Google Cloud and Isima offer a three-legged data stack that encourages agility, innovation, and efficiency for builders. In this blog post, we describe how Isima’s bi(OS) uses Google Compute Engine instances and Local SSD to deliver a competitive offering for ecommerce and retail.

bi(OS) as the differentiating component of the ecommerce data stack

bi(OS) acts as a data hub between online transaction processing (OLTP), online analytical processing (OLAP), and third-party software-as-a-service (SaaS) systems. bi(OS) includes no-code connectors to ingest data from OLTP systems (Cloud SQL), OLAP systems (Cloud Storage, BigQuery), and third-party SaaS systems (for example, Google/Facebook Ads, Clevertap, and Google Tag Manager). Low latency microservices, near-real-time analytics jobs, and ad-hoc queries can interface with bi(OS) via SQL-friendly Python, Java, and JavaScript SDKs.

bi(OS) on Google Cloud

Isima’s benchmarking of ecommerce and retail use cases on Google Cloud covered various aspects of marketing technology (martech), personalization, fulfillment, and operations. The team set up a bi(OS) cluster utilizing  n2-highmem-16 instances with 16x375GB Local SSD that was three-way resilient across three zones (a,b, and c) in the us-east-4 region. Isima engineers collected metrics after the system ran for five-plus days with five nines reliability.  During this time, all writes and reads were done with QUORUM consistency across the three zones.

At 70%+ utilization of compute, memory, and storage, the system sustained ~5,500 ops/sec.

Further, bi(OS) includes a turbo mode that reduces tail latencies for customer-facing ML micro-services. The graph below shows the distribution of these latencies measured over four-plus hours with and without turbo mode.

While bi(OS) and bi(OS) Turbo are comparable until p99, bi(OS) Turbo brought down p99.9 latencies by 8X.

The Google Cloud Local SSD touch

While bi(OS) is a multi-cloud offering, Google Cloud offers the following unique I/O benefits: 

Fine-tuning of storage: Google Cloud is the only cloud provider within the “big three” that allows you to fine-tune the amount of ephemeral block storage — in Google Cloud’s case, Local SSD. This allows Isima to offer up to one year of raw data retention, which is 40% higher than other cloud providers.

Multiple small(er) locally attached SSDs: Google Cloud’s Local SSD comes in 375GiB partitions. These multiple, smaller, Local SSD partitions allow applications like bi(OS) to use the same storage type for both write-ahead logs and data.

Stable I/O latencies: bi(OS) can only meet its QoS guarantees if the underlying infrastructure operates smoothly. Even after filling the Local SSDs by over 70%, Cloud Storage delivered good read and write I/O latencies during benchmarking.

Conclusion

Isima chose Google Cloud for its ability to run a lean, cost-effective, innovative three-tier data stack for all ecommerce and retail use cases. Isima, as a cloud-first company, loves Google Cloud’s simplicity and cloud-first approach to deploying and managing resources. bi(OS) is in the process of being listed on the Google Cloud Marketplace. To learn more, read about bi(OS), and watch a two-minute video about how ecommerce pioneers use bi(OS).

1. Each select retrieved ~10 rows.  In other words this system processed ~20K+ rows/sec.
*As measured in number of rows read + written per second.

At Bank Jago, we are on a mission to help millions of Indonesians get one step closer to their dreams, by delivering innovative money management solutions through a life-centric digital ecosystem. Founded by serial innovators with proven track records in micro-lending and digital banking, we strive to become Indonesia’s pioneering fin-tech solution through the power of Artificial Intelligence (AI)-enabled analytics. Our tech-driven approach to empowerment has allowed us to gain more than four million customers in less than 18 months.

To seamlessly onboard new customers, we needed to deploy an Optical Character Recognition (OCR) system to capture information from Indonesian identity cards in the online account setup process. It required a response time of between one to three seconds to make identity verification as fast and efficient as possible. That’s because reducing wait time in the onboarding process is critical to preventing customers from abandoning the onboarding process. 

Our original approach was to deploy CPUs for the processing power required to run our proprietary OCR system built on Kubeflow as our open-source machine learning (ML) platform on Google Kubernetes Engine (GKE). However, we discovered that there was a limit to how much performance we could squeeze from CPUs, especially since the compute-optimized (C2) CPUs were not available in the region. 

To overcome the issue, the first option we considered was to deploy libraries supporting AVX 512 instructions to enhance the CPU’s performance. This boost in processing power enables CPUs to crunch data faster, allowing users to run compression algorithms at higher speeds. The downside was that we had to compile the AVX 512 compatible library ourselves, something that became a burden on our data science team, pulling them away from the main task of developing ML applications.

Employing more powerful GPUs to process our ML pipelines seemed the only way forward. However, we faced the age-old dilemma of cost versus performance, with the GPU option seeming prohibitively expensive. We worked with the Google Cloud team to find a solution — a GKE feature called GPU time-sharing that enables multiple containers to share a single physical GPU attached to a node. Through GPU time-sharing in GKE, we could use GPUs more efficiently and save on running costs. The Google Cloud team also suggested deployment, whenever possible, of GPU spot instances, which offer unused GPU capacity in GKE at a discounted rate. 

Here is a step-by-step rundown of how we worked with the Google Cloud team to enable GPU time sharing for our OCR needs:  

The first prerequisite for GPU time-sharing is to deploy a Kubernetes cluster in a GKE Version 1.24 and above. Otherwise, GPU time-sharing will not be supported.

The next step is to set up specific flags to enable GPU time-sharing in the gcloud command line tool. 

In the same Google Cloud instance, the gcloud command line tool can also be used to enable cost-efficient spot instances. 

Once GPU time-sharing and spot instances have been enabled, the next step is to install NVIDIA drivers to optimize GPU performance.

It is then advisable to run a sample workload to ascertain the GPUs are enabled and working before actual deployment.

By using a combination of spot instances and GPU time-sharing, we were surprised to find that we achieved higher performance with GPUs compared to our older CPU model at nearly half the cost. Compared to an OCR response time of four to six seconds with CPUs, we were easily able to bring the time down to our required goal of one to three seconds. In essence, we got the power and performance of two GPUs for the price of one.

The speed of GPU processing has also been a significant time saver for our data science team. Beyond enabling them to focus on core tasks, this has liberated them to experiment more with ML projects. Shortening the cycles of ML pipelines from days on CPUs to hours on GPUs essentially enables us to “fail fast” and launch new experiments. We are confident that we won’t be wasting too much time or resources if things don’t work out. 

The creative deployment of GPUs in our ML modeling, and working with the Google Cloud team, has opened new opportunities to enhance our digital banking solutions, in particular in the areas of data protection and fraud detection. Beyond OCR, we intend to use AI solutions by Google Cloud such as Natural Language AI for sentiment analysis, as well SHAP methodologies for model explainability in our credit risk operations.

In the long run, this opens up possibilities such as having data scientists share a GPU while working on a Jupyter Notebook. These will further leverage the high-level performance of GPU processing enabled by the time-sharing and spot instance techniques we have developed with the help of the Google Cloud team.

Imagine a future in which every piece of content out there can be listened to in any language or voice, at the click of a button, with high-quality intonations and pacing.

ElevenLabs, the world’s leading voice AI research organization, is doing just this. Our mission is to make spoken content universally accessible in any language and voice. Our platform offers text-to-speech tools that can generate highly customizable voices that sound authentically human. Our services include the ability to create a clone of your own voice, selecting from a range of pre-programmed synthetic voices, or designing an entirely unique synthetic voices. These can then be used for everything from narrating long-form content like books or news, to giving voices to characters in video games.

A great example of this in action is AI Radio, an autonomous, streaming radio station created using our Prime Voice AI technology for a virtual DJ that can announce breaking news and weather along with introducing music, combined with the capabilities of Super Hi-Fi’s MagicStitch™ production service and ChatGPT.

In late February, we introduced Voice Design for creating unique artificial voices, including making adjustments for qualities such as gender, age, and accent. With our upcoming features, creators will be able to structure large texts, insert longer pauses, regenerate portions of audio to their liking, edit intonations, and assign parts to different speakers.

Our solutions are powered by our own deep learning model that was built to realistically render how people speak, with the results adjusting based on the context of the written material and accounting for implied emotions.

Spoken words to resonate with audiences worldwide

ElevenLabs was founded in early 2022. I met my co-founder, Piotr Dabkowski, in high school in Poland, where we both grew up. 

In Poland, movies frequently use a single-person voiceover. Even for scenes with many actors, audiences only hear one voice speaking all the lines, devoid of emotions or intonations. As you can imagine, it’s a pretty bad experience.

After college, Piotr went to work for Google and I worked with companies like BlackRock and Palantir. Our ongoing friendship, common experience with poor dubbing, and shared interest in new technologies led to the idea for ElevenLabs.

We set out to see how we could better analyze and convey a human voice for applications that go well beyond movie voiceovers. We felt there was an enormous opportunity to work with a broad set of content developers to generate spoken-word audio from texts, automate translations into multiple languages, and more.

We launched our beta product in January 2022, after a year of intense development and testing. From YouTube video creators to podcasters and book publishers, we’re already seeing incredible adoption. Independent authors or small publishers, for example, are now able to convert a printed book into an audiobook at a fraction of the cost and in a very short time.

Amplifying voice technologies with Google Cloud

We’ve tried all the major cloud services and we’ve been most impressed by Google Cloud. We’ve had hundreds of thousands of users on our platform so far in the first quarter of this year and feel comfortable that if we need to do 100 times or 1 million times what we do today, we can rely on the infrastructure to scale seamlessly.

We found that some other cloud providers just weren’t scaling as well. There were more hoops that we needed to jump through. With Google Cloud, it was easy. And that’s what you need when you’re growing fast.

The costs are excellent too. It helps that we enrolled in the Google for Startups Cloud Program and received $100,000 in Google Cloud credits for this year and another $100,000 for next year. That support is invaluable for startups like ours that need to experiment but might be tight on cash. The program helped us deploy our models and start serving customers, even before we started bringing in revenue.

We use a lot of Cloud GPUs to ensure high performance for our models.BigQuery is our enterprise data warehouse, enabling us to analyze our information holistically. Looker helps us with our business intelligence visualizations. We also use Google Analytics for our website traffic. 

The tools all integrate well so we can customize metrics for key indicators, such as how many characters of text each customer has used from their subscription quota and how many cloned voices they have saved. We can analyze behaviors on the macro level to understand how people use our products and which voices or vocal characteristics are most popular.

To support users that register for our services, we use Firebase Authentication. It was a very easy identity solution to connect to our application for secure, streamlined user sign-in and onboarding.

We also are big fans of using Google Workspace for all of its organizational and productivity benefits. The ecosystem is great and it’s simple for our team members to use.

ElevenLabs in the future

As we look to our future, I’m interested in exploring new use cases, such as for people who are blind or visually impaired. I was recentlyinterviewed by Jonathan Mosen on his Living Blindfully podcast where we discussed the possibilities for parents who are blind and want to read bedtime stories to their children in their own voice, and for transforming all written content into accessible content in an instant.

As AI technologies evolve, we think it’s important to balance innovations with appropriate safeguards. We’ve had safeguards in place from the start and have rolled-out additional safety features since launch, with lots more coming. 

We also think it’s important to educate the public on safe ways of using the technology, so that people better understand what is and isn’t appropriate. Of course we’ll adapt to new standards and regulations as they emerge, such as the Artificial Intelligence Act in the European Union. One of the things they are proposing is for all AI-generated content to be tagged as such. We strongly agree that should be the case and we are building tools that make this possible.

We are thankful to have a great team of employees, investors, partners, and advisors working with us. As we expand globally, we are happy to be aligned with Google Cloud.

Already today we support generating content in English, Spanish, French, German, Polish, Italian and Hindi – but keep an ear out! In the near future, our model will extend to additional 10 languages as we prepare to cover every language out there!

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

Join us as we build on the concept and use cases of Workload Identity Federation, showcasing the security benefits of “keyless authentication.” We will dive into how Workload Identity Federation can be used in the context of CI/CD pipelines and tools that are commonly found in enterprise environments. 

Workload Identity Federation can be integrated with external providers, such as Gitlab, GitHub actions, and Terraform Cloud. We will show how the tokens issued by the external providers can be mapped to various attributes and how it can be used to evaluate conditions to restrict which identities can authenticate. 

Prerequisites

A Google Cloud project

A GitHub repository

A GitHub Actions workflow

Terraform Cloud Account/ Workspace

Scenario 1:

Objective: Demonstrate how two separate GitHub repositories and their corresponding CI/CD pipelines use separate service accounts to access respective Google Cloud resources in the least-privileged access manner.

In this scenario, we will configure the workload identity pool to check for a custom attribute “repository” and validate that the corresponding service account is used for a given GitHub repository.

Instructions

1. Create a workload identity pool

It is recommended to create and manage workload identity pools from a single dedicated project as per best practices.

2. Create a workload identity pool provider 

Note the purpose of some arguments as explained below. 

a. The issuer-uri identifies GitHub as the identity provider and lets workload identity discover the OIDC metadata and JSON Web Key Set(JWKs) endpoints. Google Cloud uses these endpoints to validate tokens. 

b. GitHub issues a unique token for each workflow job. This token contains claims that describe the identity of the workflow. By using an Attribute mapping, we translate the claims in the token so that we can use them in principal/principalSet identifiers to grant access to service accounts, or to create an attribute condition. 

c. Attribute-condition: After mapping the attributes, we are asserting conditions for authentication. In this case, we only allow authentication requests from GitHub Actions workflows on any repository from the “sec-mik” GitHub organization. Either assertions or previously mapped attributes can be used to build the condition.

In this case, we are mapping the subject, repository, repository owner, and branch attributes from the claims in the token to GCP assertions. CEL (Common Expression Language) expression is used to extract the branch from the subject claim in the token and map it to the custom attribute “branch.” 

3. Create a service account for each repository and assign them appropriate IAM permissions

In this scenario, one repository/service account will be for an application team that will deploy a Hello World app to Cloud Run and one repository will be owned by the networking team.

As a result, the networking-sa service account will have “networking” related IAM permissions and the application team will have cloud run and associated IAM permissions.

4. Add IAM bindings for the workload pool 

We will utilize previously mapped attributes to create principals/principalSets, and then use them in IAM bindings to grant permissions to create short-lived credentials for the appropriate service account. In this case, we want to map the networking service account (“networking-sa”) with the networking GitHub repository that is encapsulated in the “repository” attribute.

In accordance with Google Cloud best practices, we recommend that you use the system-generated IDs such as repository_id, rather than repository name, because it is immutable and does not change between renames of entities such as renaming a repository.

Next, we will repeat the same process to associate the application team’s service account with the “application-repo” repository. In this case, we will map against a “subject” assertion that represents a specific repository name and branch, instead of a custom attribute. The “sub” claim in the JWT is comprehensive and predictable, as it is assembled from concatenated metadata, and is therefore the preferred mapping option in most cases.

Once the IAM bindings are applied, we will observe it like this from Google Cloud Console workload identity provider page.

5. Update the GitHub Actions workflow to use the workload identity pool to authenticate to Google Cloud.

a. The workload identity pool (WORKLOAD_IDENTITY_PROVIDER) is configured as “projects/987654321/locations/global/workloadIdentityPools/GitHub-actions-pool/providers/GitHub-actions-oidc” and service account email (SERVICE_ACCOUNT_EMAIL) as “application-sa@production-secmik.iam.gserviceaccount.com“. 

b. Once authenticated, our pipeline will run two jobs, which are oriented towards a networking and application deployment.

We run this GitHub action from the application-repo where the cloud run deployment succeeds, but the networking command (listing firewalls) fails.

6. Analyzing the Cloud Audit Logs

We need to enable IAM and STS Data Access Logs on the Audit Logsof all the projects that contain workload identity pools and service accounts used for workload identity federation; to detect any IAM violations and observe how token exchange works between GitHub and STS API as shown below. 

First, the `auth` action uses the STS API to retrieve a short lived federated access token. This token exchange may fail due to attribute condition mismatch or configuration error.

The following is a log of the token exchange.

Please observe the following parameters in the image:

protoPayload.authenticationInfo.principalSubject: The subject from the GitHub JWT token i.e. repo:sec-mik/networking-repo:ref:refs/heads/main

protoPayload.metadata.mapped_principal: The subject of the token, using IAM syntax to identify the principal i.e. principal://iam.googleapis.com/projects/987654321/locations/global/workloadIdentityPools/GitHub-actions-pool/subject/repo:sec-mik/networking-repo:ref:refs/heads/main

Next, the `auth` action uses the IAM Credentials API to obtain a short-lived credential for a service account. Failure may occur due to either the absence of the workloadIdentityUser IAM role or the incorrect principal attempting to create the token.

The following is a log of the creation of short-lived credentials for service accounts.

Please observe the following parameters in the image:

protoPayload.authenticationInfo.principalSubject: The subject of the federated token. i.e. principal://iam.googleapis.com/projects/987654321/locations/global/workloadIdentityPools/GitHub-action-pool/subject/repo:sec-mik/application-repo:ref:refs/heads/main

metadata.identityDelegateChain: The service account for which short-lived credentials are generated, such as example-app-sa@production-secmik.iam.gserviceaccount.com

Refer to log examples for more details.

Summary

To summarize, we saw how a GitHub repository was mapped as a principal to authenticate with Google Cloud using a GitHub OIDC token, which was subsequently exchanged for Google Cloud credentials. Once authenticated, the IAM bindings for the corresponding service account performed authorization checks and was granted access accordingly.

Scenario 2

Objective: Demonstrate how two Terraform Cloud workspaces can use two separate but corresponding service accounts for provisioning. 

In this scenario, we will explore another popular tool in the IaC space that is commonly used as an orchestrator, Hashicorp Terraform Cloud and see how workload identity federation can be used in a similar fashion. 

Below is an overview of what we will demonstrate.

Mappings (GitHub repo → Terraform Cloud Workspace → Google Cloud Service Account → Google Cloud project )

app-repo-dev → app-ws-dev → app-sa-dev → dev-secmik 

app-repo-prod → app-ws-prod → app-sa-prod → production-secmik

Instructions

1. Create a workload identity pool for dev and prod environment

In this scenario, we will create a separate workload identity pool to follow Google Cloud best practices which recommends having one-to-one mapping between external identity provider and workload identity pool to prevent subject collisions. The format of the subject is “organization:my-org:project:Default Project:workspace:my-workspace:run_phase:apply” and the reference to workspace will be different for dev and prod environments in our scenario.

2. Create a workload identity pool provider 

The next step is to configure the provider within the workload identity pool. Before this, it is important to review the various claims that are part of the Terraform JWT token. The few important claims from the JWT token are:

iss : The issuer of the token. 

terraform_full_workspace : The full workspace name.

terraform_project_id : The ID of the project that the workspace is associated with.

terraform_workspace_id : The ID of the workspace.

In the dev and prod workload identity pool configuration, the CEL condition will limit access to identity tokens from a specific workspace (corresponding to dev/ prod) within a Terraform Cloud organization and Project.

3. Create Google Cloud Service accounts

Create two service accounts that will be used by separate workspaces.

main workspace: “app-sa-prod” 

develop workspace: “app-sa-dev”

4. Add IAM bindings for the workload pool

Add IAM bindings for the workload pool providers that were created earlier. In this scenario, IAM verifies the workspace ID attribute before authorizing access to impersonate a service account. Workspace ID is immutable, meaning it cannot be changed, so if the workspace is deleted and recreated with the same name, no accidental access will be granted as per best practice from Google.

Once the IAM bindings are applied, we will observe it like this on Google Cloud Console’s workload identity provider page.

5. Terraform Cloud Configuration

We will specify the workload pool related configuration as a workspace variable as shown below. The variables can be classified as sensitive to prevent them from being viewed on the UI or in logs. For further information, please refer to HashiCorp’s public documentation.

TFC_GCP_PROVIDER_AUTH : Terraform Cloud will use dynamic credentials to authenticate to GCP.

TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL: The service account email address that Terraform Cloud will use to authenticate to Google Cloud.

TFC_GCP_WORKLOAD_PROVIDER_NAME: The canonical name of the workload identity provider. Format is : projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_NAME/providers/PROVIDER_NAME

If the credential exchange fails due to a condition check, one would see an error similar to one below:

If the token generation fails due to permissions, one would see an error similar to one below

A successful deployment would result in something like this:

6. Auditing Workload Identity Federation service accounts

To obtain an organization-wide view of all service accounts that have been provisioned to use Workload Identity Federation, follow these steps:

In the Policy Analyzer, select your organization.

Select the Workload Identity User role as a parameter.

Do not select any checkboxes.

Click Analyze.

The results will be all service accounts that have been provisioned to be used with Workload Identity Federation, including their attribute conditions.

Get started today

To get started with Workload Identity Federation with CI/CD pipelines, see Configure workload identity federation with deployment pipelines.

Turning images of damaged vehicles into accurate claims information is critical for insurance agencies. For that reason, KASIKORN Business-Technology Group (KBTG) Labs, a leading tech company in Thailand, is using Machine learning (ML) technology to help speed up this process, and ensure the accuracy of every claim.

KBTG worked with Muang Thai Insurance PCL (MTI), an insurance company in Thailand that provides fire, marine, and automobile insurance. When a vehicle is damaged in an accident the repair shop typically assesses the damage and submits a claim to the insurance company for processing. These claims are validated by the insurance company and then processed with intermediaries that validate the claims and submit photos as proof. 

Currently, this complete process of assessing the damage and evaluating the claim is completely manual, which slows down processing time and is expensive, because multiple partners and intermediaries are involved. To address this issue, KBTG Labs worked with Google Cloud to devise a solution to this problem. 

Building a solution architecture

The existing dataset used for claims processing had many data quality and quantity challenges, For example, some images might be taken from too far a distance or too close to identify the specific area of damage, or the descriptions may be incorrect.

To resolve this problem, Google Cloud proposed a solution architecture that leverages a cascaded model approach, including image quality assessment.

The inference pipeline depicted above is designed for scalability, high availability, and fault tolerance. A microservice for execution, model selection based on inputs and outputs of the intermediary models is served by Google Kubernetes Engine. The images are then stored on Cloud Storage and sent for prediction in a bucket, to be labeled later and consumed in continuous training. Cloud SQL stores the results for the inference and analytics. The Image Qualifier model uses Vertex AI to classify the input image into a good or bad image. This model type is intended to act as a filter for the image quality for the model inference and streamline the process for only executing the model if the image conforms to intended image quality standards.

Experimenting with different images

Google Cloud’s proposed architecture includes an Image Qualifier model that identifies which images are suitable for ML and which are not, as mentioned above. Later in the project, the team ran experiments by distinguishing “good” and “bad” images in this model. Through this analysis, the team found that distance was one of the most important factors in the images, and that using images with the appropriate Distance metric successfully improved metrics such as Precision and Recall of the ML model by about 30%. The table below shows the results from the experiments.

Note : 

Distance 1 = image does not show the whole car but many car parts are visible (i.e., more close-up than label 0, which is appropriate for images in which the entire vehicle is visible)

Distance 2 = images focuses only one main part in the middle of image (i.e., more close-up than label 1)

Distance 3 = images cannot be used to identify the car part because it’s shot from a very, very close-up distance

Improving model predictions with Explainable AI

To gain further explanatory power in image classification, the team used a Google Cloud product called Explainable AI. Explainable AI is a feature of Vertex AI that enables users to generate “feature attributions” or “feature importance” values for their model’s predictions. Feature attributions are an explainability method that show users how much each input feature contributes to their model’s predictions and to the model’s overall predictive power.

Below is a True Positive example of a break defect. In other words, it is an example where ML was able to correctly predict the image of an image of a break as a legitimate break. But as you can see in the images, the light is reflected in all examples. Explainable AI concentrates green dots on the areas where the car light is and the break defect is judged based on the lit up parts by Explainable AI.

Initially, the intended architecture considered damage classification separately from object detection. However, by providing insights that explained the model’s behavior and how to optimize its performance, these results showed that damage identification should be done with object detection in order to avoid the influence of objects in the background of the damage as much as possible.

Key learnings and improvements as next steps

Explainable AI revealed that the ML model focuses on not only the damage, but also the damaged object itself. From this result, we hypothesize that damage itself is more effective for object detection than learning via only a simple classification model.

Through this project, it was discovered that the distance at which an object is photographed is critical to detecting vehicle damage, and that it is important to eliminate the influence of objects in the background as much as possible. Based on these results, we would like to assess the results of the experiment, including the object detection process.

To learn more about what Vertex AI can do for your business, visit our product page, and to learn more about Explainable AI, click here.

Introduction

There are numerous scenarios where you might want to execute tasks in parallel. One common use case involves dividing data into batches, processing each batch in parallel, and combining the results in the end. This approach not only enhances the speed of the overall processing but it also allows for easier error detection in smaller tasks. 

On the other hand, setting up parallel tasks, monitoring them, handling errors in each task, and combining the results in the end is not trivial. Thankfully, Google Cloud’s Workflows can help. In this post, we will explore how you can use a parent workflow to set up and execute parallel child workflows.

Let’s get started!

Setting up the child workflow

To begin, let’s create a child workflow that will serve as the foundation for our parallel execution. 

The child workflow receives arguments from the parent workflow. In our example, we’ll use a simple iteration integer, but in real-world scenarios, it could represent a data chunk passed from the parent workflow.

The child workflow starts performing some work. In this example, it simply waits 10 seconds to simulate doing some work.

Afterwards, it returns the result or failure of the work. In this case, it just uses whether the iteration is even or odd to simulate success and failure:

You can see the full definition in the workflow-child.yaml file. Deploy the child workflow:

Setting up the parent workflow

Now, let’s create the parent workflow, which orchestrates the parallel execution of the child workflows. The parent workflow starts by initializing a map to store the results of successful and failed executions.

Next, the parent workflow employs a parallel for-loop to execute the child workflows with data chunks. In our example, we pass integers from 1 to 4 to simulate data. As each iteration is independent, we parallelize them using the parallel keyword. Note that each for-loop iteration spins up a thread and the for-loop is not waiting for a response before proceeding with the next iteration.

Within each iteration, the child workflow is executed with the iteration argument. The parent workflow then waits for the success or failure of the child workflow execution and captures the results/failures in the map.

Finally, the parent workflow returns the results/failures map.

You can see the full definition in workflow-parent.yaml file. Deploy the parent workflow:

Execute the workflow

With both workflows deployed, it’s time to execute the parent workflow:

As the parent workflow runs, you will observe four parallel executions of the child workflow. Each child workflow represents a different batch of data being processed simultaneously.

Since they all run in parallel, after 10 seconds, you should see 2 of them succeeded and 2 failed:

The parent workflow displays the results of the successful executions:

And the errors of the failed executions:

At this point, the parent workflow has the option to retry the failed executions or proceed with the successful ones, depending on your requirements.

Summary

By dividing data into batches and executing them simultaneously, we can enhance overall processing speed and detect failures more easily in each execution. In this post, we explored how to implement parallel execution of workflows and combining the results using Google Cloud Workflows. 

Check out the following video on more information on parallel steps in Workflows:

And as always, feel free to contact me on Twitter @meteatamel for any questions or feedback.

We’re excited to announce support for multiple Firestore databases in a Google Cloud project. You can now create multiple databases in a project to isolate customer data, microservices, or dev/test/staging environments. 

The multiple databases feature includes support for the following:

Firestore database CRUD management: Firestore now exposes a new API endpoint, Terraform resource, gcloud CLI command and Firebase CLI command to manage the lifecycle of one or more Firestore databases.

Conditional Identity Access Management control: You can apply different security policies to different Firestore databases with IAM Conditions or Firebase Security Rules.

Support for both Firestore database modes: You can create new databases, in the same project, in either Firestore Native Mode or Datastore Mode.

Support for all Firestore regions: You can create new databases, in the same project, in any supported Firestore region.

Billing: You can now track the cost of your databases separately through Cloud Billing and BigQuery. This makes it easier to both observe consumption and budget for each databases’ usage.

Cloud Monitoring: Firestore’s Cloud Monitoring Metrics and Stats are now aggregated at the database-level. 

Example walkthrough

Sound like a good candidate for your application? Let’s take an end-to-end example to see how it works. 

To create a Firestore database, you need to specify a database identifier, mode, and location. There are multiple ways to create a Firestore database. Here is an example of how to create a Firestore native database named reviews in the us-west1 location using the gcloud CLI:

Once your database is created, you can access it using the Firestore/Datastore console.

When you no longer need a Firestore database, you can delete the Firestore database by running.

That’s everything you need to do. Once a database is deleted, Firestore performs the data deletion on your behalf.

When using the Firestore client libraries with your newly created databases, specify the database name to get the appropriate Firestore database instance. If you don’t specify a database name, the client library falls back to the (default) database. Here’s an example for writing to a database named “test” with the Firebase Admin SDK for Java:

For examples in additional languages, see Accessing your database.

Next steps

For more information on how to set up and configure multiple databases on Firestore, check out the documentation.

Macy’s is well known for its high-end fashion worldwide. What is not as well known are the strong measures it takes to ensure its customers’ data remains secure. When Macy’s decided to move its infrastructure from on-premises to Google Cloud, it required the move be done without sacrificing security or degrading the user experience.

Migrating from on-premises to the cloud isn’t always a simple feat, especially when one of Macy’s key requirements was a managed solution that could secure their workloads’ internet access without impacting throughput and latency.

Implementing Cloud NAT at scale

Applying security safeguards without creating additional friction can be difficult, especially for Macy’s and its more than40 million active users. Macy’s needed a way to perform network address translation to ensure its clusters could create outbound connections to the internet without needing public IP addresses. Every use case led back to Cloud NAT.

Cloud NAT is a distributed, cloud-first service that provides network address translation for workloads without external IP addresses. These workloads typically access the internet to download updates or interact with SaaS services. 

Cloud NAT provided a means for Macy’s workloads to initiate outbound connections by translating the associated private IP addresses to one or more shared public IP addresses. This enabled Macy’s to reach the outside world while preventing the outside world from initiating a connection to it. What was even better for Macy’s was that Cloud NAT simplified the configuration and maintenance with no additional networking, forwarding, or routing configuration required.

Ensuring continued high performance

The software-defined networking that underpins Cloud NAT equipped Macy’s with more than just the benefit of simplified management. High availability is built into the product since it doesn’t depend on any virtual machine or physical gateway device, and it is available across different regions if a zone goes down. With an uptime SLA of 99.99%, Macy’s could feel confident that their operations would run without disruption.

Cloud NAT can also be configured to automatically scale the number of NAT IP addresses used. The proxyless architecture enables a chokepoint-free NAT operation so that workloads’ throughput and latency are minimally impacted. The myriad of benefits associated with using Cloud NAT over a traditional proxy are depicted below.

Lastly, Cloud NAT works with Google Compute Engine, Google Kubernetes Engine, and Serverless VPC Access connectors to support Cloud Functions, Cloud Run, and Google App Engine Standard. The built-in cloud integrations helped give Macy’s the confidence to deploy Cloud NAT knowing it met all the predefined requirements.

Upholding the highest security standards

Macy’s took advantage of Cloud NAT’s built-in logging and monitoring capabilities. NAT logs are automatically forwarded to a SIEM tool for transformation and analysis. This innate functionality provides greater insight into Macy’s environment and allows the security team to take action if any anomalous behavior is detected.

Cloud NAT’s inherent security and extensive benefits made it a straightforward decision for Macy’s. However, Cloud NAT is only one component of Macy’s defense-in-depth strategy, as illustrated below. It works with the existing Cloud Firewall rules to ensure only appropriate traffic is entering and exiting workloads. 

None of the Cloud Firewall rules previously established needed to be reconfigured since Cloud NAT sits in front of the firewall. Existing egress firewall rules are evaluated for each packet before it hits the NAT, and ingress firewalls rules are evaluated after the packet hits the NAT. Cloud NAT does not permit any unsolicited inbound requests from the internet even if firewall rules would otherwise permit those requests.

Macy’s continues to prove that security will never go out of style.

Learn more about Cloud NAT

Cloud NAT provides a path to help ensure your private resources stay private. Please watch this video on protecting your network with Cloud NAT to learn more about how it can help secure your environment. You can also get started using our quick start guide.

Dallas County’s mission is to provide exceptional services that promote a thriving community. As the second most populous county in Texas and the ninth largest in the country, they serve over 2.6 million constituents in the Dallas, Texas area. The county population is diverse with a large number of both English and non-English speakers. According to the latest census data, the Latino population comprises 41% of Dallas County, followed by White (27%) and Black (22%), plus a significant Vietnamese population.

Realizing that the services they offered were not easily accessible to everyone in the community, Dallas County set out to create more language-inclusive systems. They worked with Google Public Sector to create two language-support programs to promote equitable access to services across the county. This initiative was built on their existing foundation of innovation and commitment to diversity, equity, and inclusion.

Using innovation to support equitable access

In April 2023, Dallas County began using Contact Center AI (CCAI) and Enterprise Translation Hub (ETH) to provide critical language services to constituents. Both solutions, powered by Google Cloud and SpringML, enable the County to extend services to many more of their constituents effortlessly. Their ETH program translates existing service documents. Spanish speakers can also use the county’s CCAI-powered virtual assistant for self-service access.

Translating existing documents from English to over 130 different languages used to require a large amount of resources and excessive time and effort for staff to make them fit in the existing format. ETH can now translate existing documents, forms, County Commissioner newsletters, etc into over 130 different languages without the original meaning of the document getting lost in translation , saving Dallas County resources and preserving the constituent’s ability to follow along with each document as intended.

The new virtual assistant helps constituents find what they’re looking for when visiting the Dallas County website. SpringML helped design and build a custom virtual assistant trained on county data to provide answers to constituent questions, working with Dallas County to align on the top 25 most frequently asked questions based on website and call center statistics. The virtual assistant provides support through instant message conversations available in Spanish and English. This assistant, which is available 24/7, can save users a significant amount of time by providing helpful answers to commonly asked questions and contact information like email addresses, phone numbers, and locations. Both projects allow the county to communicate better by providing resources in their constituents’ native languages, bringing language access to 90% of the population.

Helping the community thrive

Within the first week of the ETH program, seven departments operating within Dallas County have enrolled to have their documents translated. Forms are already being rolled out in both Spanish and Vietnamese. In that same timeframe, the virtual assistant has held more than 2,000 conversations, covering a wide range of topics. Both are bringing services to non-English-speaking populations quickly and effectively, making it easier for individuals, companies, and community partners to conduct business with Dallas County.

This is just the start of Dallas County’s goal to use innovative technology solutions to improve people’s lives. The county is already looking to expand its use of Google Cloud solutions to build even more accessible services. 

To learn about how Google Public Sector can help state and local governments transform the way they serve their communities, visit   Google Cloud for State and Local Government , and read about Dearborn’s digital service transformation on the blog page for another example of these solutions in action.

Editor’s note: This post was originally published on the Google Workspace blog.

At Google, we are committed to the privacy and security of our customers and to fostering trust in our cloud technologies. We regularly engage with customers, privacy regulators, and policymakers around the globe to understand evolving expectations and to adapt our products accordingly. 

On July 5, 2023, the Dutch Ministry of Education affirmed to the Dutch Parliament that Google has delivered on the commitments it made as part of the data protection impact assessment (DPIA), conducted by the Dutch government and education sector representatives. This means that public sector entities and educational institutions in the Netherlands can continue to use Google Workspace and Google Workspace for Education with renewed confidence.

For the past three years, Google has worked closely with the Dutch government and education sector representatives to support their DPIA of Google Workspace, Google Workspace for Education, and ChromeOS under the General Data Protection Regulation (GDPR). In line with their cloud strategy, the Dutch government conducted this DPIA to diversify their choice of cloud providers.  

While our services have always upheld high standards for privacy and supported GDPR compliance, we invested significant time and resources throughout the DPIA to address the outlined recommendations, demonstrating our commitment to the process and our customers. As part of the DPIA, we implemented enhancements that can provide Dutch public sector and education customers with further clarity regarding the data processing roles of the parties, and greater transparency and assurance regarding Google’s processing of service data.1

These improvements will benefit our broader global customer base. As a result of the DPIA engagement, we announced our intention to offer new contractual commitments for service data, and will begin to make those commitments available later this year. Additionally, we have expanded our ISO 27001, 27017, 27018 and 27701 certifications for Google Workspace to address service data, so that our customers can have peace of mind knowing that our privacy controls for this data have been verified by independent auditors.

In parallel, following a DPIA on Chrome services, a data processor mode for managed ChromeOS devices will become available in the Netherlands this August, before expanding to other markets. A managed ChromeOS device is typically a Chromebook used in a school or enterprise setting, that is managed by an IT admin. The new ChromeOS data processor mode will give customers greater insight into, and control over, data processed by Google.

More broadly, we remain committed to helping customers navigate their DPIA processes, and have developed comprehensive resources to help them conduct their DPIAs.  

We appreciate the partnership with the Dutch government and education sector, and we will continue to partner closely on privacy matters. We believe in choice for public institutions when choosing their productivity tools and are proud to serve them as customers. We remain committed to supporting organizations across the Netherlands and globally in their compliance journeys.

Service Data is defined in the Google Cloud Privacy Noticeas the personal information Google collects or generates during the provision and administration of the Cloud Services, excluding any Customer Data and Partner Data.