AWS IoT SiteWise now supports definition of asset measurement units at the asset level, enabling customers to use a single asset model for machines reporting measurements in different units.
Read More for the details.
Amazon EMR Studio is an integrated development environment (IDE) that makes it simple for data scientists and data engineers to develop, visualize, and debug big data and analytics applications written in R, Python, Scala, and PySpark.
Read More for the details.
Amazon S3 on Outposts now supports versioning, helping you to locally preserve, retrieve, and restore each version of every object stored in your buckets. Versioning objects makes it easier to recover from both unintended user actions and application failures. After you enable versioning for your S3 on Outposts bucket, all variants of an object are stored in the same bucket, and are distinguishable by a versionId.
Read More for the details.
Today, we’re pleased to announce that Amazon SageMaker Autopilot has added a new training mode that supports model ensembling powered by AutoGluon. For moderately large datasets (< 100MB), ensemble training mode builds machine learning (ML) models with high accuracy quickly – up to 8x faster than the current hyper parameter optimization (HPO) training mode with 250 trials. Amazon SageMaker Autopilot automatically builds, trains, and tunes the best ML models based on your data, while allowing you to maintain full control and visibility. The current HPO mode uses a combination of hyper parameter values to maximize the accuracy of a single model. However, in cases when a single model is unable to capture the complex characteristics of data, combining (or ensembling) the predictions from diverse models can significantly improve overall model accuracy.
Read More for the details.
AWS Elastic Disaster Recovery (AWS DRS) is now available in the Asia Pacific (Jakarta) Region.
Read More for the details.
You can now use separate encryption keys for each customer in a single hierarchical namespace enabled storage account.
Read More for the details.
Join us online or in-person at Google Cloud Next ’22, our flagship event where over 1 million people across the globe come together to learn and grow. Don’t miss out on meeting other IT and security professionals, developers, and business leaders. Learn about the latest product announcements and hear valuable insights from Google executives, including Thomas Kurian, Google Cloud CEO, and Aparna Pappu, VP and GM for Google Workspace. You’ll also hear inspiring stories from our global customers, such as Airbus and Iron Mountain, and technology partners like Asana, about how they’re using Google Workspace in their organizations.
Here’s the Google Workspace lineup of keynotes and sessions not to miss:
1.What’s next in productivity and hybrid work
Aparna Pappu, VP and General Manager of Google Workspace delivers our keynote on the innovations coming to Workspace that help organizations and their employees collaborate, create, and innovate securely from anywhere. The session includes a live demo with Ilya Brown, VP of Product Management for Gmail.
2. Boosting collaboration in the hybrid workplace
Hybrid work is now the norm for millions of people and organizations. Learn how Google Workspace combines the virtual and physical office space to keep your teams delivering their best. Presenters: Dave Citron, Director of Product Management for Google Workspace and Nerina Martinez, Senior Director of Technology at Iron Mountain.
3. How to modernize your frontline workforce with Google Workspace
Frontline workers are often the first point of contact with your customers and provide essential services. By adopting Google Workspace, organizations can improve collaboration easily, train employees quickly and effectively, and automate simple workflows and processes. Google Workspace provides a cloud-native solution that’s secure, easy to manage and deploy, customizable, and extensible. Presenters: Frank Weigel, VP and General Manager of Product and Engineering for Google Cloud and Rhys Philips, Change and Adoption Leader at Airbus.
4.Create modern, best-in-class applications using Google Workspace
In this session we’ll share the extensibility opportunities Google Workspace brings to your organization and how they can be instrumental in adding business value. Presenters: Matt Izatt, Product Manager for Google Workspace and Strand Sylvester, Product Manager at Asana.
5. Make zero trust a reality with Google Workspace security solutions
The legacy approach to security—bolting on more complex products—has passed its useful life. But what’s the alternative? Learn how Google Workspace delivers data protection, access management, and AI-powered protection from phishing, ransomware, and malware to make zero trust a reality. Presenters: Andy Wen, Director of Product Management for Google Workspace and Mark Eimer, Senior Vice President and CTO for Hackensack Meridian Health.
6. Solving for Digital Sovereignty with Google Workspace
As more European organizations move their operations to the cloud, local governments are defining requirements to control, limit, and monitor transfers of data to and from the European Union. Discover how Google Workspace answers this call with an innovative, cloud-first approach. Presenters: Luke Camery, Product Manager, Collaboration Security & Compliance for Google Workspace and Ganesh Chilakapati, Group Product Manager, Privacy, Sovereignty, & Compliance for Google Workspace.
7.Better together: How Google Cloud makes Google Workspace better for you
Organizations of all sizes choose Google as a technology partner, and benefit from our complete technology stack. Learn how we can empower IT teams, developers, knowledge workers and frontline workers with Google Workspace and Google Cloud, and how our partners, Net Conversion and SAP, are creating better together solutions that supercharge Google Workspace. Presenters: Wesley Chun, Developer Advocate, Google Workspace and Brett Hughes, VP, Analytics & Technology, Net Conversion.
8.Grow your Workspace Administration career with Google Cloud credentials
With the increase in remote and hybrid work, the demand for Google Workspace administrators is higher than ever. Take the next step in your career by earning the Professional Google Workspace Administrator certification. Learn how it can help validate Google Workspace expertise, elevate careers, and transform businesses. Presenters: Mara Soss, Credentials Benefits and Engagement Lead, Google Cloud and Jason Butler, Sr. Deployment Engineer | Pod Lead, SADA Systems.
Register now for free, and join us online, or flag your interest in attending a local, in-person experience. We hope to see you there.
Read More for the details.
Compliance officers and platform engineering teams often find it challenging to ensure security, manage consistency, and oversee governance across multiple products, environments, and teams. Google Cloud’s Organization Policy Service can help tackle this challenge with a policy-based approach that simplifies policy administration across Google Cloud resources and projects.
We’re excited to announce the Preview release of Custom Organization Policy, and to showcase the integration with Google Kubernetes Engine (GKE). Custom organization policy for GKE can improve security and efficiency using guardrails you define tailored to your organization’s needs, and it’s offered to Google Cloud customers at no additional cost.
A policy is a statement of intent, such as “all clusters must be configured for auto-upgrade,” that gets implemented by the system. In our Organization Policy Service, a policy constraint is used to define the intent (auto-upgrade is enabled), and a policy is used to apply the constraint to a specific resource like a project or folder. Custom organization policy can extend the capability of our Organization Policy Service by helping you author your own custom constraints.
Let’s break down five ways Custom Organization Policy can help engineering organizations improve security and efficiency.
Establishing and maintaining consistent configuration and security standards across multiple services, products, and teams can be challenging. Cloud solutions are the sum of many parts and securing them often requires deep collaboration across multiple teams and stakeholders.
Our Organization Policy Service helps consolidate and simplify policy administration, providing a single framework to efficiently manage policy enforcement across your organizational hierarchy. Organization Policy supports integration with GKE and other Google Cloud services using built-in policy constraints.
With the Preview of Custom Organization Policy for GKE, we’re excited to provide you the flexibility to define and enforce policies customized to your business and team needs. Built-in and custom policy constraints are designed to be used together. Custom organization policies behave just like built-in organization policies and can be integrated into CI/CD workflows to deliver policy changes as code.
GKE can provide security “out of the box” by implementing security best practices as default values. For example, GKE uses shielded nodes, enables Cloud Logging, and disables the Kubernetes web dashboard by default. While our defaults are a solid baseline, compliance officers might have specific requirements when they attest security during audits. Your platform teams might also want to put guardrails in place to ensure these defaults and your organization’s own best practices are followed. Custom organization policy for GKE helps with both of these requirements.
At Google Cloud, we recommend managing security governance and compliance through policy. Policy establishes clear definitions and contracts across the multiple systems, processes, and teams involved. With a policy-based approach, you have additional opportunities to automate and integrate with other tools and processes to help reduce overhead and friction when tackling continuous compliance and security posture management.
The addition of Custom Organization Policy for GKE provides you with additional flexibility to define your security goals and engineering standards as policy, and to implement guardrails and enforcement at scale.
Custom Organization Policy for GKE comes ready-to-use for customers at no additional cost, and doesn’t require installation of additional cluster components. You simply define your custom policy constraints in a YAML file and then apply them to your Google Cloud resources using Cloud Shell or API.
Because Organization Policy Service is built into GKE, it can reduce the burden on platform and security teams of managing the lifecycle of another add-on, and allow administrators to easily author new policy constraints.
You can also use Custom Organization Policy alongside popular third-party policy solutions such as Gatekeeper OPA or Kyverno. Custom Organization Policy enforces constraints on the GKE API (your clusters and node pools), while the other solutions can cover resources inside your Kubernetes clusters, such as your Deployments.
Wouldn’t it be nice if cloud security and governance was “all or nothing”? But like the saying goes, “every rule has an exception”. Security and platform teams often face the challenge of defining and implementing org-wide best practices while also supporting an exemption process for scenarios where those standards cannot be met.
Organization Policy provides tools for administrators to manage policy across different projects and resources using policy inheritance and the Organization Policy resource hierarchy. Admins can use the same framework to manage exemptions; it’s as easy as modifying a policy to include a new condition that exempts a specific resource. For example, to make an exemption for a specific GKE cluster, a new condition that identifies that cluster by tag can be added to the organization policy.
Engineering organizations are increasingly looking to policy solutions to codify engineering standards, implement guardrails for developers, and integrate continuous compliance and security upstream in the development process using automation.
Custom Organization Policy for GKE provides your organization a simple way to help define and enforce engineering standards for GKE clusters and node pools. The structured, policy-focused approach means that each constraint and policy can be consistent in syntax and readability. This feature can drastically reduce developer onboarding and learning times, and can minimize the need to maintain documentation that defines engineering guidelines and how to audit and enforce them.
Engineering standards and guardrails are integral to establishing security culture, driving development efficiency, and reducing friction in cross-team collaboration. Policy provides a clear and consistent representation of these standards, and the ability to audit and enforce them.
The preview of Custom Organization Policy for GKE is a simple way to introduce flexible and powerful policy into your organization’s toolkit, and is built-in and ready to use at no additional cost. Whether it’s improving security, ensuring compliance, or implementing engineering standards, Custom Organization Policy can help.
Looking for a few examples to get started? Check out the examples on the Custom Organization Policy for GKE documentation. Want to know more about building security guardrails for developers on Google Cloud? Give this blog post a read.
Read More for the details.
Analytics data is growing exponentially and so is the dependence on the data in making critical business and product decisions. In fact, the best decisions are said to be the ones which are backed by data. In data, we trust!
But do we trust the data ?
As the data volumes have grown – one of the key challenges organizations are facing is how to maintain the data quality in a scalable and consistent way across the organization. While data quality is not a newly found need, the needs used to be contained when the data footprint was small and data consumers were few. In such a world, data consumers knew who the producers were and producers knew what the consumers needed. But today, data ownership is getting distributed and data consumption is finding new users and use cases. So the existing data quality approaches find themselves limited and are isolated to certain pockets of the organization. This often exposes data consumers to inconsistent and inaccurate data which ultimately impacts the decisions made from that data. As a result, organizations today are losing 10s of millions of dollars due to the low quality of data.
These organizations are looking for solutions that empower their data producers to consistently create high quality data cloud scale.
Earlier this year, at Google Cloud, we launched Dataplex, an intelligent data fabric that enables governance and data management across distributed data at scale. One of the key things Dataplex enables out-of-box is for data producers to build trust in the data with a built-in data quality.
Dataplex data quality task delivers a declarative, data-ops centric experience for validating data across BigQuery and Google Cloud Storage. Producers can now easily build and publish quality reports or can easily include data validations as part of their data production pipeline. Reports can be aggregated across various data quality dimensions and the execution is entirely serverless.
Dataplex data quality task provides –
A declarative approach for defining “what good looks like” that can be managed as part of a CI/CD workflow.
A serverless and managed execution with no infrastructure to provision.
Ability to validate across data quality dimensions like freshness, completeness, accuracy and validity.
Flexibility in execution – either by using Dataplex serverless scheduler (at no extra cost) or executing the data validations as part of a pipeline (e.g. Apache Airflow).
Incremental execution – so you save time and money by validating new data only.
Secure and performant execution with zero data-copy from BigQuery environments and projects.
Programmatic consumption of quality metrics for Dataops workflows.
Users can also execute these checks on data that is stored in BigQuery and Google Cloud Storage but is not yet organized with Dataplex. For Google Cloud Storage data that is managed by Dataplex, Dataplex auto-detects and auto-creates tables for structured and semi-structured data. These tables can be referenced with the Dataplex data quality task as well.
Behind the scenes – Dataplex makes use of an open source data quality engine – Cloud Data Quality Engine – to run these checks. Providing an open platform is one of our key goals and we have made contributions to this engine to integrate seamlessly with Dataplex’s metadata and serverless environment.
You can learn more about this in our product documentation.
One of our enterprise customers – American Eagle Outfitters (AEO) – is continuing to build trust in their critical data using Dataplex Data Quality Task. Kanhu Badtia, lead data engineer from AEO, shares their rationale and experience with Dataplex data quality task:
“AEO is a leading global specialty retailer offering high-quality & on-trend clothing under its American Eagle® and Aerie® brands. Our company operates stores in the United States, Canada, Mexico, and Hong Kong, and ships to 81 countries worldwide through its websites.
We are a data-driven organization that utilizes data from physical and digital store fronts, from social media channels, from logistics/delivery partners and many other sources through established compliant processes. We have a team of data scientists and analysts who create models, reports and dashboards that inform responsible business decision-making on such matters as inventory, promotions, new product launches and other internal business reviews. As the data engineering team at AEO, our goal is to provide highly trusted data for our internal data consumers.
Before Dataplex – AEO had methods for maintaining data quality that were effective for their purpose. However, those methods were not scalable with the continual expansion of data volume and demand for quality results from our data consumers. Internal data consumers identified and reported quality issues where ‘bad data’ was impacting business critical dashboards/reports . As a result, our teams were often in “fire-fighting” mode – finding & fixing bad data. We were looking for a solution that would standardize and scale data quality across the production data pipelines.
The majority of AEO’s business data is in Google’s BigQuery or in Google Cloud Storage (GCS). When Dataplex launched the data quality capabilities, we immediately started a proof-of-concept. After a careful evaluation, we decided to use it as the central data quality framework for production pipelines. We liked that –
It provides an easy declarative (YAML) & flexible way of defining data quality. We were able to parameterize it to use across multiple tables. It allows validating data in any BigQuery table with a completely serverless and native execution using existing slot reservations. It allows executing these checks as part of the ETL pipelines using DataPlex Airflow Operators. This is a huge win as pipelines can now pause further processing if critical rules do not pass. Data quality checks are executed in parallel which gives us the required execution efficiency in pipelines. Data quality results are stored centrally in BigQuery & can be queried to identify which rules failed/succeeded and how many rows failed. This enables defining custom thresholds for success. Organizing data in Dataplex Lakes is optional when using Dataplex data quality.
Our team truly believes that data quality is an integral part of any data-driven organization and Dataplex DQ capabilities align perfectly with that fundamental principle.
For example, here is a sample Google Cloud Composer / Airflow DAG that loads & validates the “item_master” table and stops downstream processing if the validation fails.
It includes simple rules for uniqueness, completeness and more complex rules for referential integrity or business rules such as checking daily price variance. We publish all data quality results centrally to a BigQuery table, such as this:
We query this output table for data quality issues & fail the pipeline in case of critical rule failure. This stops low quality data from flowing downstream.
We now have a repeatable process for data validation that can be used across the key data production pipelines. It standardizes the data production process and effectively ensures that bad data doesn’t break downstream reports and analytics.”
Here at Google – we are excited to enable our customer’s journey to high quality, trusted data. To learn more about our current data quality capabilities please refer to –
Read More for the details.
Editor’s note: This blog was originally published in Ukrainian on September 14. 2022.
Google launches cloud technology training program to support Ukrainian businesses and IT professionals
Around the world, organizations across multiple industries are in the midst of digitally transforming their businesses. And Ukrainian businesses are no different – they are looking for new ways to survive, grow and thrive digitally in an unstable and uncertain environment. The driving force behind these digital transformations for Ukrainian businesses will be people trained in the skills required to implement and maintain large-scale cloud deployments, particularly in areas like artificial intelligence, machine learning, data analytics, application development, security, and cloud architecture. We want to help Ukrainian people that are working for, and running these organizations to learn new cloud technology skills to empower them to build and grow organizations that will support the future of their country.
Today we are launching the “Grow your career with Google Cloud”program for Ukrainians looking to develop world-class expertise with practical skills in cloud to fuel the rapid digital transformation of Ukraine and invest in its future with expanded job opportunities for IT professionals.
It is our aim to train up to 10,000 Ukrainians in cloud technology by the end of 2023. The “Grow your career with Google Cloud” program will help IT developers and students gain the practical and in-demand skills they need to grow their careers or find new jobs.
The ‘Grow your career with Google Cloud’ Ukraine Program is created for:
IT specialists and developers who want to develop their cloud skills and career.
IT students who will graduate next year and need cloud skills for future jobs.
For effective learning, it is recommended to have at least a basic level of English.
What will program participants receive
Two months of no-cost access to Google Cloud Skills Boost, the definitive destination for Google Cloud learning.
Opportunity to gain real-world hands-on experience by earning Google Cloud skill badges. Skill badges validate an individual’s cloud skills in support of reaching their cloud career goals.
Individual learning paths aligned to job roles. These curated collections of content combine on-demand courses with hands-on learning. For beginners, we recommend the Getting Started with Google Cloud learning path as the content is localized in Ukrainian.
Prizes for 3 and more earned skill badges.
Access to cloud experts at regular Q&A sessions.
More details are here.
Ready to get started? Click here to register. Participants will be organized into cohorts, with the first cohort starting October 4, 2022. Once you have completed registration, you will receive an email with next steps, including information for upcoming webinars and cloud expert led Q&A sessions.
More information about the “Grow your career with Google Cloud” program is here.
Read More for the details.
AWS IoT Device Management Secure Tunneling now provides support for multiple simultaneous TCP connections, unlocking new use cases for advanced troubleshooting on remote devices. Secure Tunneling allows customers to establish bidirectional communication to remote devices using SSH, VNC or RDP. Now with multiple simultaneous TCP connections, customers can establish tunnels to access HTTP-based applications that typically make several connections for assets like images, javascript, and css. For example, you can now remotely access a web application that is running on a device to gain real-time telemetry or perform administrative tasks in a web-based GUI.
Read More for the details.
Amazon Corretto 19 is now generally available. This version supports the latest OpenJDK feature release and is available on Linux, Windows, and macOS. You can download Corretto 19 from our downloads page.
Read More for the details.
Azure regional Web Application Firewall (WAF) supports operators “Any” and “GreaterThanOrEqual”, improving WAF tuning capabilities. Azure’s global WAF now supports geo-match filtering using the socket address in addition to remote address.
Read More for the details.
In the fast-moving world of security, Google Cloud continues to constantly update our products and services to help protect your data and your applications. In the second installment of this regular series, let’s dive into what’s new and cooking in Google Cloud Security.
For those of you curious to read the previous installment, we got you covered! 🙂
Security Roundup – stories and launches from first quarter 2022
Whether you keep your secrets on-prem or in the cloud, they need to be well protected. To help protect your External keys, Cloud External Key Manager (EKM) recently launched EKM over VPC that will provide your application with an extra layer of security. You can now use asymmetric keys for signing as EKM recognizes both RSA and Elliptic curve asymmetric keys. EKM also provides an additional Organizational policy level that lets you define what type of key you want to use.
Evolving Cloud External Key Manager – What’s new with Cloud EKM
When you have tons of potentially sensitive data that needs to be analyzed for its risk profile, you can use Automatic Data Loss Prevention (DLP) with BigQuery to reduce the risk of data leaks. With this feature, DLP can be configured to run as a cron job over BigQuery data, which will report the results in a DataStudio dashboard. Automatic sync with DLP also provides insights and risk scores for each table in Chronicle which can then be used for enhancing threat detections.
Google launches Automatic DLP for BigQuery
Passwords are one of the common forms of authentication and password reuse is a big threat to account protection. Never fear – reCAPTCHA Enterprise offers a simple way to verify if your passwords have been compromised. If reCAPTCHA finds out that your password has been leaked, it warns you so that you can secure your account. This feature coupled with Account Defender or multi-factor authentication can help secure your organization better.
Announcing reCAPTCHA Enterprise password leak detection in GA
Security Command Center (SCC) released new finding types that alert customers when SCC is either misconfigured or configured in a way that prevents it from protecting your resources as expected. These findings provide remediation steps to return SCC to an operational state. Learn more and see examples:
Remediating Security Command Center errors
Cloud Armor helps secure web applications by identifying DDoS and other web attacks. You can now customize the rate-limiting per client using new rule actions. Also, Cloud Armor provides capabilities to detect and manage bot traffic at network and application layers. reCAPTCHA Enterprise for WAFwill help detect and manage bot activity at the network level, whereas Cloud Armor’s Adaptive Protection feature based on machine learning identifies Layer 7 attacks.
Announcing new Cloud Armor rate limiting, adaptive protection, and bot defense
In our recent studies, we have released a comprehensive mapping between Google Cloud’s native security offerings and MITRE ATT&CK. This can help you choose the products for your security use cases. Read more about the mapping methodology here:
Announcing MITRE ATT&CK mappings for Google Cloud security capabilities
See you in the next installment of #SecurityRoundup!
Read More for the details.
AWS Elemental MediaConnect now supports Secure Reliable Transport (SRT) caller mode for both sources and outputs in MediaConnect flows. SRT caller mode provides greater flexibility when configuring video transport workflows using MediaConnect.
Read More for the details.
Amazon SageMaker Automatic Model Tuning allows you to find the most accurate version of your machine learning model by finding the optimal set of hyperparameter configurations for your dataset. SageMaker Automatic Model Tuning is now integrated with the SageMaker Search API, which lets you quickly find and evaluate the most relevant model tuning jobs from potentially hundreds or thousands of them.
Read More for the details.
We’re excited to announce the General Availability of cross-project service referencing with Internal HTTP(S) Load Balancing and Regional External HTTP(S) Load Balancing. This new capability allows organizations to configure one central load balancer and route traffic to hundreds of services distributed across multiple different projects. You can centrally manage all traffic routing rules and policies in one URL map. You can also associate the load balancer to a single set of hostnames and SSL certificates, optimizing the number of load balancers needed to deploy your application, and lower manageability, operational costs, and quota requirements. By having different projects for each of your functional teams, you can also achieve separation of roles within your organization.
Google Cloud Load Balancing, a fully managed and distributed service, helps your applications reach planet scale, no matter where you deploy your workloads — cloud or on-prem — while supporting millions of queries per second and meeting your high availability and security requirements. Our HTTP(S) load balancers support advanced traffic management capabilities out-of-the-box, such as traffic mirroring, weight-based traffic splitting, and request/response-based header transformations, giving you fine-grained control over how traffic is handled. Our load balancers are built on the open-source Envoy Proxy, which allows you to extend your traffic management across Google Cloud, other clouds, or on-premises.
The introduction of cross-project service referencing brings numerous benefits to Cloud Load Balancing environments.
1. Reduce operational complexity and costs by exposing multi-project services using a single load balancer
As shown in the diagram above, you can now configure the frontend resources of a load balancer: forwarding rules, target proxy and URL map in one project (this project has to be within a Shared VPC setup using host and service projects, see image below), and backend services and backends in different service projects, in the same shared VPC setup.
With this capability, you can create one central load balancer and configure just one URL map with all your routing rules. This central URL map can then refer to hundreds of cross-project backend services that can be distributed across multiple different projects, but use the same centrally provisioned shared VPC network. With a shared VPC network, you don’t have to worry about trying to link multiple VPCs or managing firewall rules for many VPCs.
Further, you can expose all of your services with just one forwarding rule, thus reducing the number of hostnames and SSL certificates that you have to manage. With fewer forwarding rules and other load balancing resources, you not only incur lower costs, but also reduce your operational overhead and quota requirements.
2. Achieve separation of roles for your functional teams with the flexibility of secure cross-project access of services
Service owners can focus on building services in service projects, while network teams can provision and maintain load balancers in another project, and both can be connected using cross-project service referencing. Both teams have access to view, configure, and modify only those resources that come within their purview. This enables seamless separation of responsibilities, minimizes confusion, and accidental errors, while providing the flexibility for cross-team collaboration.
3. Provide service owners exclusive control over service-centric traffic management policies
Service owners can have exclusive control over policies that are configured at the backend service and determine how the load balancer distributes traffic to their services. For example, service owners can define policies for session affinity, health checks, identity-based access, outlier detection, and several other advanced traffic-management capabilities.
4. Expose services securely with fine-grained access control
Service owners can maintain autonomy over the exposure of their services, and control which users can access their services via the load balancer. This is achieved by a special IAM role, the Load Balancer Service User IAM role. Only users who are provided this role can access cross-project services. You can further define Organizational Policy Constraints that can limit cross-project referencing capabilities to specific projects, specific folders, or even completely disallow the usage of this feature within your organization. Using both IAM and Organizational Policies, you can achieve a granular access control as per your needs, prevent accidental misconfigurations, and follow your organization’s security norms.
(For step-by-step instructions, refer to the setup guides: Internal and External Load Balancing)
At a high level, you perform the following steps to configure your cross-project services and the central load balancer.
Step 1: As a shared VPC and network administrator, enable shared VPC on the host project and attach service projects to it. Then, create network, subnetworks and firewall rules in the host project, and grant subnetwork permissions to the service administrator and load balancer administrator.
Step 2 : As a service owner or administrator, create a backend service in a service project and attach backends to it. Then grant IAM permissions to load balancer administrators to access your backend service.
Step 3: As a load balancer administrator, create a load balancer in a different service project or the host project that directs traffic to the cross-project backend service.
This capability will be soon introduced in the Global External HTTP(S) Load Balancing, thus covering all HTTP(S) Load Balancing products. You can learn more about this capability using the guides Internal HTTP(S) Load Balancing and Regional External HTTP(S) Load Balancing.
Read More for the details.
As economic conditions change, retail brands’ reliance on ever-growing customer demand puts these companies at financial and even existential risk. Top-line revenue and active customer growth do not equal profitable growth.
Despite multi-billion dollar valuations for some brands, especially those operating the direct-to-consumer model, the rising costs of meeting shoppers’ high expectations (i.e. free shipping, free returns) along with the escalating cost of goods, fulfillment operations, and delivery costs, create pressure for brands to turn a profit. The ONLY way brands drive profitable growth is by managing variable costs in real-time. This in turn mandates adopting modern data cloud infrastructure including Google Cloud services such as BigQuery.
To unlock the value of data, Google Cloud has partnered with SoundCommerce, a retail data and analytics platform that offers a unique way of connecting marketing, merchandising, and operations data and modeling it within a retail context – all so brands can optimize profitability across the business.
Profitability can be measured per order through short-term metrics like contribution profit or long-term metrics like Customer Lifetime Value (CLV). Often, retailers calculate CLV as a measure of revenue with no consideration for the variable costs of serving that customer, for instance: the costs of marketing, discounting, delivering orders to the doorstep, or post-conversion operational exceptions (e.g. cancellations, returns).
What may first appear to be a high lifetime value customer through revenue-based CLV models, may not be profitable at all. By connecting marketing, merchandising, and operations data together, brands can understand their most profitable programs, channels, and products through the lens of actual customer value – and optimize accordingly.
The journey for brands starts with the awareness and data enablement of a more complex data set containing all variable revenue, cost, and margin inputs. What does a retailer need to do to achieve this?
All data together in one place
Matched and deduplicated disparate data
Data is organized into entities and concepts that business decision makers can understand
A common definition of key business metrics (this is especially important yet challenging for retailers because systems are siloed by the department and common KPIs like contribution profit per order may be defined differently across a company)
Branched outputs for actionability: BI dashboards vs. data activation to improve marginal profit.
Once brands understand these requirements, up next is execution. This responsibility may fall within a ‘build’ strategy on the shoulders of technical IT/data leadership and their team(s) within a brand. This offers maximum control but at maximum cost and time-to-value. Retail data sources are complicated and change often. Technical teams within brands can spend too much time building and maintaining the tactical data ingestion process, which means they are spending less time deriving business value from the data.
But it doesn’t have to be this hard. There are other options in the market that brands can consider, such as a tool like SoundCommerce which provides a library of data connectors, pre-built and customizable date mappings, and outbound data orchestrations all tailor-made and ready-to-go for retail brands.
SoundCommerce empowers retail technology leaders to:
Maintain data ownership to allow users to send modeled data to external data warehouses or layer-on business analytics tools for greater flexibility
Provide universal access to data across the organization so every employee can have access to and participate in a low-code or no-code experience
Expand and democratize data exploration and activation among both technical and non-technical users
For retail business and marketing decision-makers, SoundCommerce makes it easy to:
Calculate individual customer lifetime value through the lens of profitability – not just revenue.
Evaluate and predict lifetime shopper performance – identify which programs drive the highest CLV impact
Set CAC and retention cost thresholds – determine optimal Customer Acquisition Costs (CAC) and retention costs that ensure marketing efforts are profitable through the lens of total lifetime transactions
Below is a sample data flow that illustrates how SoundCommerce connects to all the tools a Retailer is using and ingests the first-party data, agnostic of the platform. SoundCommerce then models and transforms the data within the retail context for brands to take immediate action on the insights they gain from the modeled data.
SoundCommerce selected the Google Cloud Platform and its services to achieve what they set out to do – drive profitability for retailers and brands. SoundCommerce perfected this very need of retailers to centralize and harmonize data, map to business users’ needs to infer key metrics and insights by visualizing the data, or reuse the produced datasets to build upon other use cases specific to retailers. SoundCommerce built a cloud-native solution on Google Cloud leveraging the data cloud platform. Data from various sources are ingested in raw format, parsed, and processed as messages in Cloud Storage buckets using Google Kubernetes Engine (GKE) and stored as individual events in Cloud BigTable. A mapping engine maps the data to proprietary data models stored in BigQuery to store the produced data as datasets. Customers use visualization dashboards in Looker to access the data exposed as materialized views from within BigQuery. In many cases, these views are directly accessible by the customer per their use case.
SoundCommerce adopted BigQuery and the recent release of Analytics Hub – a data exchange that enables BigQuery users to efficiently and securely share data. This feature ensures a more scalable direct access experience for SoundCommerce’s current and future customers. It meets brands where they are in their data maturity by giving them the keys to own their data and control their analytics-driven business outcomes. With this feature, retailers can customize their analysis with additional data they own and manage.
“Retail Brands need flexible data models to make key business decisions in real-time to optimize contribution profit and shopper lifetime value,” said SoundCommerce CEO Eric Best. “GCP and BigQuery with Analytics Hub make it easy for SoundCommerce to land and maintain complex data sets, so brands and retailers can drive profitable shopper experiences with every decision.”
SoundCommerce uses Analytics Hub to increase the pace of innovation by sharing datasets with its customers in real time by using the streaming functionality of BigQuery. Customers subscribe to specific datasets through a data exchange as data is generated from external data sources and published into BigQuery. This leads to a natural flow of data that scales easily to hundreds of exchanges and thousands of listings. From the Customer’s viewpoint, Analytics Hub enables them to search listings and coalesce data from other software vendors to produce richer insights. All of the benefits are an add-on to the BigQuery features such as separation of compute and storage, petabyte-scale serverless data warehouse, and tighter integration with several Google Cloud products.
The below diagram shows a view of SoundCommerce sharing datasets with one of its customers:
A SoundCommerce GCP project that hosts the BigQuery instance contains one or more Source Datasets that are composed into a Shared Dataset for a specific customer. The dataset is wrapped around Materialized views but can include other BigQuery objects such as Tables, Views, Authorized views and datasets, BigQuery ML models, external Tables, etc.
The same SoundCommerce GCP project contains the data exchange that acts as a container regarding the shared datasets. The exchange is made private to securely share the curated dataset relevant to the customer. The shared dataset is published into the data exchange as a private listing. The listing inherits the security permissions that are configured on the exchange.
SoundCommerce shares a direct link to the Exchange to the Customer, which they can add as a Linked dataset into their project in their Google Cloud Organization. The shareable link can be pointed to a private listing. From here on, the dataset is visible in the Customer project like any other dataset and immediately available to accept queries and return results. Alternatively, the customer can also view the listing in their own project under Analytics Hub and subscribe to it by adding it as a linked dataset.
SoundCommerce is incrementally onboarding customers to use Analytics Hub for all data sharing use cases. This enables brands to get business insights faster and gain an understanding of their profitable growth quickly. Plus, it gives them the ownership to own their own data and manage it how they see fit for their business. From a technical standpoint, the adoption of Analytics Hub has led to leveraging an inherent capability in BigQuery for data sharing, faster scaling, and reducing operational overhead to onboard customers.
Through Built with BigQuery launched in April as part of the Google Data Cloud Summit, Google is helping tech companies like SoundCommerce build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:
Get started fast with a Google-funded, pre-configured sandbox.
Accelerate product design and architecture through access to designated experts from the ISV Center of Excellence who can provide insight into key use cases, architectural patterns, and best practices.
Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools, and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in.
Click here to learn more about Built with BigQuery.
We thank the many Google Cloud team members including Yemi Falokun in Partner Engineering who contributed to close collaboration with the SoundCommerce team.
Read More for the details.
Many machine learning (ML) use cases, like fraud detection, ad targeting, and recommendation engines, require near real-time predictions. The performance of these predictions is heavily dependent on access to the most up-to-date data, with delays of even a few seconds making all the difference. But it’s difficult to set up the infrastructure needed to support high-throughput updates and low-latency retrieval of data.
Starting this month, Vertex AI Matching Engine and Feature Store will support real-time Streaming Ingestion as Preview features. With Streaming Ingestion for Matching Engine, a fully managed vector database for vector similarity search, items in an index are updated continuously and reflected in similarity search results immediately. With Streaming Ingestion for Feature Store, you can retrieve the latest feature values with low latency for highly accurate predictions, and extract real-time datasets for training.
For example, Digits is taking advantage of Vertex AI Matching Engine Streaming Ingestion to help power their product, Boost, a tool that saves accountants time by automating manual quality control work.“Vertex AI Matching Engine Streaming Ingestion has been key to Digits Boost being able to deliver features and analysis in real-time. Before Matching Engine, transactions were classified on a 24 hour batch schedule, but now with Matching Engine Streaming Ingestion, we can perform near real time incremental indexing – activities like inserting, updating or deleting embeddings on an existing index, which helped us speed up the process. Now feedback to customers is immediate, and we can handle more transactions, more quickly,” said Hannes Hapke, Machine Learning Engineer at Digits.
This blog post covers how these new features can improve predictions and enable near real-time use cases, such as recommendations, content personalization, and cybersecurity monitoring.
As organizations recognize the potential business impact of better predictions based on up-to-date data, more real-time AI use cases are being implemented. Here are some examples:
Real-time recommendations and a real-time marketplace: By adding Streaming Ingestion to their existing Matching Engine-based product recommendations, Mercari is creating a real-time marketplace where users can browse products based on their specific interests, and where results are updated instantly when sellers add new products. Once it’s fully implemented, the experience will be like visiting an early-morning farmer’s market, with fresh food being brought in as you shop. By combining Streaming Ingestion with Matching Engine’s filtering capability, Mercari can specify whether or not an item should be included in the search results, based on tags such as “online/offline” or “instock/nostock.”
Large-scale personalized content streaming: For any stream of content representable with feature vectors (including text, images, or documents), you can design pub-sub channels to pick up valuable content for each subscriber’s specific interests. Because Matching Engine is scalable (i.e., it can process millions of queries each second), you can support millions of online subscribers for content streaming, serving a wide variety of topics that are changing dynamically. With Matching Engine’s filtering capability, you also have real-time control over what content should be included, by assigning tags such as “explicit” or “spam” to each object. You can use Feature Store as a central repository for storing and serving the feature vectors of the contents in near real time.
Monitoring: Content streaming can also be used for monitoring events or signals from IT infrastructure, IoT devices, manufacturing production lines, and security systems, among other commercial use cases. For example, you can extract signals from millions of sensors and devices and represent them as feature vectors. Matching Engine can be used to continuously update a list of “the top 100 devices with possible defective signals,” or “top 100 sensor events with outliers,” all in near real time.
Threat/spam detection: If you are monitoring signals from security threat signatures or spam activity patterns, you can use Matching Engine to instantly identify possible attacks from millions of monitoring points. In contrast, security threat identification based on batch processing often involves potentially significant lag, leaving the company vulnerable. With real-time data, your models are better able to catch threats or spams as they happen in your enterprise network, web services, online games, etc.
Let’s take a closer look at how you can implement some of these use cases.
Mercari built a feature extraction pipeline with Streaming Ingestion.
The feature extraction pipeline is defined with Vertex AI Pipelines, and is periodically invoked by Cloud Scheduler and Cloud Functions to initiate the following process:
Get item data: The pipeline issues a query to fetch the updated item data from BigQuery.
Extract feature vector: The pipeline runs predictions on the data with the word2vec model to extract feature vectors.
Update index: The pipeline calls Matching Engine APIs to add the feature vectors to the vector index. The vectors are also saved to Cloud Bigtable (and can be replaced with Feature Store in the future).
“We have been evaluating the Matching Engine Streaming Ingestion and couldn’t believe the super short latency of the index update for the first time. We would like to introduce the functionality to our production service as soon as it becomes GA, ” said Nogami Wakana, Software Engineer at Souzoh (a Mercari group company).
This architecture design can be also applied to any retail businesses that need real-time updates for product recommendations.
Ad recommender systems benefit significantly from real-time features and item matching with the most up-to-date information. Let’s see how Vertex AI can help build a real-time ad targeting system.
The first step is generating a set of candidates from the ad corpus. This is challenging because you must generate relevant candidates in milliseconds and ensure they are up to date. Here you can use Vertex AI Matching Engine to perform low-latency vector similarity matching, generate suitable candidates, and use Streaming Ingestion to ensure that your index is up-to-date with the latest ads.
Next is reranking the candidate selection using a machine learning model to ensure that you have a relevant order of ad candidates. For the model to use the latest data, you can use Feature Store Streaming Ingestion to import the latest features and use online serving to serve feature values at low latency to improve accuracy.
After reranking the ads candidates, you can apply final optimizations, such as applying the latest business logic. You can implement the optimization step using a Cloud Function or Cloud Run.
Interested? The documents for Streaming Ingestion are available and you can try it out now. Using the new feature is easy: For example, when you create an index on Matching Engine with the REST API, you can specify the indexUpdateMethod attribute as STREAM_UPDATE.
After deploying the index, you can update or rebuild the index (feature vectors) with the following format. If the data point ID exists in the index, the data point is updated, otherwise, a new data point is inserted.
It can handle the data point insertion/update at high throughput with low latency. The new data point values will be applied in any new queries within a few seconds or milliseconds (the latency varies depending on the various conditions).
The Streaming Ingestion is a powerful functionality and very easy to use. No need to build and operate your own streaming data pipeline for real-time indexing and storage. Yet, it adds significant value to your business with its real-time responsiveness.
To learn more, take a look at the following blog posts for learning Matching Engine and Feature Store concepts and use cases:
Vertex Matching Engine: Blazing fast and massively scalable nearest neighbor search
Find anything blazingly fast with Google’s vector search technology
Kickstart your organization’s ML application development flywheel with the Vertex Feature Store
Read More for the details.
Immutable storage for Blob Storage on containers now includes a new append capability.
Read More for the details.