gcp

You can now query Firestore using the OR operator. Your application may often need to compare one or more fields with two or more values and select the documents that match at least one of the values. This is an extension to the IN operator which could be used to compare multiple values with the same field. 

For example, if you’re searching for information about cats and dogs, you could use the OR operator to combine the two search terms into a single query:

‘cats = black OR dogs = brown’

This would return results that include information about both black cats and brown dogs.

Firestore now supports OR queries via server and client SDKs, with streaming and offline compatibility.

Use Cases

Support for OR operator now enables you to utilize Firestore for building applications which need to filter one or more fields with two or more values using any of our existing query constructs. 

Examples 

Let’s say you are running an employee management portal on Firestore and you want the ability to filter on a number of criteria like Title, Organization, Location, Tenure etc., you can now easily utilize OR to be able to do it.

Now, say you want to query for performance reviews for a given employee that occurred within the last two years, or where the given rating was higher than 4. 

You can nest multiple ORs to accomplish such queries as follows :

Nested ORs can be useful when you need to check multiple conditions, but they can also be difficult to read and understand. It is important to use them sparingly and to only use them when necessary.

OR operator can also be used on all our existing query constructs like querying across Collection Groups, or in combination with existing operators like COUNT(). 

Let’s take an example of Collection Groups called Landmarks, you can now use OR operator to filter collections within the group

We know how valuable COUNT() is to you, now you have the ability to combine it with OR to query the data more easily as shown below :

Extended limits for IN and array contains any  

Excitement continues, use multiple IN operators and also combine up to 30 clauses using an IN or array-contains-any operator in Firestore. 

Example :

Note : Not_IN operator cannot be combined with the OR operator or have more than 10 clauses.

Next Steps 

Enjoy building apps more quickly and conveniently with Firestore.

Please refer to official documentation for more detailed information.

Welcome to the second Cloud CISO Perspectives for June 2023. In my previous newsletter, I discussed the growing, important role that accurate threat intelligence can play in helping keep organizations secure, and how we centered that at our Security Summit. 

Today’s cybersecurity threats can pose dangerous risks to organizations, and have a material impact on their business. It’s vital that leaders and boards need to stay abreast of accurate, timely threat intelligence, so I’m turning the mic over to Sandra Joyce, vice president of Mandiant Intelligence at Google Cloud, to talk more about the current threat landscape.

As with all Cloud CISO Perspectives, the contents of this newsletter are posted to the Google Cloud blog. If you’re reading this on the website and you’d like to receive the email version, you can subscribe here.

What you need to know about the ‘zero-day summer’ threat landscape

By Sandra Joyce, vice president, Mandiant Intelligence at Google Cloud

A series of recent zero-day vulnerability-related incidents has led some members of the security community to refer to this season as a “zero-day summer.” The humorous overtone belies the seriousness of the problem, and it’s important for business leaders and boards to understand that these recent major incidents are rooted in larger trends.

Vulnerabilities in file transfer tools and security products have had serious consequences for several of our customers, and concerns that the trend is snowballing are growing. Zero days discovered thus far this year are slightly ahead of last year’s pace, and unfortunately, a longer, multiyear view reveals the number of zero days is growing.

In 2021 and 2022, Chinese cyber espionage actors have been associated with 10 zero days in security devices such as firewalls, VPNs, email security appliances, networking products, and virtualization software. That’s half of all zero days we tracked as exploited by Chinese cyber espionage actors. These vulnerabilities have enabled intrusions on a global scale, allowing actors such as APT41, a state-sponsored Chinese threat actor, to gain access to a variety of sectors.

A concentration in security devices and virtualization software is unlikely to be a coincidence. We believe these actors are deliberately focusing on these devices because they represent multiple tactical advantages. For instance, these devices are challenging to monitor, reducing the likelihood of detection. Many of these devices do not support endpoint detection and response (EDR) solutions, and they lack solutions to detect modifications or collect forensic images. Furthermore, most of these zero days can be exploited without social engineering, which also limits the opportunity for detection.

Chinese cyber espionage has changed dramatically over the last decade from loud, broad intrusion activity to focused targeting in deliberate operations that are far more difficult to detect. In addition to using zero days, these threat actors have largely shifted their attention to the edge of the network, gaining access through internet accessible applications and devices. They are still phishing of course, but they are certainly less reliant on social engineering like actors from North Korea and Iran. 

The infrastructure Chinese actors use to support their operations has changed as well. More frequently, we find them using a complex, ephemeral command and control scheme that involves the use of botnets of compromised systems such as small home and home office Wi-Fi routers. This evolution makes them harder for researchers to track.

Even in the case of vulnerabilities that were not first exploited as zero days by Chinese threat actors, Chinese cyber espionage actors have demonstrated an incredible ability to adopt available exploits quickly to target organizations before they can patch. The speed of their process suggests they have deliberately organized and resourced their operations to move quickly and take advantage of opportunities from recently disclosed vulnerabilities.

FIN11’s recent widespread exploitation of a file transfer software vulnerability is the fourth time we have seen this group exploit this type of software in as many years. From late 2020 to today, FIN11 has exploited multiple zero-day vulnerabilities across software and hardware from Accellion, SolarWinds, Forta, and MOVEit. 

Mass exploitation of a vulnerability in file transfer software can allow threat actors to efficiently gain access to many organizations’ sensitive files without the need for additional lateral movement. For example, in some cases FIN11 was able to start data theft within minutes of exploiting MOVEit systems, almost certainly reducing the time required to monetize access. Limiting the intrusion activity to the file transfer devices also decreases an attacker’s footprint on the network — and therefore the likelihood of detection. Threat actors may also believe that network defenders are less likely to monitor or detect malicious activity targeting these programs compared to other intrusion and post-compromise activity that interacts with more core components of network infrastructure.

The 2020 and 2021 exploitation of Accellion File Transfer Appliance and the 2023 exploitation of MOVEit involved the development of custom malware, which was specifically designed to interact with the targeted programs. Given the apparent success of these campaigns, the threat actors likely view the investment of time and resources into understanding this software, developing or purchasing exploits, and developing malware designed to interact with it as worthwhile. 

Since 2019, financially-motivated actors have exploited 30% of the zero days that we have been able to attribute. The majority of these are linked to extortion operations. This almost certainly reflects the lucrative nature of these operations, as threat actors likely reinvested profits into developing and acquiring exploits. Mandiant regularly observes threat actors advertising and seeking zero-day exploits on underground forums and Telegram channels. 

For example, in June 2023, the English-speaking threat actor “Vars_Sec” advertised a ZTE device zero day with remote code execution for $2500. Similarly, in April 2023, the exploit broker “vulns-rock” advertised a Windows LPE zero day on the Russian-language forum Exploit.in for $150,000. The continued availability of these exploits almost certainly lowers the barrier to entry for acquiring these capabilities.

Even if we aren’t in the midst of a record-breaking “zero-day summer,” the factors behind these recent high-impact cybersecurity events are taking their toll on defenders. Similar incidents are almost inevitable, as long as threat actors continue to experience success with these approaches. 

Predicting the exact nature of the next similar incident is difficult but we can take steps now to mitigate similar risks. One important step organizations can take to mitigate some of these latest zero-day threats to file-transfer systems is to follow our MOVEit Transfer: Containment and Hardening Guide.

The proverbial lazy days of summer will just have to wait.

In case you missed it

Here are the latest updates, products, services, and resources from our security teams so far this month: 

Get ready for Google Cloud Next: Discounted early-bird registration for Google Cloud Next ‘23 has sold out, but you can still register for the conference. This year’s Next comes at an exciting time, with the emergence of generative AI, breakthroughs in cybersecurity, and more. It’s clear that there has never been a better time to work in the cloud industry. Check out our scheduled security sessions, and register now.

Safer spaces: How confidential computing can grow secure data collaboration: Confidential Computing is driving a new era of privacy-enhancing tech to help organizations make the most of the cloud without compromising privacy. Read more.

How to migrate sensitive data using Google Cloud’s CDMC-certified architecture: As enterprises accelerate adoption of cloud-based services and products, they want to secure and govern rapidly-expanding volumes of their most sensitive data. The Enterprise Data Management Council (EDM Council) has accredited a new Google Cloud whitepaper and solution source code as a Cloud Data Management Capabilities (CDMC) Certified Cloud Solution. Read more.

GKE Security Posture dashboard now GA with enhanced features: The Google Kubernetes Engine (GKE) Security Posture dashboard is now generally available. It’s designed to streamline the security management of GKE clusters, and now includes a range of powerful features such as misconfiguration detection and vulnerability scanning to help ensure your applications remain safe and secure. Read more.

Now in Assured Workloads: New regions, capabilities, and supported services: Assured Workloads can help organizations more easily achieve and maintain compliance with relevant regimes around the world without refactoring. Here are several new features and services that are now generally available (GA) in Assured Workloads. Read more.

Introducing client authentication with Mutual TLS on Google Cloud Load Balancing: We are excited to announce the Preview of front-end mutual TLS (mTLS) support, allowing you  to offload client certificate authentication using External HTTPS Load Balancing. Read more.

Protect data from disasters using new Asynchronous Replication: In today’s business landscape, data availability and integrity are paramount. Disasters can disrupt operations and pose a significant risk to critical information. To address this, we have introduced Persistent Disk Asynchronous Replication, which enables disaster recovery for Compute Engine workloads. Read more.

Improving financial control and observability with security monitoring: Over the past few weeks, we’ve introduced new capabilities and solutions to better integrate security monitoring with observability and financial controls, including building better budgets. Read more.

News from Mandiant

How we secure the AI pipeline: Mandiant analysts and responders are already using Bard in their workflows to identify threats faster, eliminate toil, and better scale talent and expertise. Organizations are keen to understand how best to integrate it into their own existing business processes, technology stacks, and delivery pipelines, and ultimately drive business value. Mandiant’s approach to securing the AI pipeline builds on the recently-announced Security AI Framework. Read more. 

Detection, containment, and hardening opportunities on compromised VMware hosts: Mandiant researchers focus on the artifacts, logging options, and hardening steps to detect and prevent tactics and techniques seen being used by threat actor group UNC3886. Read more. 

Google Cloud Security podcasts

We launched a weekly podcast focusing on Cloud Security in February 2021. Hosts Anton Chuvakin and Timothy Peacock chat with cybersecurity experts about the most important and challenging topics facing the industry today. Earlier this month, they discussed:

Can IAM actually be… fun? One of the secrets nobody in cloud security wants you to know is that Identity Access Management (IAM) is exciting, dynamic, and even… enjoyable. At least, that’s according to Ian Glazer, the founder of Weave Identity and co-founder of IDPro. What makes IAM so interesting? And how is it different between cloud providers? Listen here.

Policy-as-Code can help secure your cloud environment: We present a deep dive into the new world of policy-as-code, from who should learn it, to who should manage it, to how it can help secure the cloud. Dominik Richter, founder and head of product at Mondoo, joins us for a Policy-as-Code therapy session, and we even talk about how to tell when it’s succeeding. Listen here.

Will SIEM ever die: What can its past tell us about its future: Security Information and Event Management (SIEM) has been around a long time. We discuss which old SIEM lessons still apply today, which old lessons can harm your organizations, and what are the top modern cloud security use cases for SIEM, with David Swift, security strategist at Netenrich. Listen here.

Mandiant podcasts

Threat Trends: A requirements-driven approach to cyber threat intelligence: Dr. Jamie Collier, senior threat intelligence advisor at Mandiant, joins host Luke McNamara to discuss the recent white paper from Mandiant on developing a requirements-driven approach to intelligence, challenges that organizations face in this area, and the importance of recurring stakeholder feedback to a well-functioning cyber threat intelligence team. Listen here.

To have our Cloud CISO Perspectives post delivered twice a month to your inbox, sign up for our newsletter. We’ll be back in two weeks with more security-related updates from Google Cloud.

Recently in The Modernization Imperative (TMI), my colleague Richard Seroter shared about the importance of shifting down rather than shifting left — leveraging pre-built platform functionality to lighten the load for both your developers and operators. From my work advising engineering leaders on cloud modernization, I couldn’t agree more. However, if your organization is building its own platform (and by extension platform engineering team), then do yourself a favor and ask yourself a few questions before you begin. Take my word for it: failing to do so will ensure that your platform is a massive flop.

Creating internal technology platforms in the enterprise is all the rage these days, and for good reason: they allow IT specialists to be more independently productive, while maintaining centralized governance and security at the same time. Not only is the cloud itself a platform – it also presents the perfect opportunity to develop additional internal platforms on top of it, for example: a data platform with BigQuery, an ML/AI platform with Vertex AI, an API management platform with Apigee, or a multi-tenant container runtime platform with Google Kubernetes Engine. Platforms running on top of platforms!

To a CIO or an Enterprise Architect, the strategic importance of investing in a platform might seem obvious. There’s just one catch: nobody else in your enterprise cares about your platform. Just because you build it, doesn’t mean they’ll come.

You can force people to use your platform, of course, but then you’ll only achieve half of your mission: you’ll gain the centralized governance and security posture, but you’ll set your internal users back by forcing them to relearn, retool, and replatform something they didn’t ask for and (subjectively) didn’t need, and hence will resent before they even get started. If their experience is anything short of flawless, they will blame both the platform and you.

Can you encourage adoption, or even a wide-open embrace? Are new internal platforms always doomed to fail? Thankfully, no. 

There are just three essential questions you need to answer first to set yourself up for success: For whom are you building the platform? Who is going to build it? And, last but not least, how will you measure success?

Question 1: For whom are you building?

We’ve all heard the advice to “start with the user and work backwards.” In large enterprises, however, this can be harder than it sounds. Ultimately, the actual user of your internal platform is another employee in another department, but their needs are represented by their department head or their product owner. They, in turn, may interact with a “head of platform X” who prioritizes the feature backlog and delegates architecture design to an enterprise architect, who in turn gives guidance to the platform engineers on how to build it… What exactly does your user need again?

Here’s what should be happening: the platform engineers building your internal platform should be meeting regularly with the actual user, face to face, supported by a Product Manager who is part of their team. Everybody involved in creating or consuming the platform needs to fit into one room — or at least be visible on a single Meet/Zoom/Teams screen. This is for the same reason why the best platforms are often the ones people initially built for themselves – you can’t get any closer to your user when the user is you.

In case you don’t have the luxury of having all stakeholders in the same room, there is a powerful tool you can deploy to bridge the gap: user watch parties. User watch parties are the equivalent of a focus group, gathering your engineers together to watch the user in the room next door through a one-way mirror as they interact with the platform. In actuality, you’ll simply record their computer screen and their voice (with their permission of course). All you need to do is recruit five test users and give them the same two or three goals to complete, using either the legacy platform you aim to replace or the MVP version of the new platform you’re already building. Tell them to think out loud, and voice any assumptions they are making and any confusion they may be experiencing.

You’ll be amazed to witness how even the smartest and most savvy test user will take longer than you thought they would, and get stuck in places you didn’t think it was possible to get stuck. It’s tempting to dismiss these anecdotal observations as outliers or to think that clearer documentation and additional training will cure all ills, but the reality is that you are witnessing, in its unfiltered glory, the true challenge that a new platform faces: the need to delight your users and make their work lives easier.

User watch parties also have the benefit of aligning priorities within your engineering team because everybody just witnessed the exact same thing with their own eyes and ears. Whereas before each team member might have had their personal favorite feature that they wanted to build next, it’s now painfully obvious which existing feature is the most deserving of your team’s attention. 

Question 2: Who is going to build it?

Ideas are easy – execution is hard. If you don’t approach this question self-critically, the team to build your platform will be the one that proposed the idea. This can be a tough pill to swallow, but lest we forget: this isn’t about you, your platform engineers, your Head of Platforms, or your CIO – this is about your users. You must form a platform team that best addresses your users’ needs first, and your personal ambitions second. 

Next, what should the platform team look like? In a nutshell, the ideal platform team is dedicated, small, and cross-functional. Better to have three engineers work full-time on your new platform, than to have ten people work on it next to their many other daily priorities. Yes, this will require dedicated headcount, as well as a strong executive sponsor to give this team air cover in its nascent stages of development. It’s how every digital native company sets out to build their internal platforms, and so must you.

Building a platform is a learning challenge, not an execution challenge, so quick feedback loops and shared learning are paramount to the team’s success. They will only succeed if they sit in close proximity to each other (physically or virtually) and their dependencies towards other teams are kept to a minimum. This close proximity is also essential in order to avoid the “not my job” trap, in which each individual only accepts work that matches their expertise or job profile. A shared “show me how” mindset allows the team to default to the respective subject matter expert (SME) but expects everybody on the team to be capable of solving any technical challenge.

Finally, is there a chance you’ll need someone else’s blessing to use certain services, e.g. the CISO or your enterprise architect? If so, make them (or one of their delegates) a core part of the team — at least until their contribution is baked into the platform in the form of policy as code and architecture blueprints.

Question 3: How will you measure success?

Nobody gains from a platform that is delivered on scope, on time and on budget, but that isn’t widely adopted. To mitigate against this, don’t measure your platform team’s success in these kinds of project KPIs. Instead, look to product KPIs, which measure the platform’s usefulness, adoption and user satisfaction. 

There is no one correct way to measure a successful software product, but successful answers to these three questions demonstrate progress:

How long does it take the user to achieve their goal? (less = better)

How many users use the platform and how frequently? (more = better)

How satisfied are users and how likely are they to recommend the platform to other employees?

At Google, we like to use the H.E.A.R.T. framework for many of our consumer-facing products. We also use a modified framework that we call S.U.P.E.R. for some of our internal IT platforms:

Go to them

We’ve explained why close proximity to your internal users is paramount when building a platform and we showed how you can regain some of that lost proximity by facilitating user watch parties. We’ve also highlighted the importance of creating dedicated, small, cross-functional teams and how to incentivize them to build a compelling platform that makes your employees’ work lives easier, rather than merely delivering one IT project before they move on to the next.

We’ve only scratched the surface of what it means to successfully plan, build and run a compelling platform as a product. So much more could be said about the importance of user personas, user journey mapping, and the true role of the Product Manager, to name just a few areas. 

If you take away just one thing from this article let it be this: you do not know what your users need and want. You only have hypotheses, which you have yet to prove or disprove. Just because you build it, does not mean that they will come. You must go to them.

If you use the Slurm workload manager to manage your HPC workloads on Google Cloud, get ready for some new enhancements to Slurm on Google Cloud. We’re pleased to announce a set of improvements to the Slurm on Google Cloud open-source code, developed by SchedMD. This includes support for more resource types and operating systems, simpler deployment, improved error reporting and transparency, and enhanced support for security features. Read on to understand how Slurm could be an important part of your Google Cloud HPC environment.

Slurm on Google Cloud

Slurm on Google Cloud is an open-source scheduling solution available for deployment through the Cloud HPC Toolkit, directly through Terraform, or from the Google Cloud Marketplace. It is the result of an ongoing Google and SchedMD partnership to develop Slurm on Google Cloud and we’re excited to announce the latest updates.

Slurm on Google Cloud has a number of key benefits:

Flexibility: Slurm supports a wide variety of Google HPC infrastructure, permitting multiple configurations to meet your complex workload needs.

Scalability: Slurm on Google Cloud offers impressive scalability to handle performance requirements for large cluster and exascale computers. Slurm spans HPC, HTC, and AI workloads with proven reliability.

Cost-effectiveness: Slurm on Google Cloud is an economical way to run HPC workloads.

To enhance the experience of using Slurm on Google Cloud, we’re releasing the latest iteration of the solution. 

Improvements for Slurm on Google Cloud

The latest release is now available. New developments and improvements include:

Expanded resource support: Slurm on Google Cloud now supports ARM CPUs and NVIDIA Multi-instance GPUs. This allows you to run your cluster and workloads on a wider range of hardware, giving you more flexibility and choice.

Rocky Linux support with a Slurm-ready image: The Slurm-ready image for Rocky Linux 8, based on the new HPC VM Image for Rocky Linux, makes it easy to get started with Slurm on Rocky Linux.

Streamlined deployment updates: We’ve made it easier to update your Slurm cluster. With the new streamlined deployment updates, you can quickly and easily make changes to your cluster’s partitions with automated Slurm reconfiguration.

Improved logging and reporting: We’ve improved the logging and reporting capabilities of Slurm on Google Cloud, making it easier to track the performance of your cluster and identify any potential problems.

Hybrid Slurm advancements: We’ve made several advancements to hybrid Slurm, which allows you to run your workloads on both Google Cloud and on-premises resources. These improvements simplify the management of your hybrid cluster and improve the performance of your workloads.

Improved integrations with the Cloud HPC Toolkit: There are enhanced integrations between Slurm on Google Cloud and the Cloud HPC Toolkit, making it simpler to use the Cloud HPC Toolkit to deploy your Slurm cluster.

Support for Shielded VMs: We now support Shielded VMs for Slurm on Google Cloud. Shielded VMs provide additional security features for your workloads, such as verified boot and integrity monitoring.

Improved instance tagging: We’ve improved the instance tagging capabilities of Slurm on Google Cloud. Effortlessly tag your instances with custom labels, which can be used to control access to your cluster and to track the usage of your resources.

We’re excited to announce these new developments and improvements for the Slurm on Google Cloud open-source code. We believe these advancements will make Slurm on Google Cloud an even more powerful and scalable HPC solution.

If you are interested in trying these scripts out yourself, we suggest using our Cloud HPC Toolkit tutorial for launching a stand-alone Slurm cluster on Google Cloud. If you want to learn more, please visit the SchedMD website or contact Google Cloud.

For more than 12 years,Tembici has been democratizing easy access to transportation, promoting sustainability, and improving life in urban areas through bicycle sharing programs. The company got its start in Brazil as a university project. When I joined as a software engineering specialist in 2019, the company was just expanding to Chile and Argentina. In 2020, we began introducing e-bikes. And in 2022, we debuted our service in Colombia.

We’re Latin America’s leading micromobility company, offering a fleet of more than 22,000 bicycles and e-bikes, including cargo bikes and hundreds of hand bikes that attach to wheelchairs. The mobile apps that let riders book our bikes have been downloaded more than 1,900 million times. We operate many solar-powered stations for the bikes in four countries, with more to come.

I’m fortunate that my career at Tembici has progressed along with the company’s impressive growth trajectory. Today, I head our software engineering team, which develops services to advance our company’s goals of increasing both our bike fleet and revenues by 50% in 2023. With recent investment from the National Bank for Economic and Social Development (BNDES) and an exciting new partnership with Uber to make our bikes available via the Uber app, we have a long way to grow.

How are we expanding our infrastructure from São Paulo, Rio de Janeiro, Brasília, and other major Brazilian cities to Buenos Aires, Santiago de Chile, Bogotá, and beyond? In part, with the help ofGoogle Cloud, which seamlessly scales to support our business, underpinning our IT infrastructure with powerful and reliable capabilities across South America.

Shifting into high gear with AI for bike travel

Tembici usesBigQuery and its artificial intelligence and machine learning features to identify patterns in bike usage by location and reposition bikes according to demand. Our continuous improvement algorithm helps us analyze large volumes of traffic data and reallocate bikes depending on the day of the week and time of day.

UsingLooker, customers and Tembici staff alike can visualize our station locations on a city map. With a click, customers can get more information on the stations that are most convenient for their planned route. The Tembici facilities operations team, meanwhile, uses the map chart technology to granularly assess geographic and bike usage data to optimize decisions about new station placements.

Google Kubernetes Engine makes it easy to scale our containerized applications and quickly roll out new features. Overall, our Google Cloud environment has enhanced our efficiency and agility, enabling our teams to focus on delivering better services that further promote micromobility.

With our route calculator, for example, riders can indicate the distance they plan to travel and evaluate the costs of alternative forms of transportation. For example, a daily commute of 20 km in Brazil would translate to a monthly expense of more than R$1,200 by car or R$230 in average public transportation fees. With Tembici shared bike use, however, the cost is around just R$30.

Pedaling toward a more circular economy

Our software generates a trip summary for customers each time they complete a ride. Riders can check how many kilometers they pedaled, how many calories they burned, and the volume of carbon emissions they prevented.

It’s gratifying for the Tembici team to see the impact that cyclists are making on our society. To date, our riders have pedaled enough to cycle around Earth 19,000 times. They’ve saved our planet from more than 41,000 tons of carbon emissions, equivalent to the clean oxygen produced by over 290,000 trees. 

Google Cloud’s commitment to sustainability, which is so well-aligned with Tembici’s values, is just another reason that Google Cloud is a great partner for us as we expand our bike sharing programs and make a positive global impact with technology.

If you want to learn more about how Google Cloud can help your startup, visit our pagehere to get more information about our program, andsign up for our communications to get a look at our community activities, digital events, special offers, and more.

Sources: 

https://exame.com/negocios/uber-lanca-aluguel-bicicletas-parceria-startup-tembici/
https://www.linkedin.com/posts/tembici_calculadora-revela-qual-%C3%A9-o-meio-de-transporte-activity-6874830346832109568-Tfw2?utm_source=share&utm_medium=member_desktop
https://www.linkedin.com/posts/tembici_vaidebici-micromobilidade-tembici-activity-6965031909177323520-VNCI?utm_source=share&utm_medium=member_desktop
https://www.linkedin.com/posts/tembici_mobilidadesustentaervel-mobilidadeurbana-activity-6955588895073808384-e2zs?utm_source=share&utm_medium=member_desktop

As pet owners, we all want to make sure our pets are happy and healthy. But with so many different products and services competing for our attention, it can be overwhelming to decide which ones are best.

That’s whereWoof comes in. We are on a mission to revolutionize the pet industry by providing the best services and experiences for owners and local pet shops. In recent years, we’ve grown to become Brazil’s largest affiliate pet shop network connecting 100 million homes with more than 7,500 businesses through our website and app. Woof is now landing in the U.S. (the largest pet market in the globe) aiming at empowering local pet businesses and pet tutors.

But rapid growth brings its own challenges. How do you scale your knowledge base and infrastructure while continuing to provide personalized insights for customers? This is what sparked us to launch Woofer, a virtual canine consultant with his own personality, sense of humor, and understanding of pet behavior and health. By chatting with Woofer, our customers can get the very best products and advice based on personalized recommendations. Woofer is our answer to the universal question — if our pets could talk, what would they say? Ever wondered what it would be like if your pet could speak?

How to train your virtual assistant

To create a convincing personality, Woofer required sophisticated machine learning (ML) infrastructure. We looked closely at a number of cloud providers and selected Google Cloud as our development partner based on its ability to meet our scalability and functionality requirements. Google Cloud was also the most cost-effective choice compared to other options.

The project really took off when, in 2022, Woof joined the first Brazilian cohort of Google for Startups Cloud Academy, a three-month program in which startups work directly with Google Cloud experts and AI industry leaders to acquire new users, harness the power of data analysis to drive sales growth, and leverage Google AI/ML technologies. Working alongside Google mentors, we learned how to define our technology requirements, and benefited from Google Cloud’s guidance as our development team created a product roadmap and ran development sprints.

First, the marketing team set up Woofer’s profile and personality. This was a crucial first step based on our research which showed that a sense of humor and a unique identity trigger curiosity and desire in the customer. The team then set up Woofer’s first interactions, including how the chatbot would respond to queries about his personality and temperament, as well as recommending products and services relating to specific breeds and species.

It was a highly efficient process. With a clear, simplified development path and the Google Cloud technical support, we had our conversational AI ready in less than 20 days.

Where mutts meet ML

The success of the Woofer chatbot would not have been possible without the integration of advanced natural language understanding and processing capabilities. This is whereDialogFlow CX,AutoML,Vertex AI and other solutions in Google Cloud proved to be invaluable.

DialogFlow CX enables Woofer to work 24/7, seamlessly switch between topics, handle follow-up queries, and operate across multiple channels. With AutoML, our developers were able to train Woofer using high-quality custom ML models with minimal effort and previous ML knowledge. As the final piece of the puzzle, we deployed Vertex to build and deploy ML models faster and more efficiently.

Other important Google Cloud solutions include BigQuery andLooker Studio, which we used to set up company-wide business intelligence dashboards that enable our entire team to monitor KPIs and performance in real time. We also useFirebase to control authentication via single sign-on between our apps and web platforms.

A different breed of chatbot

Sit, stay, accelerate growth! Woofer empowers our customers to make informed decisions about pet products and services. It’s also been great for local businesses because they can connect with customers and free up staff time by delegating repetitive FAQs and product questions to the Woofer chatbot.

We’ve also been impressed by the Google Cloud experts supporting us on our journey. We love working with them and they really feel like a core part of our team. Beyond helping us get the most out of the technology, they introduce us to the broader Google Cloud community and technical events that give us insights into future product releases.

We are now working with the Google Cloud team to apply similar technology to Woofer in other parts of the business. This includes sponsored campaigns, brand launches, and even a virtual investor relations officer who delivers key financial and operational data and dashboards.

With all the potential of running our AI on Google Cloud, we’re looking forward to letting Woofer off the leash to uncover new opportunities for U.S. and Brazilian pet owners and businesses.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

The widespread availability of generative AI, such as chatbots powered by large language models (LLMs), has captured our imaginations about the possibilities of artificial intelligence. In the enterprise, generative AI has the potential to revolutionize customer experiences and employee productivity. As enterprise leaders try to distill their opportunities and competitive risks, I see a fundamental shift in who will execute those grand plans—and how that shift itself will likely become a force multiplier.

Many of us still remember Steve Ballmer, then CEO of Microsoft, jumping around on stage chanting “Developers, developers, developers!” Ballmer knew that the developer community’s ability to drive business outcomes for enterprises was the biggest opportunity for platform adoption. 

Inside Google, we’re having all sorts of interesting conversations about how to enable developers with AI, but here’s my personal take: I believe we are entering a “post-training era” in which application developers will drive the bulk of the innovation in applying generative AI to solve business problems. 

This is not to say that data science and MLOps are no longer relevant or a thing of the past. On the contrary, I am seeing a huge amount of innovation and fast iterations on building out the infrastructure and approaches to training the next generation of LLMs, including more domain-specific and lighter-weight ones to meet the requirements of a much broader set of use cases. However, I believe the availability of LLMs is democratizing access to AI for the broader community of developers, who will not need to become experts in deep learning, but rather expand their skills to integrate LLMs into enterprise application architectures. I draw the parallel to compilers, which were built by few but leveraged for innovation by many.

To get an appreciation for the difference in scale, consider that, according to the Bureau of Labor Statistics, in the US alone, there are over two million software developers but only around 150,000 data scientists. These roles overlap at times, but still: imagine at least an order of magnitude more technical practitioners being able to innovate with AI.

I think of this shift as the rise of generative engineering, or GenEng. Just as developers integrated ops practices into software engineering through the DevOps movement, I see the GenEng revolution being led by developers who build deep proficiency in how to best leverage and integrate generative AI technologies into applications.

What defines GenEng? While the rise of general purpose LLM-based chatbots have captured our imagination, the majority of enterprise use cases cannot tolerate their shortcomings, such as hallucinations. In fact, the real value for enterprises comes when they combine generative AI with their proprietary data to provide accurate, domain-specific outcomes. Developers are employing techniques such as Retrieval Augmented Generation (RAG) to augment the use of LLMs for dependable and high-value systems for enterprise applications. The generative engineer is a developer who enhances their skills with prompt engineering, embeddings for proximity searches (e.g. vector-enabled databases), and frameworks that help build LLM-powered applications (e.g. LangChain). 

These GenEng practitioners will need to have many of the same skills of traditional application development, including scalable architecting, integrating enterprise systems, and understanding requirements from the business user. These skills will be augmented with the nuances of building generative AI applications, such as involving the business domain experts in validating aspects of prompt engineering and choosing the right LLM based on price/performance and outcomes.

What are some fundamental changes in runtimes, frameworks, and tools to best enable the generative engineer? That is for the next blog post. For now, welcome GenEng practitioners!

Editor’s note: Today’s article comes from John McMullan, Head of Marketing at Thunkable. Thunkable gives users the ability to create custom, native, no code mobile applications for Android, iOS, and mobile web—without any technical expertise required. In this post John shares how their customers are incorporating maps into their mobile application experiences.

Mobile applications have created a connected global community, putting personalized devices in the pockets of 85 percent of the population. And we’re all using these devices to navigate life—figuratively and literally. That means that giving your customers or users an extra layer of information in the form of a map can enhance the user experience. 

But how do you do that with limited coding knowledge or development experience? Our platform, Thunkable, gives anyone the power to create custom, native, no code mobile applications for Android, iOS, and mobile web—without any technical expertise. And with the power of no code comes back-end integrations that can really make your app idea come alive.

Incorporating geospatial capabilities from Google Maps Platform into your app gives your users the right information at the right time. Here’s a little inspiration from our creators on what that looks and feels like. 

Location and traffic-based decision-making 

Before heading out the door and hopping in the car, maybe you check the traffic or turn on an app to help you get from point A to B as smoothly as possible. You might go the same way every day and then your device tells you to take a side road to avoid an accident. These types of quick adjustments based on changing conditions happen all the time. It’s one thing to know where you’re going, but it’s another to know and avoid slowdowns before you hit the road. 

Adding maps to your mobile app is a great way to help users quickly eliminate unforeseen delays and be on their way. You may be thinking, “Yeah, but isn’t that what the Google Maps app is for?” Sure, but what if your destination changes based on traffic or road closures? Integrating Google Maps Platform will give you more control over the user experience and eliminate frustrations caused by real-world events that you can’t control for when using Google Maps’ consumer application.

One of our creators, the Missoula (Montana) Fire Department (MFD) responds to a variety of calls all day, every day. In an emergency, every second counts. MFD developed a mobile app that digitized all their procedures and checklists to make sure all members of their crew had quick access to step-by-step instructions for incident response.

Once first-contact care is provided, the job isn’t done. An EMS or Fire Department crew must transport the injured to the nearest hospital. Minimizing the time from initial response to hospital care is critical for survival. 

With many options for hospitals and care facilities, adding Google Maps Platform  functionality directly into the app allows the crew to quickly assess the nearest location based on distance, traffic, and other factors, and make a split-second decision on where to transport the patient. 

Not all navigational needs are urgent, but if you have multiple location options, giving your app users a clear display of their choices will save time — and maybe even a life. 

A personal tour guide in the palm of your hand

Every city has a story to tell, but how do you give your guests that local flair? 

Francesco Veratti is an avid historian located in Bologna, Italy. This northern city is teaming with cultural and historical sites that the average visitor may stroll past without knowing the significance. Francesco didn’t want visitors to miss out.

He created a mobile app based on curated information from historians, locals, and crowd-sourced data that gives visitors a guided tour of all of the sights, sounds, and history within Bologna’s city center. The app highlights key destinations and builds a personalized journey for each tourist with mapping built with Google Maps Platform. No matter where you are in the city, you can quickly find and walk to the nearest cultural landmark. Since some history is tucked away in back alleys or behind new buildings, Street View is incorporated to show the user where the hidden gems are. Upon arrival at one of the featured destinations, visitors are greeted with details about the significance of the destination. 

Imagine building an app for your campus, office park, or town with easy-to-follow navigational directions and a dash of insight. An app like this is a great way to welcome new students to campus, new employees to your office park, or tourists to your town. 

Hunting for rewards — and customers

No matter our age, we still love playing games. What if you were able to merge the experience of a scavenger hunt with advanced mapping abilities? That’s what plenty of app developers have done, on a global, regional, or even a hyper-local level. 

Using Google Maps Platform embedded in a mobile app is a popular way Thunkable app creators build scavenger hunt games. These apps incorporate clues or maps to direct users to specific locations where treasure, rewards, or activation points occur. 

The app experience is very simple but engaging for users, with changing challenges, waypoints, and destinations. The benefit of these apps for brands or organizations is to encourage activity and participation, and even gain new customers. 

Put maps in your app today 

Incorporating maps into your mobile app is a great way to provide your users with a customized experience without making them switch apps or screens. And while we’ve highlighted three examples here, there are endless possibilities mapping in apps. 

For more information and to get started with Google Maps Platform, visit our website.

We’ve been saying it for years, the benefits and potential of the cloud abound. 

And yet, more than 80% of respondents in a survey of 753 business leaders point to managing cloud spend as their top organizational challenge, and these same respondents estimate that nearly 1/3 of their cloud spend is inefficient or wasted (Flexera, 2023). Many organizations are new to optimizing cloud costs and ensuring resources are used efficiently.

As your organization digitally transforms you may be realizing what other organizations are realizing too: When it comes to business value, simply migrating to the cloud isn’t enough. Achieving the full benefits of cloud requires fundamental changes to both mindset and behaviors around existing financial-management practices. It requires changing the way your disparate teams work together. 

Enter the Cloud FinOps Building Blocks

Cloud FinOps is a framework, discipline, and cultural shift combining people, processes, and technology to drive financial awareness and accountability. FinOps practices align engineering, finance, technology and business leaders and teams under a primary objective: to maximize business value from the cloud. With Cloud FinOps practices, every business stakeholder is charged not only to take responsibility for their spending and costs, but also to optimize them. These practices enable businesses to manage consumption and make sound, data-informed cloud-spend decisions. Cloud FinOps is comprised of five building blocks:

Accountability and enablement
Establishing governance and policies to manage cloud spend and realize business value.

Measurement and realization
Driving financial accountability and value realization with a defined set of KPIs and success metrics.

Cost optimization
Providing financial visibility and recommendations of IT resource usage to optimize cloud spend.

Planning and forecasting
Modernizing budgeting, forecasting, and chargeback methods to allow for iterative, innovative and cost effective development practices.

Tools and accelerators
Deploying and integrating a set of cloud cost tooling to effectively manage and track cloud spend. Learn more here. 

For a general overview of the Cloud FinOps framework and more on the five building blocks, check out these resources:

Video | What is FinOps and 3 reasons why you should care about it.
This 5-minute video provides an overview of FinOps, the 5 building blocks, and how they can benefit your organization. 

Podcast | FinOps with Joe Daly
In this podcast, Joe Daly of the FinOps Foundation shares about the key principles of FinOps, which he refers to as financial DevOps. Daly discusses how this framework is helping companies make better and more efficient financial decisions while taking advantage of the cloud.

Blog | Decoding Cloud FinOps to accelerate digital transformation
This blogpost discusses the critical role of FinOps in a successful digital transformation. It outlines key metrics to help measure and track business value and to increase visibility into the effects of digital transformation on top-line revenue. 

Article | Cloud FinOps: The secret to unlocking the economic potential of public cloud
This Forbes article profiles OpenX, the first major advertising exchange platform to migrate entirely to the cloud. It details the 5 key pillars of the Cloud FinOps framework, which OpenX leveraged in their digital transformation strategy. In just 9 months, they reduced their per-unit costs by more than 60%.  

Importantly, Cloud FinOps isn’t about saving money; it’s about making money. It’s about promoting a cost-conscious culture, financial accountability, and business agility in the cloud. Whatever stage of the cloud journey you’re at, cloud FinOps practices will help you get the most value out of Google Cloud. This framework can help to remove blockers, implement the building blocks, and empower your teams to make better business decisions. 

The Cloud FinOps Journey 

Implementing Cloud FinOps is neither a destination nor a box your organization will check then archive. Rather, Cloud FinOps is an ongoing journey and discipline. It’s inherently iterative. As such, growth and maturity across processes, capabilities, and domains requires action, repetition, and continuous learning. 

Across the five FinOps building blocks, we’ve identified 50 subprocesses to best understand organizations’ FinOps proficiency, capabilities, practice domains, and blind spots. We scale them from 1 to 5 and categorize them in one of three phases of maturity: Crawl, Walk, or Run. Organizations in the Crawl phase tend to focus on technical problem solving and cloud-cost visibility. Organizations in the Walk phase emphasize strategic improvements such as employing cost visibility dashboards to realize better business value. And organizations in the Run phase are focused primarily on transformational change and strategic innovation, factoring cost considerations into both processes and cloud architecture. 

Through this “crawl, walk, run” maturity model, we can evaluate proficiency, establish a benchmark, and recommend a targeted action plan for FinOps adoption. And whatever your level of maturity, your organization can take quick scalable action not only to foster improvement but also to evaluate outcomes and gain insights. 

The key here is that regardless of your organization’s Cloud FinOps maturity level, you can take small steps now toward continuous improvement. Here are some common focus areas and several more resources organized by maturity level that you can access.

Crawl phase 

Improve cloud-cost visibility. 

Whitepaper | Drive Cloud FinOps at scale with Google Cloud Tagging
Tags and labels can be useful and flexible tools to help your organization segment cloud spend and allocate costs. This whitepaper introduces Google Cloud Tags and best practices for implementing them. It differentiates tags, which offer reliable reporting and governance features, from labels, which can be prone to problems, including poor coverage and a lack of integrity in data labeling. 

Whitepaper | Unlocking the value of Cloud FinOps with a new operating model
This white paper unpacks the details of the FinOps operating model, including roles, organizational alignment, and driving culture change. It details how to establish strong financial governance and a cost-conscious culture. 

Whitepaper | Cloud FinOps: Shared services cost allocation
In this whitepaper, you’ll explore the elements of cost allocation as well as the complexities and challenges associated with shared-services cost allocation. While some of these concepts and models are interchangeable between legacy and cloud environments, this whitepaper focuses primarily on cloud computing and associated services.

Walk phase

Improve business-value realization. 

Blog | 5 key metrics to measure Cloud FinOps impact in your organization in 2022 and beyond
To drive business growth and topline revenue, business leaders must be able to connect cloud investments to business outcomes. As such, traditional IT metrics and KPIs must continue to evolve. In this blogpost, we’ll explore five key business-value metrics aligned to the five Cloud FinOps building blocks. 

Whitepaper | Maximize business value with Cloud FinOps
The cloud introduces new complexity and challenges to traditional IT financial management. As such, it requires strategic financial governance, processes, and partnership across the organization. This whitepaper explains how Cloud FinOps helps enterprises that have invested in cloud to drive financial accountability and accelerate business value.

Run phase 

Improve strategic cloud innovation. 

Whitepaper | Unit costing: The next frontier in cloud
In this whitepaper, you’ll explore the nature of and need for cloud unit costing, the standard by which FinOps practitioners obtain full business context for their cloud costs. It features examples from cloud-first organizations that have pioneered FinOps practices. Additionally, it examines several cloud forecasting and budgeting methods, ranging from least to most rigorous. 

Blog | You get what you pay for: Principles for designing a chargeback process
Chargeback, a crucial Cloud FinOps capability, is the process of mapping cloud consumption to internal users within an organization. It provides transparency, facilitates accountability,  enables recovery of cloud costs, and fosters a culture of fiscal responsibility. This blogpost will walk you through some best practices in designing an effective chargeback process in Google Cloud. 

Success with Cloud FinOps

As global markets continue to face challenges, there’s never been a better time to increase the return on your cloud investments. Adopting and implementing FinOps practices will help. For some real-world examples of how organizations across a range of FinOps maturity levels have collectively saved millions of dollars on their overall cloud spend, check out these customers’ stories. 

Video | Next 2022: Top 10 ways to lower your costs on Google Cloud with General Mills
In this video, which highlights ten leading cloud cost optimization practices, hear how General Mills, which is on pace to increase their cloud footprint by 60%, has approached the discipline of cost savings and accelerated their adoption of Cloud FinOps to drive waste out of their cloud usage. 

Video | How Nuro optimized their costs on Google Cloud
In this video, you’ll get an overview of the Google Cloud FinOps framework, a deep dive on cost-optimization best practices, and hear about how startup Nuro AI has adopted their own cost-savings discipline and Cloud FinOps practice. 

Video | How OpenX reduce per unit costs by 60%
In this video, you’ll learn how to establish a cost center of excellence within your cloud practice, explore several cost-optimization recommendations, and hear from OpenX about how they reduced their costs on Google Cloud. 

Case Study | How Sky saved millions with Google Cloud
In this case study, read how a few years into their cloud adoption journey, media and entertainment company, Sky Group discovered over $1.5 million in savings and optimized costs with BigQuery, Compute Engine, and Cloud Storage. 

Case Study | Etsy: Doing more with less cost and infrastructure
In this case study, read how after migrating their data center and ecommerce platform to the cloud, Etsy realized more than 50% savings in compute energy and leveraged committed use discounts (CUDs) to reduce their compute costs by 42%. 

It’s important to remember that FinOps success looks different for different organizations. It’s neither a one-time fix nor a destination reached by way of a single path. But for every organization, success requires small actions, refinement, and continuous improvement. As you leverage Google Cloud FinOps resources and tools, your organization can:

Drive financial accountability and visibility.

Optimize cloud usage and cost efficiency.

Enable cross organizational trust and collaboration.

Prevent cloud-spend sprawl.

Break down departmental silos.

Accelerate innovation. 

Getting started with Cloud FinOps

At Google, we have a team of experts in leading FinOps practices dedicated to helping you create an actionable plan to optimize cloud spend and drive cost efficiency. We’ve created numerous resources to help you get started from any stage in the FinOps journey. 

Whitepaper | Maximize Business Value with Cloud FinOps
This whitepaper outlines steps to help your organization implement FinOps. It details required teams and processes as well as the optimal behaviors, approaches, and outcomes to help maximize your investment on Google Cloud. 

With Google Cloud FinOps, your organization can also accelerate business value in the cloud. To find out more, join us on the Google Cloud Twitter channel twice a month for open Twitter Spaces discussions or reach out to your Google Cloud Sales Representative for a 1:1 discussion.

The past couple of years have been tough for IT organizations. Between headwinds from the COVID-19 pandemic and other macroeconomic factors, teams have been tasked with optimizing their cloud infrastructure footprint while keeping the services that are core and crucial to the business up and running. Today, we’re excited to publish the inaugural State of Kubernetes Cost Optimization report to provide insights and best practices to the Kubernetes community about running cost-efficient clusters in the public cloud without compromising the performance or reliability of their workloads. 

Why we authored this report

The report addresses the intersection of IT organizations looking to reduce costs and the continued rise of Kubernetes adoption across industries. We performed a large-scale analysis of Kubernetes clusters to understand what makes high performers for cost optimization. And now, we’re excited to share our key findings.

How we conducted this research

The report centers around four “golden signals” for Kubernetes cost optimization (not to be confused with the four golden signals of monitoring). These signals, derived from years of collaboration with Kubernetes users, can be used to measure how well you balance workload reliability and cost-optimization of your clusters.

Using these golden signals as a baseline measurement, we looked at large-scale, anonymized data from Google Kubernetes Engine (GKE) clusters, sorting clusters into At Risk, Low, Medium, High,andElite segments using a classification tree weighted with quasi-equal intervals. With these five segments in place, we compared and analyzed how high-performing clusters perform against these golden signals. 

The most important takeaway: set your requests!

Do you know how many workloads are not setting requests in your production clusters? One of our key observations is that many developers are not setting requests for their workloads. And that’s worrisome. Because Kubernetes reclaims resources when node-pressure occurs, it is critical to set requests for workloads that require even a minimum level of reliability. 

Not setting requests implicitly assigns the BestEffort Quality of Service (QoS) class to your Pods. In times of resource scarcity on a given Node — and without any warning or graceful termination — BestEffort Pods are often the first to be killed. This can lead to intermittent performance or reliability issues for your workloads, and can occur depending on Pod resource utilization and where the scheduler places Pods. When these issues arise, they can be difficult to identify and debug. 

To identify workloads that do not set requests, you can use one of the following tools:

If you are running Kubernetes clusters in GKE, use the GKE Workloads at Risk dashboard. This identifies workloads that have not set requests across your fleet of GKE clusters, along with other workloads at a performance or reliability risk based on how they have requests set.

If you want a very simple script to list containers that are not setting requests in any Kubernetes cluster, check out kube-requests-checker.

Once you’ve set requests for your workloads, you can then proceed with workload rightsizing. This golden signal is at the heart of the cost optimization journey; if requests more closely reflect reality, then the decisions Kubernetes makes using requests will be more effective. 

We are seeing this focus on proper setting of resource requests across the community. Ajay Tripathy, the author of the OpenCost project, noted that “setting appropriate resource requests and prioritizing workload rightsizing…is the biggest area of opportunity for OpenCost users.”

In conclusion

No one team alone is responsible for Kubernetes cost optimization — rather, it’s a joint effort that spans developers, platform admins, and even billing and budget owners. The report contains insights and recommendations for each of these personas in its key findings. 

We also know that lessons from these findings are not one-time fixes. Rather, they are continuous practices that you should build into your team culture over time.

To learn more about how we take lessons from the State of Kubernetes Cost Optimization report and build them into GKE, check out the resources below:

A solution guide on best practices for running cost-optimized on GKE

A solution guide on rightsizing your workloads at scale in GKE

A demo video on rightsizing your workloads at scale in GKE

A demo video on using the cloud console for GKE Optimization

An interactive tutorial to try out GKE with sample workloads

And finally, be sure to download the report here.

We will discuss this and other key findings in our @googlecloudtech Twitter Space on August 9th, 2023. Be sure to follow and join us! If you are not able to join, please stay tuned for a series of blog posts where we will review each key finding one by one.

Data analytics? We’ve got you.
The latest on large language models? Covered.
Scaling your hot new idea? We’re here for you.
Need to secure the latest deployment? We can show you how.

Google Cloud Next 2023 has it all: we’re ready to educate, share and innovate with you in the cloud. And today you can see the full list of sessions that will be available to you in-person in San Francisco this August 29-31.

Like always, we’ll be covering a huge range of topics, products and solutions. Wondering how to containerize ALL THE THINGS? We can help, with sessions on serverless best practices, managed Kubernetes and more.

Need to brush up on your AI skills (because who doesn’t)? Look out for exclusive trainings, talks and access to AI experts.

Next also has sessions to help your DevOps teams reduce toil and speed up launches, and advice to help your data scientists operate more quickly and efficiently. Application developers can learn about the latest frameworks and tools, while architects will enjoy hands-on challenges to grow their skills.

Everyone is excited to be back in person, allowing face-to-face time with Google experts, industry practitioners and the latest talent (who you might try to hire?). And you’ll have a chance to meet numerous partners who help expand the Cloud ecosystem and make your journey smoother and easier.

So join us to build, to learn and – most importantly – to network, as in network with actual humans. (Of course, you’ll also be able to learn about Google Cloud Networking…)

Register for Next ’23 and grab a spot for your favorite sessions. We’re looking forward to seeing you in San Francisco!

In today’s business landscape, data availability and integrity are paramount. Disasters, whether natural or man-made, can disrupt operations and pose a significant risk to critical information. 

To address this, today we are introducing Persistent Disk Asynchronous Replication, which enables disaster recovery for Compute Engine workloads by replicating data between Google Cloud regions, providing a sub-1 min Recovery Point Objective (RPO) and low Recovery Time Objective (RTO). 

Simplicity is central to the design of this solution. Replication is managed with a few API calls — there are no required VM agents, no dedicated replication VMs, no constraints on supported guest operating systems, and no performance overhead on the workload. PD Async Replication works at the block infrastructure level, delivering a fast, common infrastructure foundation to protect against disasters. It is designed to be simple to onboard, operate, and monitor, and can be used in combination with other forms of data protection, including disk clones, regional (synchronous replication) disks, and snapshots. Replication is performed directly from a primary disk in the region where you run your workload, to a recovery disk you create in a secondary region.

“Persistent Disk Asynchronous Replication delivers the solid foundation we need for infrastructure-based Disaster Recovery, and helps us meet our regulatory requirements for low RPO/RTO data protection. PD Async Replication supports our full lifecycle of DR testing, failover and failback.” – Nicola Carotti, Head of Cloud Center of Excellence, Intesa Sanpaolo

Setting up replication

You can enable PD Async Replication on your existing PD disks with just two calls in the API, gcloud, or Google Cloud console. First, create a new blank disk in the secondary region with a reference to the primary disk you want to protect. Then, start replication from the primary disk with a reference to the secondary disk. From that point on, data is automatically replicated between disks, typically with an RPO of less than a minute, depending on the change rate of the disk. This setup workflow helps to ensure that an explicit action is taken in both regions before any data is transferred. You don’t need to reconfigure your network to use PD Async Replication.

Once PD Async Replication is running, you can observe the time since last replication and the network bytes sent in Cloud Monitoring. It is up to the operations teams responsible for a workload to decide when a disaster has occurred in the primary region and when to initiate a failover. To begin the failover, stop replication between disks and attach the secondary disk to a VM in the secondary region; you can accomplish this within minutes. To restore the workload back to the primary region after a previous failover to the secondary region, create a new replication pair back to the primary region so that the workload can failback. By replicating data between data centers in different regions, you create resilient data replicas that safeguard against localized disruptions caused by natural disasters or other localized events.

Use consistency groups for complex stateful workloads

In cases where workloads have distributed, dependent data across disks and VM instances, consistency groups allow for coordinated management of dependent data. With consistency groups, PD Async Replication enables simultaneous and atomic data replication by automatically synchronizing the replication period across all disks in the group. This helps to ensure that data is consistent between primary and secondary disks, for successful workload recovery in the event of a disaster.

Testing disaster recovery

To help ensure that recovery procedures will work in a real disaster, we recommend running tests periodically in the secondary region. You can do this without disrupting or disconnecting PD Async Replication by bulk-cloning the secondary disks with a consistency group applied, even while they are receiving new data.

Deploy for high availability and disaster recovery

Regional Persistent Disk (Regional PD) and PD Async Replication are designed to be used together for workloads that require both high availability (HA) and disaster recovery. You can configure a Regional PD to be the primary or secondary async disk, which can work in combination with a zonal disk in the primary or secondary region. In a scenario where an outage occurs in just one zone in the primary region where Regional PD is configured, the disk will continue to replicate from the remaining healthy zone to the secondary region. Note that PD Async Replication is set up between two distinct disks, while each Regional PD is a single disk that stores data in two zones and is attachable in two zones.

Improve your HA and DR posture

By leveraging PD Async Replication and consistency groups, businesses gain robust data protection and recovery capabilities. This approach helps you safeguard critical data against disasters, minimize downtime, provide data consistency, and enhance fault tolerance. With the power of asynchronous replication and consistency groups, you can help build a resilient platform that provides continuous access to data even in the face of unforeseen disruptions.

You can access PD Async Replication through the console, Compute Engine API, gcloud tool, Terraform, and Cloud Monitoring. You can learn more by reading the public documentation.

We introduced Cloud Functions in 2016 with the mission of making it easy for developers to build highly scalable, event-driven applications without having to worry about managing infrastructure. We are committed to providing flexible and fast application development while serving our customer needs for portability in our commitment to our open cloud. 

Our functions-as-a-service (FaaS) platform reflects our commitment to open source, leveraging innovations like our Functions Frameworks and Buildpacks. And today, we are pleased to share that Forrester Research has ranked Google Cloud a Leader in The Forrester Wave™:  Functions-As-A-Service, Q2 2023 report.

Evolving the limits of FaaS 

In the past year, Cloud Functions has pushed the boundaries of what is possible on FaaS, with the general availability of Cloud Functions 2nd gen, which provides powerful and efficient compute options. New features include:

Integration with events from 140+ GCP, third-party, and custom sources using Eventarc

Longer HTTP workloads with up to 60 minutes of execution time

Larger memory and CPU for workloads up to 32 GB of RAM and 8 vCPUs

Concurrent request processing (up to 1,000) with a single instance, drastically reducing cold starts, improving latency, and lowering cost

With Cloud Functions 2nd gen, our container-first approach gives any customer leveraging our FaaS the ability to run it both on-premises and on another cloud platform and easily go from Cloud Functions to Cloud Run and Google Kubernetes Engine. This approach lets you save money, and reduces the need for workload refactoring if you want to move your workloads between platforms.

Evaluation highlights

Google was among nine companies that Forrester invited to participate in its evaluation, and we achieved the highest scores possible in the Vision, Adoption and Observability criteria. Cloud Functions secured the top Roadmap criterion among all vendors evaluated. “[Cloud Functions’] impressive roadmap includes regional footprint expansion and better support for enterprise customers,” writes Forrester.

Support for specialized workloads 

Forrester noted that Cloud Functions is “the only FaaS platform that allows customers to configure memory allocation and CPU completely independently, which shores up its ability to handle specialized workloads.”

Cloud Functions achieved a score of 4.20 in the support for specialized workloads criteria, the highest score out of all the vendors. We believe this reflects our investment in providing a powerful platform for developers that are implementing Google Cloud’s AI and ML offerings such as generating machine learning predictions from Vertex AI models and for creating computer vision applications with Vision API.  In addition, Cloud Functions can be used for real-time data streaming for use cases such as fraud detection. 

“Google Cloud Functions stands out with its flexibility for configuring hardware and instance types,” Forrester states in the report. This showcases Functions’ ability to support content-centric workloads, including its support for Cloud CDN.

Integrated across Google Cloud

Our FaaS platform supports a wide range of use cases across Google Cloud, and our “investments in Firebase as a managed platform for web and mobile applications create additional use cases for the FaaS platform,” Forrester notes in the report. We recently pushed to GA the powerful Firebase Functions 2nd gen, which brings enhanced infrastructure and broader event coverage.  We also launched Firebase Extensions, which are pre-packaged serverless solutions built on Cloud Functions.

In short, Cloud Functions is a powerful tool that you can use to extend and automate your Google Cloud services. Recent innovations such as  BigQuery Remote Functions let you extend BigQuery SQL with your custom code in Cloud Functions for use cases such as enriching BigQuery data in real time using external APIs, calling models in Vertex AI and other machine learning platforms, and for data encryption and tokenization.

We are delighted and honored to be named a Leader in the The Forrester Wave™ Functions-As-A-Service Q2 2023report. If you would like to start trying out Cloud Functions, Google Cloud offers a generous free tier where your first 2 million invocations are free.

Google Maps Platform has enabled a growing number of businesses in India to build informative and engaging geospatial experiences. Today at I/O Connect Bengaluru, we announced Address Descriptors, available in 25 Indian metro areas.With Address Descriptors, you can make it easier for your users to find an address by sharing the most relevant landmarks and area names relative to a geocoded point.

Addresses in many regions, including India, have a unique format. The complexities of an address–with various components like house number, area name, road name and landmark–make it harder to explain and locate. People in India are used to communicating their addresses relative to a landmark or an area name. For example, telling a food delivery driver that their apartment building is behind the local post office or calling their ride-share driver to say they’re waiting opposite a specific college is a common way of communicating addresses. 

Through our conversations with developers in India, we’ve learned that they often try to mirror this way of communicating addresses by building a landmark field into the address form. Address Descriptors shows the most relevant landmarks and area names relative to an address. This experimental feature will help communicate addresses through Reverse Geocoding API at no additional cost. It uses a combination of machine learning signals on proximity, prominence, and visibility–all informed by in-depth on the ground research–to identify and rank the most relevant landmarks and area names relative to an address.

Developers in India have asked for landmark-aware capabilities, and Address Descriptors is the first feature designed for our developers building geospatial experiences for Indian cities. Using the feature, developers can build an address form that includes a pre-selected landmark and area. They also have the option to replace the suggested landmark with another from a stack-ranked list of the five most relevant landmarks.

With Address Descriptors, developers can save time, reduce errors and deliver a more intuitive customer experience. Imagine not having to call your food delivery driver to share a landmark near your apartment because they now have in-app access to the most relevant landmarks and/or area names near the drop-off point.  

Address Descriptors is currently in the experimental stage, which means we’re actively looking for feedback. Read the documentation to learn more and find out how to request the feature. If you have feedback, you can share it with us via email. If the feature is not available in your region, and you think it would be helpful, please let us know.

Just last week we made it more intuitive for developers in India to get started with Google Maps Platform capabilities like Address Descriptors. You’ll now get more context on payment mechanisms, an improved billing experience, and more guidance throughout the process. Starting later this month, a special promotion will be available to eligible new developers in India. To learn more, take a look at the FAQs and sign up to be notified when the offer is live.

We hope these changes will make it easier for developers in India to build more intuitive experiences for their customers. 

Editor’s note: This post was originally published on the Google Workspace blog.

Starting today, LinkedIn learners from around the world can learn digital skills to work collaboratively and boost productivity with Google Workspace, through a curated collection of nine on-demand courses on LinkedIn. This is made possible through Google Cloud’s new partnership with LinkedIn Learning, an engaging skill-building solution that draws on real-time data and insights from the world’s largest professional network. 

Millions of people turn to LinkedIn every day to get the skills they need to transform their careers, with LinkedIn members adding 446 million skills to their profiles over the last year alone. We aim to help even more people develop the digital skills they need to succeed in a modern workplace. 

“We are excited to launch our Google Workspace training content on LinkedIn Learning,” said Jaime Farinos, Interim Head of Curriculum and Content for Cloud Learning Services. “This partnership presents a fantastic opportunity for learners around the world to build high-demand productivity and collaboration skills on a high-quality learning platform.” 

These courses are designed for anyone looking to get more done and boost their effectiveness at work, using the apps in Google Workspace that support more than 3 billion users. The program includes introductions to Google Docs, Gmail, Drive, Calendar, Meet, Chat, Slides, Sheets and Google Sheets – Advanced topics. This online training is taught through hands-on activities, videos, and quizzes. After completing these courses, learners will know how to securely connect, collaborate, and create, and to customize their workflows to meet their unique work style. After finishing these courses, participants can earn a LinkedIn Learning Certificate of Completion. 

To enroll in the Google Workspace End User training courses on LinkedIn Learning, visit https://www.linkedin.com/learning/instructors/google-cloud.

Data and AI provide organizations with endless opportunities to personalize customer experiences, improve supply chain efficiency, reduce operating costs, and drive incremental revenue. However, growing volumes and types of data, disparate analytical tools, limited machine learning skilled resources, and several other challenges all increase the time it takes to see value from data and AI investments.

Google’s Data and AI Cloud is well positioned to help customers navigate these challenges with a unified analytics offering, an open data ecosystem, and built-in machine learning intelligence. Today, we are excited to announce that Forrester Research has recognized Google as a Leader in The Forrester Wave™: Cloud Data Warehouses, Q2 2023. We believe this is a testament to our vision and strong track record of delivering continuous product innovation.

Download the complimentary report: The Forrester Wave™: Cloud Data Warehouses, Q2 2023.

In this report, Forrester evaluated 15 cloud data warehouse (CDW) providers against pre-defined criteria, evaluating them on their current offerings and strategy. In addition to being named a Leader, Google received the highest possible score for 10 different evaluation criteria, including roadmap, partner ecosystem, machine learning (ML) and data science optimization, performance optimization, built-in streaming, in-database analytics and data lake integration.

Tens of thousands of enterprise customers across industries accelerate digital transformation with BigQuery, our cloud data warehouse. ANZ bank uses it to provide high-quality data across the organization to reduce financial crime. BBC relies on BigQuery to provide a reliable and stable service without worrying about scale. And BigCommerce gathers, analyzes and acts on retail data with BigQuery’s help. 

Unified data management

Google Cloud provides a unified data platform that allows organizations to manage every stage of the data lifecycle — from running operational databases for applications, to managing analytical workloads across data warehouses and data lakes, to data-driven decision-making, and to AI and ML. How we’ve architected our platform is unique and enables customers to bring together their data, people, and workloads. 

“BigQuery integrates with Cloud BigTable, Google Cloud Storage, Cloud AI Notebooks, Dataplex, Looker and Google Sheets, allowing users to join data across various systems.”

To unify data pipeline management, we recently announced the general availability of Dataform, which helps data engineers and data analysts of all skill levels build production-grade SQL pipelines in BigQuery while following software engineering best practices such as version control with Git, CI/CD, and code lifecycle management. 

These integrations increase productivity and bring down costs. To help you further manage data cloud costs, we launched BigQuery editions with three pricing tiers — Standard, Enterprise and Enterprise Plus — for you to choose from, with the ability to mix and match for the right price-performance based on your individual workload needs. 

BigQuery’s flexible new pricing options and autoscaling capabilities are driving results for customers. For example, Lytics, a leading customer data platform (CDP), has seen 15% operational improvement and a 20% decrease in costs with BigQuery. 

BigQuery editions feature two innovations. First, compute capacity autoscaling adds fine-grained compute resources in real-time to match the needs of your workload demands, and help you only pay for the compute capacity you use. Second, physical bytes billing pricing allows you to only pay for data storage after it’s been highly compressed. With compressed storage pricing, you can reduce your storage costs even while increasing your data footprint. 

The backbone of these innovations is BigQuery’s unique architecture, which separates storage and compute, allowing BigQuery to scale both storage and compute independently, based on demand. 

Built-in intelligence

BigQuery is more than just a traditional data warehouse. BigQuery ML empowers data analysts to use machine learning through existing SQL tools and skills. It saw over 200% growth in usage in 2022 as customers ran hundreds of millions of prediction and training queries.

“Reference customers like Google’s serverless architecture, high-end scale and performance, geospatial and robust AI/ML capabilities, and support for broad analytical use cases.“ 

We continue to bring the latest advancements in AI technology to make our data cloud services even more intelligent. The new BigQuery ML inference engine, for example, allows you to run predictions not only with popular models formats directly in BigQuery, but also using remotely hosted models and Google’s state-of-the-art pretrained models. 

Then there’s BigLake, which lets you work with data of any type, in any location. This allowed us to deliver object tables, a new table type that provides a structured interface for unstructured data. Object tables let you natively run analytics and ML on images, audio, and documents, changing the game for data teams worldwide, who can now innovate without limits with all their data in one unified environment. Additionally, support for Apache Iceberg through BigLake is now generally available, so you can build a unified data and AI platform where multiple engines spanning analytics and ML/AI runtimes work seamlessly over a single copy of data. 

And now, the models you create in BigQuery using BigQuery ML are now instantly visible in Vertex AI model registry. Once in the model registry, you can then deploy them to Vertex AI endpoints for real-time serving, use Vertex AI pipelines to monitor and train models, and view detailed explanations for your predictions through BigQuery ML and Vertex AI integration. 

Priceline plans to deploy Google Cloud’s generative AI technologies to enable customers  to engage with a new, generative AI-powered chatbot and to receive personalized offerings when searching for hotels worldwide. By leveraging BigQuery, Looker, and Recommendations AI, TIME has gone from zero to two million connected users on time.com, enabling a 360-degree view of their audiences to create deeper relationships with them.

An open data ecosystem 

Google Cloud provides industry-leading integration with open-source and open APIs, providing portability, flexibility, and reducing the risk of vendor lock-in. At the same time, we significantly expanded our partner ecosystem and are increasing investments across many new areas. 

“Google’s partner ecosystem stands out; GCP Marketplace has over 500 partner solutions and offers analytics hub integration for data sharing with thousands of commercial/public datasets.”

Today, more than 900 software partners are building their products using Google’s Data Cloud, and more than 50 data platform partners offer validated integrations through our Google Cloud Ready – BigQuery initiative. Learn how Airports of Thailand , SKY and EVme are adopting Google Cloud’s open data cloud to deliver sustainable, digital-first travel experiences.

We also introduced BigQuery Partner Center, a new user interface in the Google Cloud console that lets you easily discover, try, purchase and use a diverse range of partner products that have been validated through the Google Cloud Ready – BigQuery program. 

We look forward to continuing to innovate and partner with you on your digital transformation journey and are honored to be a Leader in The Forrester Wave™: Cloud Data Warehouses, Q2 2023.

We encourage you to learn more about how organizations are building their data and AI clouds with Google Cloud solutions. And of course, be sure to download the complimentary Forrester Wave™: Cloud Data Warehouses, Q2 2023 report.

In 2022 I published the networking 101 Google Cloud sheet v1. The purpose of the sheet is to help you get a quick lightweight understanding of common networking terms and also services that exist on Google Cloud. In 2023, I have updated the sheet with more topics to include data center terms.

Networking 101 Google Cloud reference sheet v2

The networking 101 Google Cloud sheet version 2 provides you with a quick glance of some common general networking terms. In addition to the areas covered in version 1, there’s a new section for data center networking terms. 

The document is divided into several topic areas and short definitions are placed under each. In addition to the descriptions, you will also find some common questions with answers included in some of the sheets sections. You can check out various topics under the following headings:

Global Network I 

Global Network II

VPC and IP addressing

OSI model and Internet model

TCP, TCP three-way handshake, UDP, QUIC

Packet, Frame and MTU

ARP, RARP, DNS & NAT

Routing, Cloud Router, Dynamic Routing, BGP, MPLS (updated)

Data Center Networking – (new)

Connectivity and Hybrid connectivity (updated)

Network Security (updated)

Traffic handling, Load balancing, Content Delivery (updated)

Troubleshooting & Monitoring (updated)

What happens when you type www.google.com in a browser

Hands-on practice

You can gain hands-on practice on the Google Cloud platform though the following methods:

Sign up for skills boost

Visit codelabs

Jump Starts solution. Click and deploy.

You can see more Google Cloud services online via the interactive online developer cheat sheet. 

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin and send me a message. Be sure to view the document and download the pdf of the Networking 101 Google Cloud reference sheet v2 on GitHub.

In December 2022 we announced the general availability of AlloyDB for PostgreSQL, a fully-managed, PostgreSQL-compatible database service that provides a powerful option for modernizing from legacy, proprietary databases and for scaling existing PostgreSQL workloads. Earlier in 2022, we launched the preview of AlloyDB for PostgreSQL migrations using Database Migration Service (DMS). Today we’re thrilled to announce the general availability of DMS for AlloyDB migrations from PostgreSQL sources. 

As customers look to standardize on AlloyDB for PostgreSQL, they expect a smooth migration path. They need a solution that is easy to set up and use, with no management overhead. Additionally, it should be trusted to move data accurately and securely, while causing minimal disruption to their applications. And that’s what the Database Migration Service offers.

AlloyDB for PostgreSQL provides a range of benefits that makes it an attractive choice for a target database. It unlocks better scalability, higher availability, and faster performance compared to open-source PostgreSQL. In our performance tests, AlloyDB is more than 4x faster for transactional workloads and delivers up to 100x faster analytical queries than standard PostgreSQL. With its full compatibility with PostgreSQL, leveraging this technology is seamless and effortless.

While AlloyDB offers significant performance improvements over traditional PostgreSQL databases, it needs to meet our customers’ migration requirements. DMS provides an easy-to-use migration solution with no management overhead. It also provides accurate and secure data transfers, minimizing disruptions to applications, making it a reliable choice for organizations looking to transition to it as their standard database solution.

What we’ve learned in the preview

Since launching DMS support for AlloyDB migrations, we’ve helped businesses of all sizes improve their database performance, scalability, and availability. Our customers have benefited from features like index advisor and adaptive autovacuum, which have reduced management overhead and improved performance. We’ve also helped customers achieve higher application availability by migrating to AlloyDB’s highly scalable and resilient infrastructure. Full PostgreSQL compatibility and transparent pricing makes it easy to take advantage of this technology. 

DMS provided those customers with a fast, serverless, and secured migration path from PostgreSQL sources to AlloyDB for PostgreSQL, regardless of whether their source was located in on-premises databases, self-managed databases on Google Cloud, or cloud databases such as Amazon Aurora.

“Using DMS was really amazing, the configuration and setup was extremely simple and we were able to migrate our workloads without absorbing downtime. Once we landed in AlloyDB we were amazed how seamless and perfect the experience was where it met all of our requirements including handling our peak traffic,” said Hiroaki Karasawa, Full-stack Node.js development and Cloud Native infrastructure manager at dinii inc. 

“As a cloud-based SaaS ticketing platform, data continuity and system uptime are critical to our operations. We used DMS to move from Amazon Aurora to Cloud SQL in 2021, and at the end of 2022, used it  again to migrate to AlloyDB for PostgreSQL. Our primary concern was minimizing downtime, and this is where DMS proved its mettle,” said Oliver Morgan, CEO at Ventrata. “The beauty of DMS lies in its simplicity and reliability. The simple reality is we would still be on Amazon Aurora if it wasn’t for Google’s DMS, and to have performed two significant database migrations in the course of a year has given our team the flexibility to keep up with the latest and greatest products cloud providers like Google Cloud have to offer.” 

What’s new in the GA version

The GA version offers enhanced security with Customer-Managed Encryption Keys (CMEK), for organizations with strict encryption policies. CMEK is now available for PostgreSQL to AlloyDB migrations, giving you greater control over the keys you use to encrypt data at rest, ensuring the highest level of data protection during the migration process and beyond.

Many Google Cloud services support CMEK (here is the list of supported services). When you protect data in Google Cloud services with CMEK, the CMEK key is within your control.

Migrating to AlloyDB using Database Migration Service

Migrating to AlloyDB is easy with DMS. To start, navigate to the Database Migration page in the Google Cloud console, create a new migration job, and take these five simple steps:

Choose the database type you want to migrate, and see what actions you need to take to set up your source.

Create your source connection profile, which contains information about the source database. The connection profile can later be used for additional migrations. Here you can set up your Customer-Managed Encryption Keys (CMEK).

Create an AlloyDB for PostgreSQL destination cluster that fits your business needs.

Define a connectivity method: DMS offers a guided connectivity path to help you connect.

Test your migration job and get started whenever you’re ready.

Once the migration job starts, DMS takes an initial snapshot of your data, then replicates new changes as they happen. The migration job will continue to replicate the source data until you decide to initiate the cutover. Once Cutover is initiated, the replication stops, and you can redirect your application to use your brand new AlloyDB cluster, which is ready with all your source data.

Learn more and start your database journey 

For more information to help get you started on your migration journey, head over to the documentation or start training with this Database Migration Service Qwiklab. You can get  started with an AlloyDB free trial by navigating to the AlloyDB console and creating your first cluster. And to get started with the new Database Migration Service for PostgreSQL to AlloyDB migrations, simply visit the database migration page in the console.

Editor’s note: Today we hear from PLAID, the company behind KARTE, a customer experience platform (CxP) that helps businesses provide real-time personalized and seamless experiences to their users. PLAID recently re-architected its real-time user analytics engine using Cloud Bigtable, achieving latencies within 10 milliseconds. Read on to learn how they did it. 

Here at PLAID, we rely on many Google Cloud products to manage our data in a wide variety of use cases: AlloyDB for PostgreSQL for relational workloads, BigQuery for enterprise data warehousing, and Bigtable, an enterprise-grade NoSQL database service. Recently, we turned to Bigtable again to help us re-architect our core customer experience platform — a real-time user analytics engine we call “Blitz.” 

To say we ask a lot of Blitz is understatement: our high-traffic environment receives over 100,000 events per second, which Blitz needs to process within a few hundred milliseconds end-to-end. In this blog post, we’ll share how we re-architected Blitz with Bigtable and achieved truly real-time analytics under heavy write traffic. We’ll delve into the architectural choices and implementation techniques that allowed us to accomplish this feat, namely, implementing a highly scalable, low-latency distributed queue.

What we mean by real-time user analytics

But first, let’s discuss what we mean when we say ‘real-time user analytics’. In a real-time user analytics engine, when an event occurs for a user, different actions can be performed based on event history and user-specific statistics. Below is an example of an event data and a rule definition that filters to a specific user for personalized action.

This is pseudo-code for a rule to verify whether “userId-xxx” is a user who is a “member”, has an average purchase price of 10,000 yen or more in a year, had a session count of 10 or more last week, but who purchased little to nothing.

When people talk about real-time analytics, they usually mean near-real-time, where statistics can be seconds or even minutes out-of-date. However, being truly real-time requires that user statistics are always up-to-date, with all past event histories reflected in the results available to the downstream services. The goal is very simple, but it’s technically difficult to keep user statistics up-to-date — especially in a high-traffic environment with over 100,000 events per second and required latency within a few hundred milliseconds. 

Our previous architecture

Our previous analytics engine consisted of two components: a real-time analytics component (Track) and a component that updates user statistics asynchronously (Analyze).

The key points of this architecture are:

In the real-time component (Track), the user statistics generated in advance from the key-value store are read-only, and no writing is performed to it.

In Analyze, streaming jobs roll up events over specific time windows.

However, we wanted to meet the following strict performance requirements for our distributed queue:

High scalability – The queue must be able to scale with high-traffic event numbers. During peak daytime hours, the reference value for the write requests is about 30,000 operations per second, with a write data volume of 300 MiB/s.

Low latency that achieves both fast writes and reads within 10 milliseconds.

But the existing messaging services could not meet both high-scalability and low-latency requirements simultaneously — see Figure 4 below.

Our new real-time analytics architecture

Based on the technical challenges explained above, Figure 5 below shows the architecture after the revamp. Changes from the existing architecture include:

We divided the real-time server into two parts. Initially, the frontend server is responsible for writing events to the distributed queue.

The real-time backend server reads events from the distributed queue and performs analytics.

In order to meet both our scalability and latency goals, we decided to use Bigtable, which we already used as our low-latency key-value store, to implement our distributed queue. Here’s the specific method we used.

Bigtable is mainly known as a key-value store that can achieve latencies in the single-digit milliseconds and be scaled horizontally. What caught our attention is the fact that performing range scans in Bigtable that specify the beginning and end of row keys is also fast.

The specific schema for the distributed queue can be described as follows:

The key point is the event timestamp added at the end. This allows us to perform range scans by specifying the start and end of the event timestamps.

Furthermore, we were able to easily implement a feature to delete old data from the queue by setting a time-to-live (TTL) using Bigtablegarbage collection feature.

By implementing these changes, the real-time analytics backend server was able to ensure that the user’s statistics remain up-to-date, regardless of any unreflected events. 

Additional benefits of Bigtable

Scalability and low latency weren’t the only things that implementing a distributed queue with Bigtable brought to our architecture. It was also  cost-effective and easy to manage.

Cost efficiency

Thanks to the excellent throughput of the SSD storage type and a garbage collection feature that keeps the amount of data constant by deleting old data, we can operate our real-time distributed queue at a much lower cost than we initially anticipated. In comparison to running the same workload on Pub/Sub, our calculations show that we operate at less than half the cost.

Less management overhead

From an infrastructure operation perspective, using the Bigtable auto-scaling feature reduces our operational cost. In case of a sudden increase in requests to the real-time queue, the Bigtable cluster can automatically scale out based on CPU usage. We have been operating this real-time distributed queue for over a year reliably with minimal effort.

Supercharging our real-time analytics engine

In this blog post, we shared our experience in revamping our core real-time user analytics engine, Blitz, using Bigtable. We successfully achieved a consistent view of the user in our real-time analysis engine under high traffic conditions. The key to our success was the innovative use of Bigtable to implement a distributed queue that met both our high scalability and low latency requirements. By leveraging the power of Bigtable low-latency key-value store and its range scan capabilities, we were able to create a horizontally scalable distributed queue with latencies within 10ms.

We hope that our experience and the architectural choices we made can serve as a valuable reference for global engineers looking to enhance their real-time analytics systems. By leveraging the power of Bigtable, we believe that businesses can unlock new levels of performance and consistency in their real-time analytics engines, ultimately leading to better user experiences and more insightful decision-making.

Looking for solutions to up your real-time analytics game? Find out how Bigtable is used for a wide-variety of use cases from content engagement analytics and music recommendations toaudience segmentation, fraud detectionand retail analytics.

“CPG manufacturers and retailers are dependent on product data to understand their markets, inspire innovation, and serve customers, but this is a challenge with the common data sources across the industry,” says Cem Kent, CEO of Harmonya. “Data sets are siloed, products are categorized differently across sources, and the descriptive attributes and characteristics about products are not evolving to reflect industry or consumer perspectives. That’s where Harmonya comes in.”

Harmonya is an all-in-one, AI-powered, product data enrichment, categorization, and insights platform. The company enriches its customers’ product data with deeper attributes and characteristics to power more impactful analytics and decision-making. Harmonya is committed to empowering its customers with greater control over their product analysis and categorization while maintaining a fresh, consistent view of the categories in which they operate. With Harmonya, customers can unlock a wide range of use cases, including: 

category management 

merchandising

innovation

e-commerce content, search and recommendation applications 

Solution approach 

Harmonya’s proprietary technology enriches product data by ingesting information from millions of online product listings and tags products with unique concepts informed by titles, descriptions, structured attributes, consumer reviews, and more. This harmonized data asset empowers brand and retail teams that use product data to unlock new opportunities for their business through a better understanding of what matters most to consumers. We’ll discuss several use cases of this enrichment in detail later in this article.

On top of this enrichment, Harmonya builds robust analytical tools to help uncover insights about the consumer and marketing drivers of in-market performance, improve assortment and merchandising, guide product innovation, engage target audiences more effectively, and categorize products. Fortune 500s and other industry leading CPG manufacturers and retailers rely on Harmonya to enrich their product data and help them compete in a fast-changing marketplace. 

Solution details

Harmonya builds and maintains data pipelines that process massive amounts of data, training and serving machine learning models on top of the BigQuery data warehouse used throughout the organization. BigQuery’s integration with other Google Cloud components and pay-per-use model enables near-limitless scalability for data processing, providing significant value that allows Harmonya to focus on bringing value to its customers. Below is an example illustrating the data access model and deployment model between Harmonya’s internal environment and the customer-facing multi-tenant environment on the right side of the diagram.

The above diagram shows that Harmonya’s stack is split into two separate environments. The first is an internal environment (left side, yellow background) independent of Harmonya’s customers and their data. There, the Harmonya Product Language is created, starting (from left to right) with scheduling data acquisition tasks, querying the current state of the normalized product data vs. the scrape-state DB and deciding which new scrape tasks should be performed. 

Then, Cloud Functions are triggered to gather the relevant data from the web and store the raw results in Cloud Storage. From there, the process of the Harmonya Graph creation takes place, where products are clustered into a consistent view, and relations between products are discovered. Following that process, a set of NLP models are used to extract any meaningful concepts related to the products forming a detailed taxonomy.

The second environment (right side, red background) is a multi-tenant environment where each customer has their own complete separation of resources, ensuring nothing is being shared between any two customers of Harmonya.

The processing starts with a customer sharing raw point-of-sale data point with Harmonya. This data is processed using BigQuery in a streamlined and scalable way and merged with a snapshot of the Harmonya Language, relying on BigQuery’s capability to join data between separate projects. The merged dataset is then processed in Harmonya’s data pipelines, running ML processing to generate customer-specific insights, stored in Cloud SQL for real-time serving in Harmonya’s SaaS based application, running on Node.js and accessed by customers online at https://app.harmonya.com.

BigQuery is an essential tool for Harmonya when working with product data for several reasons:

Scalability: BigQuery is a cloud-based data warehouse that can scale automatically to handle large and complex data sets. This makes it an ideal solution for Harmonya, which needs to manage growing amounts of data without the need for expensive infrastructure investments.

Cost-effective: BigQuery operates on a pay-as-you-go model, which means Harmonya only pays for the resources we use. This makes it a cost-effective solution for startups with limited budgets.

Speed: BigQuery’s high-speed processing of large data sets enables Harmonya to analyze data and make decisions in real-time. This provides a competitive advantage to customers that need to react quickly to market changes. 

Accessibility: BigQuery is accessible through a web-based interface, as well as through a range of programming languages, including SQL and Python. This means that Harmonya’s team, with different levels of technical expertise, can use the tool to analyze and visualize their data integration: BigQuery can integrate with a range of other tools, including data visualization and business intelligence tools, as well as with other Google services. This makes it a versatile tool for Harmonya, which needs to work with data from multiple sources.

Simple data ingestion: BigQuery can ingest data from a variety of sources, including Cloud Storage, Cloud Pub/Sub, Cloud SQL, and more. Harmonya uses these integrations to seamlessly move data from their existing data sources into BigQuery.

On top of that, BigQuery’s flexible scheme allows it to store various data types and query them in a dynamic fashion. Harmonya stores a mixture of structured and semi-structured json files within the same tables in BigQuery, simplifying data ingestion and allowing for a wide variety of use-cases with less data duplication.

Creating meaningful selling stories and trends

Enriching product data unlocks a wide variety of commercial and operational use cases on the brand and retail sides of the commerce chain. A popular use for Harmonya’s enrichment is in creating more impactful and dynamic selling stories. 

Manufacturers rely on retailers to sell their products, so it’s crucial for manufacturers to create unique selling stories that resonate with retailers to stand out in the highly competitive marketplace. Enriching product data with unique attributes and characteristics with Harmonya can help manufacturers tell better selling stories to retailers in several ways:

Deeper understanding of performance drivers: When product data is enriched with unique attributes and characteristics, brands and retailers have a differentiated understanding of in-market dynamics. This helps them make better decisions, identify the true drivers of brand and category performance, and develop more successful strategies to drive growth.

Improved product descriptions: Manufacturers can provide more detailed and accurate product descriptions to retailers when they have a more holistic understanding of how owned and competitive portfolios resonate with consumers. This helps brands and retailers create more compelling product descriptions and marketing materials that drive sales.

Better targeting: Enriched product data can help manufacturers target specific customer segments more effectively based on the combination of first party data and enriched transactional data. By understanding the unique attributes and characteristics of a product and the demographics and behaviors of purchasers, manufacturers and retailers can tailor their outreach and marketing messages to specific customer needs and preferences with unprecedented precision.

Differentiation: Retailers carry many products from various manufacturers, and it’s important for manufacturers to create a unique selling story that sets their product apart from the competition. A unique selling story can make the difference between a single or multiple facings and preferential shelf placement, especially when both the brand and the retailer understand the unique attributes that set those products apart.

Harmonya’s collaborative approach to data enrichment is brought to life via its suite of applications that allow customers to explore and analyze their enhanced datasets.

Detecting trends in product data is challenging for brands and retailers because of the vast amount of information generated by multiple sources, such as sales data, customer feedback, social media, and industry reports. Extracting insights from this data requires powerful analytics tools, expertise in data analysis, and a deep understanding of the market.

This is where Harmonya comes in. Their proprietary algorithms can analyze sales data at the attribute and characteristic level of products, providing granular insights into consumer preferences and trends. Harmonya’s technology can also identify emerging trends and changes in consumer behavior, allowing brands and retailers to adapt their product strategies in real-time.

By leveraging Harmonya’s technology, brands and retailers can gain a competitive edge by staying ahead of the curve in product innovation and positioning. They can also optimize their product portfolios and pricing strategies, improve customer engagement, and ultimately drive revenue growth.

In addition to analyzing sales data at the attribute and characteristic level of products, Harmonya also provides an intuitive, user-friendly interface that allows brands and retailers to visualize their data and trends in a way they haven’t been able to before. Their platform displays data in interactive dashboards and charts, making it simple for users to identify patterns and correlations that may be difficult to spot with traditional analysis methods.

Furthermore, Harmonya’s app simplifies the process of detecting trends by automating the analysis and reporting process, eliminating the need for manual data processing and freeing up valuable time for teams to focus on other strategic initiatives. By leveraging machine learning algorithms, Harmonya’s platform can quickly identify and report on trends, providing brands and retailers with timely insights that enable them to make informed decisions about product development and marketing campaigns.

Overall, Harmonya’s technology and app enable brands and retailers to gain a deeper understanding of their customers’ preferences and behaviors, leading to better product development, pricing strategies, and customer engagement. By providing powerful insights in an easy-to-use interface, Harmonya is helping companies stay ahead of the curve in a constantly evolving market.

Real-world impact

According to a Fortune 50 multi-category manufacturer, “Harmonya achieved 98% accuracy of UPC coding and classification during their engagement. This has enabled us to enrich and automate core data processes around how we manage our product catalog and harmonize external data structures. We are really impressed with the accuracy and quality of their outputs, and we are accelerating the expansion of our partnership to take full advantage of Harmonya’s strategic capabilities more broadly.” 

Conclusion

Google’s data cloud provides a complete platform for building data-driven applications from simplified data ingestion, processing, and storage to powerful analytics, AI, ML, and data sharing capabilities — all integrated with the open, secure, and sustainable Google Cloud platform. With a diverse partner ecosystem, open-source tools, and APIs, Google Cloud can provide technology companies the portability and differentiators they need to serve the next generation of customers. 

To learn more about Harmonya on Google Cloud, visit Harmonya. Click here to learn more about Google Cloud’s Built with BigQuery initiative.

We thank the Google Cloud team members who co-authored the blog: Banruo Yu, Technical Account Manager, Google Cloud, and Christian Williams, Principal Architect, Google Cloud