We face a concerning shortage of cybersecurity professionals. Worldwide, an estimated 3.4 million more cybersecurity experts are needed, with more than 700,000 required in the U.S. alone. The skills gap is even more alarming given the more than 41% increase in the number of cybercrime victims between 2021 and 2022. 

Cybrary is working to address these challenges. Founded in 2015, the EdTech company provides specialized, online, cybersecurity skills development and has enabled more than 3 million learners, ranging from individuals to small businesses and Fortune 1000 organizations, to defend against today’s threats.

Cybrary shares Google Cloud’s belief that a properly-resourced cybersecurity team can detect, investigate, and help stop cyber threats that target businesses and users before attacks cause damage or loss. For instance, the ability to follow national and international privacy and security standards earns trust with customers and partners who expect their data to be protected at every moment. 

Teaming up to expand access to security training and tools 

Cybrary and Google believe that every organization deserves access to the tools and skills that can help secure and protect their data. Our partnership expands the availability of cutting-edge cybersecurity training to more learners..

Cybrary’s Teams plan offers full access to the platform and all of its skills development content, including courses, hands-on labs, assessments, and an admin dashboard where managers can create custom learning paths to and track their team’s progress. The plan is accessible to existing Google Cloud customers who can use Google Cloud credits to purchase licenses for their teams. Participants can use their Google Cloud account to access the platform, simplifying the billing and provisioning elements for Cybrary users.  

Individualizing security training at scale 

Delivery at scale enables Cybrary to offer plans at competitive rates. As many students or working professionals lack the time or resources to complete longer-term degrees or bootcamps, Cybrary offers on-demand and affordable skills development content to upskill. The company puts learners into simulated environments and measures their interactions to ensure they’re developing hands-on skills, build confidence and support flexible continuous learning opportunities. 

Google Cloud and Cybrary possess decades of combined experience in security, and this partnership aims to help more people get access to affordable and high-quality security training required to protect businesses from increasingly complex threats, now and in the future.

Learner and operational analytics powered by Google Cloud

Cybrary makes extensive use of Google Cloud services for networking, databases, and storage to support the agility and performance its business requires. By running on Google Cloud, the company can respond quickly to spikes in demand and avoid latency issues that could impact the quality of services that customers enjoy.

Given the nature of Cybrary’s work and its objectives to scale further, it needed a solution for its internal analytics needs. The company recognizes that access to the training platform is critical, and partnered with Google Cloud to ensure scalability and reliability that can support the number of learners needed to meet this education challenge.”

In addition, Cybrary usesLooker for business intelligence, data applications, and embedded analytics. By exploring, sharing, and visualizing its data, Cybrary can make better, faster business decisions. The company has also tapped into services from other third-party tech partners on Google Cloud Marketplace.

This combines to position Cybrary for success in providing an ever-growing number of learners with security training offerings. 

Visit the Cybrary Partner Profile on the Google Cloud Marketplace to connect with their team and learn how they can help your organization close skills gaps and succeed against evolving threats.

“CPG manufacturers and retailers are dependent on product data to understand their markets, inspire innovation, and serve customers, but this is a challenge with the common data sources across the industry,” says Cem Kent, CEO of Harmonya. “Data sets are siloed, products are categorized differently across sources, and the descriptive attributes and characteristics about products are not evolving to reflect industry or consumer perspectives. That’s where Harmonya comes in.”

Harmonya is an all-in-one, AI-powered, product data enrichment, categorization, and insights platform. The company enriches its customers’ product data with deeper attributes and characteristics to power more impactful analytics and decision-making. Harmonya is committed to empowering its customers with greater control over their product analysis and categorization while maintaining a fresh, consistent view of the categories in which they operate. With Harmonya, customers can unlock a wide range of use cases, including: 

category management 

merchandising

innovation

e-commerce content, search and recommendation applications 

Solution approach 

Harmonya’s proprietary technology enriches product data by ingesting information from millions of online product listings and tags products with unique concepts informed by titles, descriptions, structured attributes, consumer reviews, and more. This harmonized data asset empowers brand and retail teams that use product data to unlock new opportunities for their business through a better understanding of what matters most to consumers. We’ll discuss several use cases of this enrichment in detail later in this article.

On top of this enrichment, Harmonya builds robust analytical tools to help uncover insights about the consumer and marketing drivers of in-market performance, improve assortment and merchandising, guide product innovation, engage target audiences more effectively, and categorize products. Fortune 500s and other industry leading CPG manufacturers and retailers rely on Harmonya to enrich their product data and help them compete in a fast-changing marketplace. 

Solution details

Harmonya builds and maintains data pipelines that process massive amounts of data, training and serving machine learning models on top of the BigQuery data warehouse used throughout the organization. BigQuery’s integration with other Google Cloud components and pay-per-use model enables near-limitless scalability for data processing, providing significant value that allows Harmonya to focus on bringing value to its customers. Below is an example illustrating the data access model and deployment model between Harmonya’s internal environment and the customer-facing multi-tenant environment on the right side of the diagram.

The above diagram shows that Harmonya’s stack is split into two separate environments. The first is an internal environment (left side, yellow background) independent of Harmonya’s customers and their data. There, the Harmonya Product Language is created, starting (from left to right) with scheduling data acquisition tasks, querying the current state of the normalized product data vs. the scrape-state DB and deciding which new scrape tasks should be performed. 

Then, Cloud Functions are triggered to gather the relevant data from the web and store the raw results in Cloud Storage. From there, the process of the Harmonya Graph creation takes place, where products are clustered into a consistent view, and relations between products are discovered. Following that process, a set of NLP models are used to extract any meaningful concepts related to the products forming a detailed taxonomy.

The second environment (right side, red background) is a multi-tenant environment where each customer has their own complete separation of resources, ensuring nothing is being shared between any two customers of Harmonya.

The processing starts with a customer sharing raw point-of-sale data point with Harmonya. This data is processed using BigQuery in a streamlined and scalable way and merged with a snapshot of the Harmonya Language, relying on BigQuery’s capability to join data between separate projects. The merged dataset is then processed in Harmonya’s data pipelines, running ML processing to generate customer-specific insights, stored in Cloud SQL for real-time serving in Harmonya’s SaaS based application, running on Node.js and accessed by customers online at https://app.harmonya.com.

BigQuery is an essential tool for Harmonya when working with product data for several reasons:

Scalability: BigQuery is a cloud-based data warehouse that can scale automatically to handle large and complex data sets. This makes it an ideal solution for Harmonya, which needs to manage growing amounts of data without the need for expensive infrastructure investments.

Cost-effective: BigQuery operates on a pay-as-you-go model, which means Harmonya only pays for the resources we use. This makes it a cost-effective solution for startups with limited budgets.

Speed: BigQuery’s high-speed processing of large data sets enables Harmonya to analyze data and make decisions in real-time. This provides a competitive advantage to customers that need to react quickly to market changes. 

Accessibility: BigQuery is accessible through a web-based interface, as well as through a range of programming languages, including SQL and Python. This means that Harmonya’s team, with different levels of technical expertise, can use the tool to analyze and visualize their data integration: BigQuery can integrate with a range of other tools, including data visualization and business intelligence tools, as well as with other Google services. This makes it a versatile tool for Harmonya, which needs to work with data from multiple sources.

Simple data ingestion: BigQuery can ingest data from a variety of sources, including Cloud Storage, Cloud Pub/Sub, Cloud SQL, and more. Harmonya uses these integrations to seamlessly move data from their existing data sources into BigQuery.

On top of that, BigQuery’s flexible scheme allows it to store various data types and query them in a dynamic fashion. Harmonya stores a mixture of structured and semi-structured json files within the same tables in BigQuery, simplifying data ingestion and allowing for a wide variety of use-cases with less data duplication.

Creating meaningful selling stories and trends

Enriching product data unlocks a wide variety of commercial and operational use cases on the brand and retail sides of the commerce chain. A popular use for Harmonya’s enrichment is in creating more impactful and dynamic selling stories. 

Manufacturers rely on retailers to sell their products, so it’s crucial for manufacturers to create unique selling stories that resonate with retailers to stand out in the highly competitive marketplace. Enriching product data with unique attributes and characteristics with Harmonya can help manufacturers tell better selling stories to retailers in several ways:

Deeper understanding of performance drivers: When product data is enriched with unique attributes and characteristics, brands and retailers have a differentiated understanding of in-market dynamics. This helps them make better decisions, identify the true drivers of brand and category performance, and develop more successful strategies to drive growth.

Improved product descriptions: Manufacturers can provide more detailed and accurate product descriptions to retailers when they have a more holistic understanding of how owned and competitive portfolios resonate with consumers. This helps brands and retailers create more compelling product descriptions and marketing materials that drive sales.

Better targeting: Enriched product data can help manufacturers target specific customer segments more effectively based on the combination of first party data and enriched transactional data. By understanding the unique attributes and characteristics of a product and the demographics and behaviors of purchasers, manufacturers and retailers can tailor their outreach and marketing messages to specific customer needs and preferences with unprecedented precision.

Differentiation: Retailers carry many products from various manufacturers, and it’s important for manufacturers to create a unique selling story that sets their product apart from the competition. A unique selling story can make the difference between a single or multiple facings and preferential shelf placement, especially when both the brand and the retailer understand the unique attributes that set those products apart.

Harmonya’s collaborative approach to data enrichment is brought to life via its suite of applications that allow customers to explore and analyze their enhanced datasets.

Detecting trends in product data is challenging for brands and retailers because of the vast amount of information generated by multiple sources, such as sales data, customer feedback, social media, and industry reports. Extracting insights from this data requires powerful analytics tools, expertise in data analysis, and a deep understanding of the market.

This is where Harmonya comes in. Their proprietary algorithms can analyze sales data at the attribute and characteristic level of products, providing granular insights into consumer preferences and trends. Harmonya’s technology can also identify emerging trends and changes in consumer behavior, allowing brands and retailers to adapt their product strategies in real-time.

By leveraging Harmonya’s technology, brands and retailers can gain a competitive edge by staying ahead of the curve in product innovation and positioning. They can also optimize their product portfolios and pricing strategies, improve customer engagement, and ultimately drive revenue growth.

In addition to analyzing sales data at the attribute and characteristic level of products, Harmonya also provides an intuitive, user-friendly interface that allows brands and retailers to visualize their data and trends in a way they haven’t been able to before. Their platform displays data in interactive dashboards and charts, making it simple for users to identify patterns and correlations that may be difficult to spot with traditional analysis methods.

Furthermore, Harmonya’s app simplifies the process of detecting trends by automating the analysis and reporting process, eliminating the need for manual data processing and freeing up valuable time for teams to focus on other strategic initiatives. By leveraging machine learning algorithms, Harmonya’s platform can quickly identify and report on trends, providing brands and retailers with timely insights that enable them to make informed decisions about product development and marketing campaigns.

Overall, Harmonya’s technology and app enable brands and retailers to gain a deeper understanding of their customers’ preferences and behaviors, leading to better product development, pricing strategies, and customer engagement. By providing powerful insights in an easy-to-use interface, Harmonya is helping companies stay ahead of the curve in a constantly evolving market.

Real-world impact

According to a Fortune 50 multi-category manufacturer, “Harmonya achieved 98% accuracy of UPC coding and classification during their engagement. This has enabled us to enrich and automate core data processes around how we manage our product catalog and harmonize external data structures. We are really impressed with the accuracy and quality of their outputs, and we are accelerating the expansion of our partnership to take full advantage of Harmonya’s strategic capabilities more broadly.” 

Conclusion

Google’s data cloud provides a complete platform for building data-driven applications from simplified data ingestion, processing, and storage to powerful analytics, AI, ML, and data sharing capabilities — all integrated with the open, secure, and sustainable Google Cloud platform. With a diverse partner ecosystem, open-source tools, and APIs, Google Cloud can provide technology companies the portability and differentiators they need to serve the next generation of customers. 

To learn more about Harmonya on Google Cloud, visit Harmonya. Click here to learn more about Google Cloud’s Built with BigQuery initiative.

We thank the Google Cloud team members who co-authored the blog: Banruo Yu, Technical Account Manager, Google Cloud, and Christian Williams, Principal Architect, Google Cloud

Editor’s note: Today we hear from PLAID, the company behind KARTE, a customer experience platform (CxP) that helps businesses provide real-time personalized and seamless experiences to their users. PLAID recently re-architected its real-time user analytics engine using Cloud Bigtable, achieving latencies within 10 milliseconds. Read on to learn how they did it. 

Here at PLAID, we rely on many Google Cloud products to manage our data in a wide variety of use cases: AlloyDB for PostgreSQL for relational workloads, BigQuery for enterprise data warehousing, and Bigtable, an enterprise-grade NoSQL database service. Recently, we turned to Bigtable again to help us re-architect our core customer experience platform — a real-time user analytics engine we call “Blitz.” 

To say we ask a lot of Blitz is understatement: our high-traffic environment receives over 100,000 events per second, which Blitz needs to process within a few hundred milliseconds end-to-end. In this blog post, we’ll share how we re-architected Blitz with Bigtable and achieved truly real-time analytics under heavy write traffic. We’ll delve into the architectural choices and implementation techniques that allowed us to accomplish this feat, namely, implementing a highly scalable, low-latency distributed queue.

What we mean by real-time user analytics

But first, let’s discuss what we mean when we say ‘real-time user analytics’. In a real-time user analytics engine, when an event occurs for a user, different actions can be performed based on event history and user-specific statistics. Below is an example of an event data and a rule definition that filters to a specific user for personalized action.

This is pseudo-code for a rule to verify whether “userId-xxx” is a user who is a “member”, has an average purchase price of 10,000 yen or more in a year, had a session count of 10 or more last week, but who purchased little to nothing.

When people talk about real-time analytics, they usually mean near-real-time, where statistics can be seconds or even minutes out-of-date. However, being truly real-time requires that user statistics are always up-to-date, with all past event histories reflected in the results available to the downstream services. The goal is very simple, but it’s technically difficult to keep user statistics up-to-date — especially in a high-traffic environment with over 100,000 events per second and required latency within a few hundred milliseconds. 

Our previous architecture

Our previous analytics engine consisted of two components: a real-time analytics component (Track) and a component that updates user statistics asynchronously (Analyze).

The key points of this architecture are:

In the real-time component (Track), the user statistics generated in advance from the key-value store are read-only, and no writing is performed to it.

In Analyze, streaming jobs roll up events over specific time windows.

However, we wanted to meet the following strict performance requirements for our distributed queue:

High scalability – The queue must be able to scale with high-traffic event numbers. During peak daytime hours, the reference value for the write requests is about 30,000 operations per second, with a write data volume of 300 MiB/s.

Low latency that achieves both fast writes and reads within 10 milliseconds.

But the existing messaging services could not meet both high-scalability and low-latency requirements simultaneously — see Figure 4 below.

Our new real-time analytics architecture

Based on the technical challenges explained above, Figure 5 below shows the architecture after the revamp. Changes from the existing architecture include:

We divided the real-time server into two parts. Initially, the frontend server is responsible for writing events to the distributed queue.

The real-time backend server reads events from the distributed queue and performs analytics.

In order to meet both our scalability and latency goals, we decided to use Bigtable, which we already used as our low-latency key-value store, to implement our distributed queue. Here’s the specific method we used.

Bigtable is mainly known as a key-value store that can achieve latencies in the single-digit milliseconds and be scaled horizontally. What caught our attention is the fact that performing range scans in Bigtable that specify the beginning and end of row keys is also fast.

The specific schema for the distributed queue can be described as follows:

The key point is the event timestamp added at the end. This allows us to perform range scans by specifying the start and end of the event timestamps.

Furthermore, we were able to easily implement a feature to delete old data from the queue by setting a time-to-live (TTL) using Bigtablegarbage collection feature.

By implementing these changes, the real-time analytics backend server was able to ensure that the user’s statistics remain up-to-date, regardless of any unreflected events. 

Additional benefits of Bigtable

Scalability and low latency weren’t the only things that implementing a distributed queue with Bigtable brought to our architecture. It was also  cost-effective and easy to manage.

Cost efficiency

Thanks to the excellent throughput of the SSD storage type and a garbage collection feature that keeps the amount of data constant by deleting old data, we can operate our real-time distributed queue at a much lower cost than we initially anticipated. In comparison to running the same workload on Pub/Sub, our calculations show that we operate at less than half the cost.

Less management overhead

From an infrastructure operation perspective, using the Bigtable auto-scaling feature reduces our operational cost. In case of a sudden increase in requests to the real-time queue, the Bigtable cluster can automatically scale out based on CPU usage. We have been operating this real-time distributed queue for over a year reliably with minimal effort.

Supercharging our real-time analytics engine

In this blog post, we shared our experience in revamping our core real-time user analytics engine, Blitz, using Bigtable. We successfully achieved a consistent view of the user in our real-time analysis engine under high traffic conditions. The key to our success was the innovative use of Bigtable to implement a distributed queue that met both our high scalability and low latency requirements. By leveraging the power of Bigtable low-latency key-value store and its range scan capabilities, we were able to create a horizontally scalable distributed queue with latencies within 10ms.

We hope that our experience and the architectural choices we made can serve as a valuable reference for global engineers looking to enhance their real-time analytics systems. By leveraging the power of Bigtable, we believe that businesses can unlock new levels of performance and consistency in their real-time analytics engines, ultimately leading to better user experiences and more insightful decision-making.

Looking for solutions to up your real-time analytics game? Find out how Bigtable is used for a wide-variety of use cases from content engagement analytics and music recommendations toaudience segmentation, fraud detectionand retail analytics.

In December 2022 we announced the general availability of AlloyDB for PostgreSQL, a fully-managed, PostgreSQL-compatible database service that provides a powerful option for modernizing from legacy, proprietary databases and for scaling existing PostgreSQL workloads. Earlier in 2022, we launched the preview of AlloyDB for PostgreSQL migrations using Database Migration Service (DMS). Today we’re thrilled to announce the general availability of DMS for AlloyDB migrations from PostgreSQL sources. 

As customers look to standardize on AlloyDB for PostgreSQL, they expect a smooth migration path. They need a solution that is easy to set up and use, with no management overhead. Additionally, it should be trusted to move data accurately and securely, while causing minimal disruption to their applications. And that’s what the Database Migration Service offers.

AlloyDB for PostgreSQL provides a range of benefits that makes it an attractive choice for a target database. It unlocks better scalability, higher availability, and faster performance compared to open-source PostgreSQL. In our performance tests, AlloyDB is more than 4x faster for transactional workloads and delivers up to 100x faster analytical queries than standard PostgreSQL. With its full compatibility with PostgreSQL, leveraging this technology is seamless and effortless.

While AlloyDB offers significant performance improvements over traditional PostgreSQL databases, it needs to meet our customers’ migration requirements. DMS provides an easy-to-use migration solution with no management overhead. It also provides accurate and secure data transfers, minimizing disruptions to applications, making it a reliable choice for organizations looking to transition to it as their standard database solution.

What we’ve learned in the preview

Since launching DMS support for AlloyDB migrations, we’ve helped businesses of all sizes improve their database performance, scalability, and availability. Our customers have benefited from features like index advisor and adaptive autovacuum, which have reduced management overhead and improved performance. We’ve also helped customers achieve higher application availability by migrating to AlloyDB’s highly scalable and resilient infrastructure. Full PostgreSQL compatibility and transparent pricing makes it easy to take advantage of this technology. 

DMS provided those customers with a fast, serverless, and secured migration path from PostgreSQL sources to AlloyDB for PostgreSQL, regardless of whether their source was located in on-premises databases, self-managed databases on Google Cloud, or cloud databases such as Amazon Aurora.

“Using DMS was really amazing, the configuration and setup was extremely simple and we were able to migrate our workloads without absorbing downtime. Once we landed in AlloyDB we were amazed how seamless and perfect the experience was where it met all of our requirements including handling our peak traffic,” said Hiroaki Karasawa, Full-stack Node.js development and Cloud Native infrastructure manager at dinii inc. 

“As a cloud-based SaaS ticketing platform, data continuity and system uptime are critical to our operations. We used DMS to move from Amazon Aurora to Cloud SQL in 2021, and at the end of 2022, used it  again to migrate to AlloyDB for PostgreSQL. Our primary concern was minimizing downtime, and this is where DMS proved its mettle,” said Oliver Morgan, CEO at Ventrata. “The beauty of DMS lies in its simplicity and reliability. The simple reality is we would still be on Amazon Aurora if it wasn’t for Google’s DMS, and to have performed two significant database migrations in the course of a year has given our team the flexibility to keep up with the latest and greatest products cloud providers like Google Cloud have to offer.” 

What’s new in the GA version

The GA version offers enhanced security with Customer-Managed Encryption Keys (CMEK), for organizations with strict encryption policies. CMEK is now available for PostgreSQL to AlloyDB migrations, giving you greater control over the keys you use to encrypt data at rest, ensuring the highest level of data protection during the migration process and beyond.

Many Google Cloud services support CMEK (here is the list of supported services). When you protect data in Google Cloud services with CMEK, the CMEK key is within your control.

Migrating to AlloyDB using Database Migration Service

Migrating to AlloyDB is easy with DMS. To start, navigate to the Database Migration page in the Google Cloud console, create a new migration job, and take these five simple steps:

Choose the database type you want to migrate, and see what actions you need to take to set up your source.

Create your source connection profile, which contains information about the source database. The connection profile can later be used for additional migrations. Here you can set up your Customer-Managed Encryption Keys (CMEK).

Create an AlloyDB for PostgreSQL destination cluster that fits your business needs.

Define a connectivity method: DMS offers a guided connectivity path to help you connect.

Test your migration job and get started whenever you’re ready.

Once the migration job starts, DMS takes an initial snapshot of your data, then replicates new changes as they happen. The migration job will continue to replicate the source data until you decide to initiate the cutover. Once Cutover is initiated, the replication stops, and you can redirect your application to use your brand new AlloyDB cluster, which is ready with all your source data.

Learn more and start your database journey 

For more information to help get you started on your migration journey, head over to the documentation or start training with this Database Migration Service Qwiklab. You can get  started with an AlloyDB free trial by navigating to the AlloyDB console and creating your first cluster. And to get started with the new Database Migration Service for PostgreSQL to AlloyDB migrations, simply visit the database migration page in the console.

In 2022 I published the networking 101 Google Cloud sheet v1. The purpose of the sheet is to help you get a quick lightweight understanding of common networking terms and also services that exist on Google Cloud. In 2023, I have updated the sheet with more topics to include data center terms.

Networking 101 Google Cloud reference sheet v2

The networking 101 Google Cloud sheet version 2 provides you with a quick glance of some common general networking terms. In addition to the areas covered in version 1, there’s a new section for data center networking terms. 

The document is divided into several topic areas and short definitions are placed under each. In addition to the descriptions, you will also find some common questions with answers included in some of the sheets sections. You can check out various topics under the following headings:

Global Network I 

Global Network II

VPC and IP addressing

OSI model and Internet model

TCP, TCP three-way handshake, UDP, QUIC

Packet, Frame and MTU

ARP, RARP, DNS & NAT

Routing, Cloud Router, Dynamic Routing, BGP, MPLS (updated)

Data Center Networking – (new)

Connectivity and Hybrid connectivity (updated)

Network Security (updated)

Traffic handling, Load balancing, Content Delivery (updated)

Troubleshooting & Monitoring (updated)

What happens when you type www.google.com in a browser

Hands-on practice

You can gain hands-on practice on the Google Cloud platform though the following methods:

Sign up for skills boost

Visit codelabs

Jump Starts solution. Click and deploy.

You can see more Google Cloud services online via the interactive online developer cheat sheet. 

Want to ask a question, find out more or share a thought? Please connect with me on Linkedin and send me a message. Be sure to view the document and download the pdf of the Networking 101 Google Cloud reference sheet v2 on GitHub.