With products that serve over 150 million students globally, Google has a responsibility to meet the rapidly evolving needs of students, educators, and administrators. Google for Education and Google Cloud work together to partner with the most innovative EdTech companies to help to make learning more personal, safer, and accessible in the classroom and beyond. 

GoGuardian, a leading provider of digital learning solutions, has partnered with Google for Education over the last 7 years to provide safer and more innovative classrooms with solutions available on Chrome, such as GoGuardian Admin, which leverages advanced filtering to provide thoughtful guardrails and helps keep students safe and on task. Recently, the partnership expanded to include close collaboration with Google Cloud to bring the latest advancements in AI to GoGuardian’s products. These advancements in AI will enable GoGuardian to more efficiently address evolving needs in the classroom, such as students’ need for targeted, in-the-moment support and individualized learning content that is personally relevant and evolves alongside each learner.   

GoGuardian and Google believe that everyone—educators and learners at every age and stage—deserve the tools and skills that set them up for success in building the future they want for themselves. Shortly after being founded in 2014, GoGuardian was serving more than 30,000 students. Today, that number is over 27 million — more than half of all public K-12 students in the U.S.

Let’s take a closer look at how schools around the U.S. are utilizing technology solutions from the GoGuardian and Google partnership.

Advancing learning for everyone

Close partnership between Google and GoGuardian directly benefits schools that use their seamlessly integrated tools in the classroom day to day. Adding Google Cloud to their existing Google integrations allows GoGuardian the opportunity to introduce new GenAI tools that can help educators build personalized learning content and resources faster for their classrooms. 

“Effective teaching and learning depend on many interrelated activities,” said Sharad Gupta, GoGuardian’s Chief Product Officer. Schools and districts are tasked with challenges ranging from classroom management and engagement to student performance and safety to emotional well-being. Our portfolio supports all these critical areas, and our partnership with Google helps ensure we can deliver at the level schools need to succeed.”

With GoGuardian and Google tools, educators can create active learning experiences that foster self-expression, deepen student-teacher relationships, keep learning safe, and amplify student voices. Together, GoGuardian and Google help schools tackle some of education’s toughest challenges, such as creating inclusive learning environments that have been shown to result in statistically significant improvements in classroom achievement and graduation rates.

Partnering to bring rapid innovation to millions of teachers, students, and administrators

“Deep integration across a number of Google for Education tools has helped shape our company’s growth and success, and the opportunity to expand our work with Google to bring the best of AI to our users is an exciting evolution,” said Advait Shinde, GoGuardian’s CEO and Co-Founder.

GoGuardian ultimately chose Google Cloud as a partner thanks to clear momentum with AI tools and a shared commitment to innovate responsibly with the needs of the education community in mind. Product and engineering teams from GoGuardian and Google have experience working together on product integrations that support the development of next-generation educational applications in a cost-effective manner. This experience enables both partners to rapidly iterate on and validate AI use cases in education.   

Recently, Google and GoGuardian product teams have collaborated to explore Google Cloud’s suite of generative AI products, such asVertex AI, to build education solutions that were not previously feasible. By working closely together, product teams are able to rapidly prototype, validate, and bring new solutions to market with a focus on efficacy. As teaching and learning continue to evolve, GoGuardian’s digital learning tools have adapted and grown alongside educators, empowering them to foster creativity and engagement in every lesson. Google Cloud will be a key partner in helping GoGuardian address the challenges of today while envisioning the solutions of tomorrow.

GoGuardian: A suite of tools for more effective, engaging learning 

GoGuardian helps schools create more effective online learning environments. With GoGuardian, schools can encourage student engagement with materials and track individual performance.

“Our goal is to enable every student to realize their full potential with the help of our learning platform,” said Shinde. “We want learners to be ready to tackle any challenge that’s in front of them. By integrating the best in learning and science technology into our platform, we can help transform teaching and learning.”

GoGuardian’s suite of tools include: 

GoGuardian Admin to help students remain safer and more productive on their learning journey

GoGuardian Teacher for more effective instruction in digital learning environments

Beacon to assist with student mental health and safety management

Giant Steps for gamified student collaboration and independent practice in K-12 classrooms and beyond

Pear Deck to transform presentations into active learning experiences to engage all learners

Edulastic for more effective assessments, access to instant data-driven insights, and standards mastery tracked for a more engaged classroom 

TutorMe for high-quality, on-demand tutoring across all subjects and all grade levels.

These solutions support GoGuardian’s mission, and today serve 740,000 educators in 11,500 schools and 7,400 districts, including all 25 of the largest school districts in the U.S. 

* * *

For more on Google Cloud’s latest solutions in EdTech, visit our Google Cloud for EdTech solutions page and learn about how we’re making education more personal, safe and accessible. If you’re an EdTech startup that’s looking to access a large network of industry experts and deepen your partnership with Google Cloud, apply to StartEd, sponsored by Google Cloud, today.

As businesses continue to grapple with the unique storage and access demands of data-intensive AI workloads, they’re looking to the cloud to deliver highly capable, cost-effective, and easily manageable storage solutions. 

To deliver the right cloud storage solution for the right application, today we’re launching three new solutions: 

Parallelstore, a parallel file system for demanding AI and HPC applications that use GPU/TPUs

Cloud Storage FUSE for AI applications that require file system semantics

Google Cloud NetApp Volumes, for enterprise applications running in the cloud 

Parallelstore

Google Cloud customers training AI models often turn to GPU/TPUs to get the performance they need. But let’s face it: those resources are limited and an infrastructure asset you want to fully utilize. Google Cloud Parallelstore, now in private Preview, helps you stop wasting precious GPU resources while you wait for storage I/O by providing a high-performing parallel file storage solution for AI/ML and HPC workloads. 

By keeping your GPUs saturated with the data you need to optimize the AI/ML training phase, Parallelstore can help you significantly reduce — or even eliminate — costs associated with idle GPUs. 

Based on the next-generation Intel DAOS architecture, all compute nodes in a Parallelstore environment have equal access to storage, so VMs can get immediate access to their data. With up to 6.3x read throughput performance compared to competitive Lustre Scratch offerings., Parallelstore is well suited for cloud-based applications that require extremely high performance (IOPS and throughput) and ultra low latency. 

As companies seek to migrate their data across pre-processing, model development, training, and checkpoint stages, Parallelstore is a differentiated high performance solution for when they need to push the limits of I/O patterns, file sizes, latency, and throughput. For high-performance AI/ML workloads, Parallelstore can be configured to eliminate waste on unnecessary storage so you’re not caught flat-footed with a solution that can’t handle your workload requirements. 

For more details, check out our Parallelstore web page.

Cloud Storage FUSE

Cloud Storage FUSE lets you mount and access Cloud Storage buckets as local file systems. With Cloud Storage FUSE you get a smooth experience for AI applications that need file system semantics to store and access training data, models, and checkpoints, while preserving the scale, affordability, performance, and simplicity of Cloud Storage. 

Now generally available as a first-party Google Cloud offering, Cloud Storage FUSE is focused on delivering four key benefits:

Compatibility: Because Cloud Storage FUSE enables objects in Cloud Storage buckets to be accessed as files mounted as a local file system, it removes the need for refactoring applications to call cloud-specific APIs.

Reliability: As a first-party offering, Cloud Storage FUSE is integrated with the official Go Cloud Storage client library, and has been specifically validated for PyTorch and TensorFlow at high scale and long duration using ViT DINO and ResNet ML models. 

Performance: Because Cloud Storage FUSE lets developers treat a Cloud Storage bucket as a local file system, there’s no delay caused by moving data from sources to GPUs and TPUs, eliminating much of the resource idle time that read-heavy machine learning workloads incur. OpenX, a global adtech company, reduced pod startup time by 40% by using Cloud Storage FUSE with Google Kubernetes Engine (GKE). Previously, OpenX relied on a homegrown solution to fetch data files from Cloud Storage at pod startup.

Portability: You can deploy Cloud Storage FUSE in your own environment as a Linux package, using pre-built Google ML images, as part of the Vertex AI platform, or as part of a turn-key integration with GKE through the Cloud Storage Fuse CSI driver.

For more details, check out today’s blog post on our new first-party Cloud Storage FUSE offering, or the Cloud Storage FUSE product page for full documentation. 

Google Cloud NetApp Volumes

Many enterprise customers that have architected their applications on top of NetApp storage arrays now want to migrate those workloads to the cloud. NetApp Volumes simplifies the process by providing a fully Google-managed, high-performance file storage service designed specifically for demanding enterprise applications and workloads. 

NetApp Volumes helps customers unlock the full potential of today’s most demanding workloads thanks to:

The capability to increase volumes from 100GiB to 100TiB for maximum scalability 

The ability to implement ONTAP data management for hybrid workloads, providing a familiar management interface for longtime NetApp customers 

The power and flexibility to run either Windows or Linux applications as virtual machines without refactoring

For more information, check out our blog post announcing NetApp Volumes or the NetApp Volumes product page.

The right storage options

AI has become instrumental in automating data management. As you adapt to these workloads, we want to make the process as seamless as possible with options tailored to your training model needs. With the right storage solution, you can simplify operations, unlock innovation, reduce costs, and position your business to meet the changing needs of your workloads and applications.

Start using Cloud Storage FUSE or NetApp Volumes today by visiting the Google Cloud console. To begin using Parallelstore, contact your Google account manager.

In this blog, we hear from Codeway, an app development company that builds innovative apps and games using AI, and how it uses the recently launched Cloud Storage FUSE solution as part of its generative AI workflow on Google Kubernetes Engine (GKE) and Cloud Storage. Cloud Storage FUSE allows objects in Cloud Storage buckets to  be accessed as files mounted as a local file system, providing file-system semantics for data stored in Cloud Storage buckets, simplifying deployments and reducing idle time for scarce TPU and GPU resources. In addition, the new first-party Cloud Storage FUSE CSI driver for GKE allows Kubernetes applications to mount Cloud Storage using the familiar Kubernetes API, and is offered as a turn-key deployment managed by Google Cloud. For more on the latest version of Cloud Storage FUSE, read today’s blog.

At Codeway, we build highly scalable mobile apps and games, and our products reach over 115 million users in six continents. We curate bold ideas and transform them into fast moving products — from concept, through development, up to market expansion — and to do so, we’ve incorporated AI into several of our products, including Voi, an AI-based avatar portrait maker that turns uploaded portraits into customizable AI-generated avatars.

The process of generating a custom avatar happens on GKE and Cloud Storage, our source of truth for everything from the user’s uploaded images, their dedicated model, to the images we generate for them. We also use Cloud Storage FUSE integration, which has greatly reduced the complexity of our setup. 

When the user joins the platform, we build them their own model data using pictures that they uploaded of themselves and save the model to a Cloud Storage bucket. Then, when the user requests a custom avatar, we rely on Cloud Storage FUSE to perform the read operation. 

Previously, we downloaded the files locally first using GSUTIL. Now, with the Cloud Storage FUSE integration, the processes inside our pods access the model files from Cloud Storage on-demand, whenever needed, like regular files on a local file system. This allowed us to eliminate custom logic of copying data locally, substantially decreasing complexity and speeding up inference time.

Further, integration with GKE through the sidecar container injection makes the entire onboarding process easy to manage. It is very easy to authenticate, and the service account access makes it possible to only allow required pods to access required Cloud Storage objects/folders. We are able to manage the access control easily via Google IAM.

Here’s an overview of the workflow:

1. After a user downloads the Voi app, they upload 15 pictures of themselves, which are stored in a Cloud Storage bucket. 

2. As soon as the upload is complete, the model training process begins using Pytorch on GKE with GPUs.

3. For each individual user, the application generates a custom AI model as a file in a file system hierarchy that is stored in a Cloud Storage bucket as a binary file.

4. When the user asks the app to generate an image in a given style (e.g., “paint me as a 3D rendered character with big eyes”), the inference pod within GKE accesses the user’s model from the Cloud Storage bucket via the Cloud Storage FUSE integration, treating it like a local file. It then generates the artistic visual, and saves it to a different Cloud Storage bucket. 

Typically, users request multiple digital art generation in a given user session, so the user’s model is used a couple of times. In order not to duplicate the user model data across different GKE pods, we implemented internal logic to keep track of which pods have which models in runtime with a specified time-to-live (TTL). After the TTL expires, the user model data file is deleted.

5. Finally, using Google Cloud CDN, the image is then served back to the user.

With Cloud Storage FUSE as a Google Cloud supported product, we’re making it easier for companies to leverage the scale and breadth of Cloud Storage with models that require file system access, eliminating the need to refactor AI applications. Ready to start using Cloud Storage FUSE for your own generative AI data pipelines? Learn more about Cloud Storage FUSE here, and how to get started on GKE.

Google Cloud’s Cloud Storage is home to reams of training data, models and checkpoints that you need to train and serve AI workloads, delivering the scale, performance, simplicity and cost-effectiveness that are the hallmarks of a cloud storage system. But when it comes time for an AI workload to actually access that data, it isn’t always straightforward, since most AI workloads require file system semantics, rather than the object semantics that Cloud Storage provides.

Linux’s Filesystem in Userspace, or FUSE, is an interface used to export a file system to the Linux kernel. An open-source version of Cloud Storage FUSE, has been available for some time, allowing objects in Cloud Storage buckets to be accessed as files mounted as a local file system. Today we are taking an important next step: delivering Cloud Storage FUSE as a first-party Google Cloud offering, with new levels of portability, reliability, performance and integration.

The new Cloud Storage FUSE is particularly important for AI workloads. Because applications can access data directly (rather than downloading it locally), there’s no custom logic to implement, and less idle time for valuable resources like TPUs and GPUs while the data is copied over. Further, a new Cloud Storage FUSE CSI driver for Google Kubernetes Engine (GKE) allows applications to mount Cloud Storage using familiar Kubernetes API, and it’s offered as a turn-key deployment managed by GKE.

Let’s take a closer look at each at how the new first-party Cloud Storage FUSE delivers increased portability, reliability, performance, and integration. 

Portability 

Cloud Storage is a common choice for AI/ML workloads because of its unlimited scale, simplicity, affordability, and performance. But while some AI/ML frameworks have libraries that support native object-storage APIs directly, others require file system semantics; or sometimes, the organization has standardized on file system semantics for a consistent experience across hybrid and multicloud environments. To overcome this, developers have to instrument their training code with logic to first copy the training data from Cloud Storage to a local disk. 

With Cloud Storage FUSE, objects in Cloud Storage buckets can be accessed as files mounted as a local file system, providing file system semantics while being able to continue using Cloud Storage. 

AnyConcept develops a new form of functional, no-code software tests using Deep Reinforcement Learning agents, and uses Cloud Storage FUSE as part of its dataset pipeline for its AI models, first as part of pre-processing data using Jupyter Notebooks, and after for training its AI models.

“Since we need file system semantics and the data is too large to be copied to our TPU based training VMs, Cloud Storage FUSE allows us to access this data directly, giving us unlimited space with the convenience of a file system while maintaining cost-efficiency.” – Manuel Weichselbaum, CTO, AnyConcept GmbH

Reliability 

By making Cloud Storage FUSE a Google-supported product, we aimed to achieve Google standards for reliability, so you can run production workloads with full support. During testing, we uncovered and fixed several stability issues with the original code base. We also integrated Cloud Storage FUSE with the official Go Cloud Storage client library, and validated Cloud Storage FUSE for PyTorch and TensorFlow at high scale and long duration using ViT DINO and ResNet ML models. As part of our production readiness, we also overhauled the documentation to make it easier to use. 

Global credit reporting firm Equifax hosts its Equifax Ignite® platform on Google Cloud, which customers use to apply high-end machine learning capabilities on Equifax data for predictive models and insights. 

“Integration of Jupyter Notebook, installed on Google Kubernetes Engine (GKE), with Google’s Cloud Storage is a core component of Equifax Ignite, and the Cloud Storage FUSE integration with GKE through the CSI drive made it seamless and easy to use. We are pleased that it is now available as part of the Equifax Ignite fully cloud native service offering.” – Vibhu Prakash, Vice President, Analytics Platform, Equifax

Performance

AI/ML workloads typically use accelerators, in the form of GPUs and TPUs, for training and inference workflows. These accelerators are data-hungry, and keeping them idle while they wait for I/O only increases the cost of using them. For applications that need to consume Cloud Storage data via a file system, developers typically implement complex logic to first copy data from Cloud Storage to a local disk, resulting in idle time for the compute resources as they wait for objects to download. With Cloud Storage FUSE, developers can treat a Cloud Storage bucket as a local file system, and stream data directly to the application as if it were local. You can see published benchmarks here.

OpenX is a global adtech company that runs its exchange in Google Cloud, where it processes hundreds of billions of ad requests daily. It previously relied on a home-grown solution to fetch data files from Cloud Storage into init containers at pod start-up, and to periodically refresh the data in running pods. 

“With the Cloud Storage Fuse GKE integration, all of that goes away; all it takes is a simple annotation in the pod spec and a volume definition to make the data available. Using Cloud Storage FUSE with the GKE CSI driver has resulted not only in vastly simplified configuration for our applications, but has also reduced the pod startup time by up to 40%.” – Mark Chodos, Staff Engineer, OpenX

Integration

You can deploy Cloud Storage FUSE in your own environment in a variety of ways:

As a Linux package

Using pre-built image Google ML images, such as Deep Learning Virtual Machines and Deep Learning Containers

As part of the Vertex AI platform

And now, as part of a turn-key integration with Google Kubernetes Engine (GKE) through the Cloud Storage Fuse CSI driver

Pathology researcher Reveal Biosciences pinpoints and categorizes diseases and leverages machine learning to refine its models for superior accuracy, resulting in improved patient prognosis. 

“A pivotal asset in our journey has been Cloud Storage FUSE. Our data is stored in a Cloud Storage bucket, but because our application needs to access these files using file-system semantics, we used to have to download the data locally first. This remarkable tool now enables us to process terabytes of data without needing to manage locally attached storage to VMs or Kubernetes clusters, giving us an efficient alternative with scalable capacity that is not tied to local compute. Google has been instrumental in propelling us towards our performance goals, providing invaluable support.” – Bharat Jangir, MLops Engineer, Reveal Biosciences 

Using the Cloud Storage Fuse CSI driver on GKE

Previous Fuse solutions with Kubernetes required elevated privileges, had noisy neighbor issues , and authentication challenges. The new Cloud Storage FUSE CSI driver does not need privileged access, is fully managed by the CSI lifecycle, and has built-in authentication with Workload Identity, all while allowing Kubernetes pods to access data in Cloud Storage buckets using file-system semantics.

“The integration with GKE through the sidecar container injection makes the entire onboarding process easy to manage. It is very easy to authenticate, and the service account access makes it possible to only allow required pods to access required Cloud Storage objects/folders. Therefore, we are able to manage the access control easily via IAM.” – Uğur Arpaci, Lead DevOps Engineer, Codeway

There are two ways to provision Cloud Storage-backed volumes:  

Using ephemeral volumes, where you simply specify your Cloud Storage bucket and authentication information in the pod spec. We recommend this approach for its simplicity.

Using static provisioning with PersistentVolumes and PersistentVolumeClaim. This approach is recommended if compatibility with the traditional ways of accessing storage on Kubernetes is important for your organization.

The Cloud storage FUSE CSI is supported on both GKE Standard and GKE Autopilot starting with GKE version 1.26 with a plan to backport to earlier versions in subsequent releases. 

Additionally, Terraform templates are also available. If you are using ephemeral volumes, you simply need to specify your Cloud Storage object along with your Cloud Storage bucket and authentication information and the data will be available to your pod. Learn more here. Example pod spec below.

File system semantics, without the fuss

With Cloud Storage FUSE, you can continue to use Cloud Storage as your source of truth for your AI/ML workloads, without sacrificing file-system semantics, wasting valuable resources, or having to implement complex integration logic. To learn more, read about how Codeway leverages Cloud Storage FUSE for generative AI, check out the official documentation, or see the Cloud Storage FUSE GitHub page.

Enterprise applications require a solution that delivers control, protection, and efficiency without sacrificing flexibility. That’s why we’re excited to announce that, starting today, Google Cloud NetApp Volumes is now generally available as a first-party, fully managed service on Google Cloud. 

Imagine being able to provision, automate, and scale a file storage service with just a few button clicks. Need to migrate complex mission-critical enterprise applications? Reduce cloud waste? Reduce storage costs? Check, check, and check; NetApp Volumes can do it — and more.

By adding the award-winning NetApp ONTAP data management software to Google Cloud, we’re extending our five-year partnership with NetApp to offer a fully managed service that delivers cloud-first file storage. With NetApp’s market-leading on-prem file storage and advanced capabilities, you can unlock the full potential of today’s most demanding workloads. 

What is Google Cloud NetApp Volumes?

NetApp Volumes is a fully managed, high-performance file storage service that makes it easier, safer, and more cost-efficient to migrate and run demanding enterprise applications and workloads in Google Cloud. With NetApp Volumes, you can run either Windows or Linux applications as virtual machines quickly and efficiently, without any refactoring. 

In addition to providing NFS and dual protocol support, NetApp Volumes is Google’s first and only managed storage solution to enable SMB support for provisioning file storage for your application environment. 

With no application performance impact and space-optimized Snapshot technology, NetApp Volumes provides backup and recovery copies of your data that don’t consume additional storage resources or affect your application performance. In addition, you can leverage volume cloning, remote replication, and cross-region backup to enhance the business continuity of your different data volumes. 

NetApp Volumes unlocks new opportunities to work with cloud-first architectures along with the capability to scale volumes from 100GiB to 100TiB, so you can drive innovation without worrying about managing the underlying data. By using today’s container approaches, NetApp Volumes lets you modernize a wider range of workloads, while delivering the performance, availability, durability, and security required to migrate and support traditional workloads in Google Cloud. It’s also an ideal solution for customers looking to implement the powerful management capabilities of ONTAP for enterprise data management. 

The benefits of Google Cloud NetApp Volumes 

By partnering with NetApp to enhance our storage offerings, we’re helping you achieve your objectives for application performance, manageability, security, and costs. 

According to IDC analysis, NetApp Volumes helps organizations realize an average annual savings of $4.7 million and a three-year ROI of 457% compared to a typical on-premises deployment by:

Drastically reducing the occurrence of unplanned outages to reduce the impact of downtime

Improving the productivity and performance of IT teams through easier configuration, auditing, testing, provisioning, and scaling

Enabling organizations to foster an environment of higher end-user productivity and revenue growth through better performance, availability, and agility

“As an existing NetApp customer leveraging Cloud Volumes ONTAP and Cloud Volumes Service on Google Cloud, we are thrilled to embrace the new Google Cloud NetApp Volumes. This fully managed file storage service will provide seamless migration of enterprise applications, like Kubernetes, to Google Cloud without refactoring code or redesigning processes. Ultimately, it will simplify operations across all our IT environments.” – Anthony Lloyd, VP Technology Services, OpenText

Transform data management with Google Cloud NetApp Volumes

Start simplifying your cloud journey with continuous availability and data protection at the highest levels. Learn more about Google Cloud NetApp Volumes.

Today, our data and analytics partners are rolling out new updates and integrations with Google Cloud to help customers significantly reduce the time it takes to go from managing data to building new AI tools and workflows. These partner solutions represent a significant step forward in Google Cloud’s mission to offer the industry’s most open data cloud ecosystem—a major asset for customers building generative AI. 

By utilizing Google Cloud services like BigQuery, our powerful foundation models in Vertex AI, and our trusted infrastructure, partners can help customers improve some of their most common (and complex) use cases for data. Data and analytics companies are then providing more tooling to help businesses address industry-specific needs, develop bespoke generative AI models, and apply AI to help users more efficiently explore their data.

Supporting industry-specific use cases for generative AI

Businesses have an opportunity to use generative AI to address common, industry-specific challenges, such as helping a healthcare provider instantly generate patient notes for clinician review, or enabling a loan officer to instantly generate a summary of open applications. Data underpins all of these industry scenarios, and now our partners are launching several new solutions to address them.

Confluent, which provides a widely-adopted data streaming platform, will use Google Cloud’s generative AI to launch new solutions for retail and financial services customers, improving business insights and operational efficiencies. For example, generative AI can help an inventory manager better predict supply shortages based on local demand trends to mitigate “out of stock” items, while a banking security expert can apply it to improve fraud detection based on more accurate risk models and alerts.

DataRobot, a platform for building AI solutions, will use Vertex AI to help clinicians better understand and make decisions with their data. By querying a patient’s name and symptoms, a new solution will run a model that uses BigQuery for analytics and can reference personalized medical history stored securely on AlloyDB to support a clinician in determining next steps in care. The solution will give healthcare organizations full control over their data, with preview access available to customers later this year.

MongoDB, a leading developer data company, is working with Exafluence to develop a platform called Exf ChemXpert for the chemical industry to help researchers and chemists plan for synthesis and predict forward reaction, reaction completion, and retrosynthetic routes. The platform will use AI and data mining techniques with MongoDB Atlas Vector Search and foundation models from Google Cloud to discover new molecules. Exf ChemXpert will also include configurable components for a wide variety of applications in the chemical industry, such as property prediction to guide the design of new molecules, chemical reaction optimization to make developing molecules more environmentally friendly, and novel drug discovery in the pharmaceutical industry. 

Helping customers build custom generative AI models and applications

Partners like Dataiku, Redis, SingleStore, and Starburst are all making it easier for customers to train AI models and build new generative AI applications using data stored within their platforms. Now, each of these partners is working to help customers apply powerful AI models on top of their relevant data.

Dataiku, an AI and data science platform, will integrate with Vertex AI to bring the PaLM 2 foundation model to its Prompt Studios interface to improve how engineers design, test, and operationalize generative AI prompts. With PaLM 2, users will be able to easily deploy a powerful model directly within Dataiku workflows, which enables enterprises to more quickly build and scale their generative AI applications. 

Redis, which delivers an enterprise-grade data platform, will help customers build custom generative AI applications together with Vertex AI. By utilizing contextual data stored in Redis as a vector database, and Google Cloud’s powerful foundation models, customers will be able to create unique generative AI applications like virtual shopping assistants, automated customer support services, and more. 

SingleStore, a cloud-first database system, will utilize Vertex AI to help customers build bespoke generative AI applications based on relevant data stored across its systems. This will allow SingleStore users to more easily identify and contextualize real-time business insights through experiences like a chatbot, which will apply Google Cloud’s natural language processing to provide more contextual, accurate responses. SingleStore will launch these capabilities later this year.

Starburst, the fast-growing data lake analytics platform, will integrate with Vertex AI to enable customers to build and train generative AI models from data stored across multiple cloud providers and on-premises environments. Starburst’s federation engine unites this data in a single environment, where customers will be able to run advanced analytics workloads and improve analysis with Google Cloud’s foundation models. 

Exploring large datasets with generative AI

Generative AI can further democratize data by letting more people explore large datasets with accuracy and confidence. With Vertex AI, many of our partners are utilizing Google Cloud’s LLMs and other foundation models in Model Garden to enhance how users search and analyze data stored across a wide array of systems. 

Datastax, which provides a scalable vector database, will integrate with Vertex AI to help customers build secure, production-ready AI applications. Through a new extension in DataStax’s Astra DB, developers will be able to easily combine data stored in Astra DB with Google Cloud’s foundation models to enable more accurate, consistent, and contextual responses to search queries with natural language understanding. The new capabilities will launch this fall. 

Elastic, a leading platform for search-powered solutions, will integrate Vertex AI with the Elasticsearch Relevance Engine. By combining enterprise data with Google Cloud’s foundation models, Elastic users can expect a secure deployment that generates more relevant, factual answers to users’ complex questions. Elastic will launch these capabilities later this year.

Neo4j, a leading graph database provider, will integrate with Vertex AI to add vector search to its AuraDB database offering. This capability improves the search experience by providing faster performance and more contextual and accurate results—for example, a consumer searching for umbrellas on a retail customer’s website could also see results for rain boots and jackets. AuraDB customers can start using these features next week.

We want to provide customers with the industry’s most open, flexible cloud platform, and that is echoed in our work in generative AI. New integrations with data and analytics companies will be invaluable in helping to extend the power of generative AI to more customers, and we believe they can dramatically improve how businesses use their data moving forward.

Editor’s note: The post is part of a series showcasing our partners, and their solutions, that are Built with BigQuery.

For decades, organizations have relied on Geographic Information Systems (GISs) to combine analytics with geospatial data to inform their data-driven strategies. This includes industries like telecommunications, where GISs can determine the optimal locations to deploy infrastructure, or environmental planning, to identify the best places to plant trees for maximum benefit. Even the advertising industry uses GIS to pinpoint the most effective billboard locations for campaigns.

However, while GISs have increased in sophistication, allowing companies to hone in on locations or demographics at a granular level, they have also increased in complexity. Today, the full potential of GISs are only easily available to experts in the field, and most GIS or map applications are usually underused due to their complexity barrier.

With the help of Large Language Models (LLMs) and generative AI, CARTO is enabling organizations to overcome this accessibility challenge with Conversational GIS – allowing users of all levels to replace complex analytics workflows with dialogue.

The complexity of GISs, map applications, and portals prevents many users from taking full advantage of their potential. The UI for these traditional apps and systems have a multitude of options and layers to perform analysis, which means that while the capabilities are strong, they can only be accessed by experts trained to navigate the complex analytics workflows.

A second challenge to these applications is the need to run on specific geospatial software separated from the rest of the analytics stack. Solutions that aren’t cloud native typically require duplicating or caching the data in a separate system – which leads to more complex architecture with limited data governance and higher costs. A way to overcome this is by running geospatial analytics in the same cloud-native platform as the rest of the organization.

Simplifying GIS with LLMs and Generative AI

Generative AI is transforming the analytics landscape, enabling users of all technical abilities to navigate complex data with ease. Chat experiences, for example, provide a conversational approach to analytics and allow you to define what they need using only a query or demand – without needing to navigate multiple filters, layers or analytics workflows.

Seeing the benefit of LLMs and generative AI, CARTO has been able to solve the complexity problem of apps and portals and provide a much simpler GIS interface by removing the need for manually selecting filters, options, and other components that stand in the way of GIS functionality. With Conversational GIS, you can simply ask the system about the best locations for your needs. For example: “what locations in my city get the most foot traffic in July?” or “where should I place an advertisement targeting new parents?” Conversational GIS will help you select the best filters for your specific questions – acting as an expert geographer by your side. Not only will you get a simpler, more streamlined experience but you will extract more value from your GIS and get more thorough answers to your questions. 

Optimizing campaigns with Conversational GIS

One organization taking advantage of Conversational GIS is Clear Channel Outdoor, a media company at the forefront of driving innovation in the out-of-home (OOH) advertising industry. Clear Channel operates in 21 countries and manages more than 470,000 billboards around the world.

When it comes to building an effective advertising campaign with OOH, the key is aligning the audience with location analysis. It’s important to identify the right locations where your target audience will see them, making sure your advertisements are optimized to reach people at the right times, in the right moments, and at the point of decision during their customer journey.

Clear Channel Outdoor’s RADARView is an audience insights platform that has a map-based interface, built on Carto, that combines aggregated demographic data, behavioral insights, and location and proximity targeting to easily discover and select the out-of-home inventory that most efficiently delivers an Advertiser and Brands desired audience.

Using RADARView, Clear Channel Outdoor builds OOH media plans using many options that unlock the right combination of behavioral audience data, demographic information, location proximity to a wide range of points of interest to generate the best OOH audience-based campaign. However, with more than 70,000 options (regions, audience segments, brands, etc.) to choose from, there are millions of possible combinations, and choosing the right filters for a campaign is highly complicated. This means the full functionality is only accessible to a team of experts trained on how to optimize the platform. When Generative AI capability is added to RADARView, the product will be easier to use, helping Clear Channel Outdoor’s employees and customers work with data more efficiently to plan OOH campaigns.

By applying CARTO’s Conversational GIS to Clear Channel Outdoor’s RADAR tool, there’s a more natural way of uncovering the right context and audience for an OOH campaign. Now, users of RADARView can have a conversation with the product to build even more successful OOH campaigns. This self-service approach, based on natural language processing and conversation, is expected to increase utilization of the tool among all teams at Clear Channel Outdoor, because the enhancement of semantic navigation of filters simultaneously simplifies the user experience while maximizing the effectiveness of the media plans that are generated, resulting in a very improved and high performing OOH audience-based campaigns – a real win-win.

Built with BigQuery: unlocking the power of geospatial data with generative AI

Clear Channel is just one example of Conversational GIS being underpinned by BigQuery and Vertex AI – as well as a range of other Google Cloud services. BigQuery provides the foundation for all the information the generative AI will need, while BigQueryML connects to Vertex AI and PaLM2. Here’s how it works:

When you type in a request or prompt, this query goes to an endpoint in Cloud Functions that calls the PaLM2 model by using the BigQuery ML function ML.GENERATE_TEXT to extract the key entities in the request or prompt

These entities are converted into embeddings, then used to retrieve custom data like audiences, Points of Interest (POIs), brands, and more, all stored in an embeddings database in Vertex AI Matching Engine.

Once the application has the response from the PaLM2 model and the custom data, the application will transform all of these filters into a set of queries that run directly to BigQuery through CARTO.

The final output is a set of billboards in a map that satisfies the filters. The application itself is deployed by using Firebase Hosting and Firestore and it uses Google Maps as basemaps.

Building Conversational GIS not only requires a solid data foundation, scalable systems, and modern data stack architecture, for security, it’s also important that the data never leaves the data warehouse. The combination of CARTO, BigQuery and Vertex AI uniquely provides the capability of LLMs and native geospatial data, all within the same system. 

With the addition of geospatial support to BigQuery, and the CARTO platform on top, it is now possible to provide full GIS capabilities in the same analytics platform as the rest of the organization. This makes GIS scalable, cost effective and secure – with no need to move data to any specialized systems, simply run it on the Data Cloud.

CARTO and BigQuery to modernize your GIS application 

CARTO enables full GIS capabilities on top of BigQuery native GEO support. This cloud-native architecture avoids having to duplicate your geospatial data to a different system and connects it to the rest of the analytics world. This modern stack results in a more scalable, powerful and cost-effective platform. Learn more about CARTO’s advanced spatial capabilities for BigQuery users here. And if you are interested in how AI can modernize and improve the value of your GIS system, contact one of our experts to guide you.

The Built with BigQuery advantage for ISVs and Data Providers

Google is helping companies like CARTO build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs through the Built with BigQuery initiative. Participating companies can: 

Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices. 

Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in.

Click here to learn more about Built with BigQuery.

Editor’s note: The post is part of a series showcasing our partners, and their solutions, that are Built with BigQuery.

In the time it takes to read this sentence, around five football fields of forest will be lost to deforestation globally. Despite growing concerns from companies, communities, and regulators, global deforestation is still responsible for millions of hectares of forest being destroyed every year. In 2022, the European Union Deforestation Regulation (EUDR) bill was passed, banning imports of raw materials linked to deforestation and establishing more transparency over supply chains by verifying sourcing footprints. Now, the burden of proof has shifted to consumer packaged goods companies to verify that the raw materials they source from suppliers are deforestation free and not just from certified sources. This includes commodities like palm oil, cocoa, soy, coffee, wood, rubber, beef, and leather. 

The key requirements for operators is to establish a diligence process that addresses:

Supply chain mapping: collecting data from producers and traders that enables first mile verification of the sourcing footprints.

Risk assessment: using satellite imagery and supply chain mapping data to identify cases of deforestation.

Risk mitigation: mitigating instances of more than negligible risk of non compliance.

TraceMark is a sustainable sourcing monitoring platform powered by Google Cloud, designed to give enterprises greater transparency into global supply chains to meet sustainability commitments to the sourcing of raw materials. The powerful earth observation analytics provided by TraceMark, including deforestation, reforestation, estimates of carbon, and land use change, is delivered through an advanced data sharing ecosystem that facilitates automated, transparent and consistent exchange of supply chain data to optimize supplier mapping.

Companies with global supply chains are struggling to gain real-time visibility into operations at a global scale and access accurate information on the sourcing of raw materials. And each of the key requirements set out by the EUDR comes with its own challenges:

Supply chain mapping documentation is often limited and inconsistent across suppliers.

Risk assessment is difficult as independent validation can’t scale across global networks.

Risk mitigation is delayed without active supplier engagement or real-time reporting.

Many organizations that are currently using certification as a means of generating deforestation free claims must now extend into footprint level verification activities. To be able to accurately monitor, verify and report on deforestation across global supply chains, manufacturers will need to build their supply chain traceability through direct engagement with suppliers.

However, the capability to deliver two way engagement between organizations and their suppliers is a major challenge for sustainable sourcing teams globally. The main barriers include:

A lack of control over how supplier data is shared and used

Inconsistent data and approaches used to generate key metrics

Batch exports resulting in out of date data

The inability to facilitate two way conversations between manufacturers and suppliers

A lack of automation and scale with sharing data across many suppliers and manufacturers

These barriers make it difficult for organizations to build out their supply chain maps effectively, and in many instances, requires these organizations to be creative with their traceability approaches.

TraceMark is designed for the future of sustainable supply chains where data can be seamlessly shared between organizations and suppliers in a consistent, automated and controlled way that will ultimately reduce barriers, improve reporting accuracy and timeliness, optimize traceability, and simplify security by keeping the data within the TraceMark platform. 

Through TraceMark, Google Cloud and NGIS are leveraging Analytics Hub to enable global organizations to securely exchange first mile sourcing data with their suppliers through an automated and managed data sharing ecosystem. Analytics Hub addresses the technical barriers with facilitating a data sharing ecosystem in the sourcing of raw materials. The data exchange platform enables organizations to share data and insights at any scale across boundaries with a robust security and privacy framework — while enhancing analysis with commercial, public, and Google datasets through a marketplace. 

To enable suppliers and operators to share supply chain data, TraceMark provides a custom user interface, on top of Analytics Hub, designed to create a simplified and seamless end-to-end experience for sharing data easily and securely.

This ecosystem approach will not only enable global companies to address deforestation regulation, but it will also provide the foundation to expand into more accurate emissions reporting through the seamless integration of stakeholder-specific data. Where organizations are currently estimating indirect greenhouse gas emissions, they will be able to increase the accuracy and consistency of these assessments by leveraging footprint data from the source. 

For example, there are a number of other regulation advancements for sustainable supply chains including:

The Fashion Sustainability and Social Accountability Act, introduced in the state of New York, will require greater transparency and reporting by fashion businesses

The French Climate and Resilience Law includes mandatory environmental labeling for products

The German Supply Chain Due Diligence Act imposes extensive new obligations on companies with regard to human rights along the supply chain

With TraceMark providing planetary-scale first-mile supply chain monitoring, and the data sharing ecosystem facilitated through Analytics Hub delivering the ability to build and evolve supply chain mapping, enterprises can combine data sources and have a bidirectional approach to sustainability.

Together, TraceMark and Google Cloud are helping brands gain a deeper understanding of sustainable sourcing practices across supplier networks, helping them get real-time, reliable information into operations at a local supplier level, globally. Google’s recent 2023 Environmental Report highlighted how we’re helping Unilever build a more holistic view of the forests, water cycles, and biodiversity that intersect its supply chain.

Better together: TraceMark and BigQuery

TraceMark could really only be built with Google technology — it’s a 100% Google Cloud solution that delivers global sustainability monitoring through Google’s next-generation geospatial tools in Earth Engine, BigQuery and Google Maps. It leverages over 80 petabytes of satellite data to detect deforestation, changes in land-use, estimates of carbon and other data points used by organizations to report on their corporate commitments to sustainability. The use of geospatial capabilities is key to being able to layer multiple sources of data against organization-specific supply chains.

With the use of truly unique geospatial technology in Google Earth Engine and Google Maps, NGIS has been able to unlock petabytes of earth observation data about our changing planet to provide first mile deforestation analytics in through TraceMark.

To deliver the traceability and transparency required for the monitoring of raw materials, TraceMark has implemented a multi commodity data model through Google Data and AI Cloud for Supply Chain. This data sharing ecosystem is built with Google Analytics hub, delivering the capability to efficiently and securely share data assets across organizations and build complex supply chains with traceability from factory to footprint.

And the final piece of the puzzle is the Google Cloud Ready – Sustainability program. This initiative connects organizations with significant sustainability commitments to the data-rich solutions that help reduce their carbon footprints. This program has provided a spotlight for TraceMark and other Google Cloud partner solutions to help global businesses and governments accelerate their sustainability programs.

To continue exploring Tracemark, visit the website here. 

The Built with BigQuery advantage for ISVs and data providers

Google is helping companies like NGIS build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs through the Built with BigQuery initiative. Participating companies can: 

Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices. 

Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in.

Click here to learn more about Built with BigQuery.

Enterprises are selecting Google Cloud for their biggest digital transformation projects, utilizing our trusted infrastructure, powerful data and AI capabilities, cybersecurity services, industry solutions, and more to address their biggest opportunities. 

And because we’re partner-led in services delivery, our ecosystem—particularly our global systems integrator and consulting partners (GSIs) like Accenture, Capgemini, Cognizant, Deloitte, and TCS, and more—play a tremendous role helping enterprise customers plan, execute, and manage their digital transformations on Google Cloud. These large innovation projects are only accelerating, which means it’s critical that partners continue to develop the services capacity and deep expertise that customers need.

Today, I’m excited to share a new study from IDC that demonstrates the significant investments that GSIs have made in their Google Cloud practices over the last several years. In fact, IDC found that our GSI partners have grown their skilled Google Cloud resources by as much as 10X in the past three years. Furthermore, GSI’s Google Cloud practices are strongly outpacing industry growth—a testament to the customer value that these partners create with Google Cloud products.

According to the study, this growth will continue: IDC expects GSI partners will more than triple their professional services capacity for Google Cloud customers by 2025. Additionally, the new IDC research demonstrates that our differentiated approach to partnering, along with GSIs growing investments in services delivery, are creating value for partners and customers alike. Here were some of the topline findings:

We collaborate, not compete with our partners: GSIs validated we are staying true to our mission of being a partner-first organization. IDC said they “reaffirmed that this commitment is a differentiator for their businesses,” and they “are able to work in clearly-defined swim lanes alongside Google Cloud, and this is not often the case among hyperscalers.”

GSIs are seeing larger and longer-term engagements with customers: Customer deals have increased significantly in size, complexity, and duration, with a major increase in large-scale deals, usually ranging from two to five years.

Our open, multi-cloud approach benefits customers and partners: According to IDC, “Elsewhere in the industry, customers and partners can face licensing barriers that prevent them from adopting more modern, cost-effective solutions alongside their legacy infrastructure or software. Google Cloud’s position as a ‘cloud-first’ company minimizes this friction.”

GSIs are on board to help customers see value from bleeding-edge technology like generative AI: GSIs have collectively committed to train more than 150,000 experts on Google Cloud’s generative AI, which will ensure customers identify the best applications of generative AI and successfully deploy it within their businesses. 

Google Cloud’s strengths in data and infrastructure modernization, analytics, and AI are driving many customer engagements for GSIs: Partners said that these product areas are among Google Cloud’s primary differentiators, and they are creating meaningful opportunities for customers to solve business challenges. 

The IDC White Paper echoes what we hear from customers and partners every day. For example, Accenture is helping LendLease protect its infrastructure with a security solution that utilizes Chronicle Security Operations and Google Cloud AI; Capgemini supported L’Oréal with a solution based on Google Cloud, Apigee API Management, and BigQuery to connect the tools and systems to support scannable QR codes; and, Deloitte is working with Kroger to improve grocery associate productivity using Google Cloud’s data analytics and AI.

IDC’s White Paper is titled: “Google Cloud: Empowering Global System Integrators for Joint Success*” and was sponsored by Google. You can read it here, and read more about Google Cloud’s partner program here. 

*doc #US50743723, August 2023