GCP – This migration from Snowflake to BigQuery accelerated model building and cut costs in half

In 2024, retail sales for consumer packaged goods were worth $7.5 trillion globally. Their sheer variety — from cosmetics to clothing, frozen vegetables to vitamins  — is hard to fathom. And distribution channels have multiplied: Think big box stores in the brick-and-mortar world and mega ecommerce sites online. Most importantly, jury-rigged digital tools can no longer keep pace with the ever-growing web of regulations designed to protect consumers and the environment. 

SmarterX uses AI to untangle that web. Our Smarter1 AI model aggregates and triangulates publicly available datapoints — hundreds of millions UPCs and SKUs, as well as product composition and safety information — from across the internet. By matching specific products to applicable regulatory information and continuously updating our models for a particular industry or client, SmarterX helps retailers make fully compliant decisions about selling, shipping, and disposing of regulated consumer packaged goods. 

And just like our clients, we needed to accelerate and expand our capabilities to keep pace with that data deluge and build better AI models faster. Migrating to Google Cloud and BigQuery gave us the power, speed, and agility to do so. 

Embracing BigQuery: a flexible, easy-to-use, AI-enabled data platform

Because we deal with data from so many sources, we needed a cloud-based enterprise data platform to handle multiple formats and schemas natively. That’s exactly what BigQuery gives us. Since data is the foundation of our company and products, we began by migrating all our data — including the data housed in Snowflake — to BigQuery. 

With other data platforms, the data has to be massaged before you can work with it: a time-consuming, often manual process. BigQuery is built to quickly ingest and aggregate data in many different formats, and its query engine allows us to work with data right away in whatever format it lands. Coupled with Looker, we can create easy-to-understand visualizations of the complex data in BigQuery without ever leaving Google Cloud. 

In addition, because Gemini Code Assist is integrated with BigQuery,  even our less-technical team members can do computational and analytical work.

An integrated tech stack unleashes productivity and creativity

After 10 years in business, SmarterX was also suffering from system sprawl. 

Just as migrating data between platforms is inefficient, developers become less efficient when they have to bounce around different tools. Even with the increase in AI agents to help with coding and development, the tools struggle, too: When hopping among multiple systems, they pick up noise along the way. And governing identity and access management (IAM) individually for all those systems was time-consuming and left us vulnerable to potential security risks caused by misapplied access privileges.

Google Cloud provides a fully integrated tech stack that consolidates our databases, data ingestion and processing pipelines, models, compute power — even our email, documents, and office apps — in a single, unified ecosystem. And its LLMs are integrated throughout that ecosystem, from the Chrome browser to the SQL variants themselves. This obviates building custom pipelines for most new data sets and allows us to work more efficiently and coherently: 

• We’re now releasing new products 10-times faster than we were prior to migrating to Google Cloud. 

• We onboard new customers in less than a week instead of six months. 

• Our data pipelines handle 100 times the data they did previously, so we can maintain hundreds of customer deployments with a relatively small staff.

Consolidating on Google Cloud also lowered our overhead by 50% because we deprecated several of other SaaS platforms and teams can easily engage with Google’s tools without specialized expertise. Our entire team now lives in Google Cloud: Not an hour goes by that we aren’t using some form of the platform’s technology. 

Eliminating system sprawl also means we no longer need to maintain security protocols for separate platforms. Permissioning and identity and access management are handled centrally, and Google Cloud makes it easy to stay current on compliance requirements like SOC-2. 

A vision for AI in tune with our own: Gemini

The value SmarterX provides our customers relies heavily on our platform’s AI-driven capabilities. Finding the right AI model development platform and AI models was therefore one of the driving forces behind our choice of a new data platform. And when it comes to creating AI models, philosophy matters.  

Google’s philosophy dovetails with our own because they’ve always been at the forefront of understanding how people want to access information. Since the company’s expertise makes web data searchable on an enterprise scale, its Gemini models are tuned beautifully to do what SmarterX needs them to. Before switching to Vertex AI and Gemini, it took us months to release a new model; we can now do the same work in a matter of weeks.

When SmarterX hires new team members, we look for creative thinkers, not speakers of a specific coding language. And we want to give our developers the brainspace to focus on complex problem-solving rather than puzzling over syntax for SQL coding. Gemini Code Assist in BigQuery is easy to learn and can accurately handle the syntax for them. That leaves our developers more time for finding innovative solutions. 

A smooth migration by a team that knows its stuff

We couldn’t have completed our migration without the support of the Google Technical Onboarding Center. They really know their way around their technology and had spelunking tools at the ready for tricky scenarios we encountered along the way. 

In less than a month, we migrated terabytes of data from Snowflake to BigQuery: more than 80 databases and thousands of tables from 21 different sources. We used a two-pronged approach that leveraged external tables for rapid access to data and native tables for optimized query performance. 

Prior to the migration, Google provided foundational training for managing and operating the Google Cloud Platform. They also took the time to understand SmarterX technology. So instead of being constrained by a cookie-cutter migration plan, the Google team helped us to design and schedule a migration — with minimal disruptions or downtime — in the way that made the most sense for SmarterX and our customers. Google’s expertise in best-practices for security and identity and access management further enhanced the security of our new cloud environment.  

Even though we’re not a huge customer pumping petabytes of data through Google Cloud daily, the team treated us as if we were on par with larger organizations. When you’re literally moving the foundation of your entire business, it feels good to know that Google has your back.

Snowflake felt like a traditional enterprise data warehouse grafted onto the cloud, completely uninfluenced by the AI revolution, with a database that forced us to work in a specific, predetermined way. With BigQuery, we have a real information production system: a computing cloud with a built-in SQL-friendly data platform, a wide-ranging toolset, embedded AI and model development, and a single user interface for developing products our own way.

Unlimited imagination, not roadmaps

Many people are surprised when I tell them that SmarterX doesn’t have roadmaps — we make bets. We’re wagering that companies want AI to solve whatever real-world use cases arise. Rather than telling you what to do, AI has the ability to understand things that were previously impossible to understand, and to help people express ideas that were previously impossible to express. 

SmarterX works with over 2,000 brands. Ultimately, what they’re purchasing is the speed at which we can help them solve their business challenges with artificial intelligence. In much the same way, Google Cloud is solving our own technology challenges, sometimes before we even know we have them, so we can deliver top-notch products to our customers. 

Instead of doing battle with a growing sprawl of outdated technology, BigQuery and the rest of the Google Cloud integrated toolset is allowing us to relentlessly reinvent ourselves. Not a week goes by when I don’t hear someone say, “Oh, wow, we can do that with Google Cloud too?” 

^{Company description
SmarterX helps retailers, manufacturers, and logistics companies minimize regulatory risk, maximize sales, and protect consumers and the environment by giving them AI-driven tools to safely and compliantly sell, ship, store, and dispose of their products. Its clients include global brands that are household names all across the world.}