GCP – Announcing Dataform in GA: Develop, version control, and deploy SQL pipelines in BigQuery

Data teams building SQL pipelines grapple with the challenge of manually piecing together custom processes and infrastructure, creating a lag time in development, wasting time troubleshooting issues, and often preventing data analysts from contributing to the process. To help, today we are announcing the general availability of Dataform, which lets data teams develop, version control, and deploy SQL pipelines in BigQuery. Dataform helps data engineers and data analysts of all skill levels build production-grade SQL pipelines in BigQuery while following software engineering best practices such as version control with Git, CI/CD, and code lifecycle management. 

Dataform offers a single unified UI and API with which to build, version control, and operationalize scalable SQL pipelines. In this single environment, data practitioners can develop new tables faster, ensure data quality, and operationalize their pipelines with minimal effort, making data more accessible across their organization.

An end-to-end SQL pipeline experience in BigQuery

With Dataform, data and analytics teams can:

Develop complex pipelines with SQL in code with Dataform core, an open-source framework that brings automated dependency management, data-quality testing, code reuse, and table-documentation features to SQL development.

Develop pipelines on the web from the BigQuery console, where users can work from individual, isolated workspaces, visualize their pipelines dependencies, get real-time errors, and version control their code with Git. 

Deploy SQL pipelines in different execution environments, on a schedule or via API triggers, without having to manage any infrastructure.

A unified experience for all data practitioners

Dataform helps organizations standardize SQL pipeline development across their organization around a single piece of tooling and a single development process. 

Data teams can collaborate following software engineering best practices like Git, CI/CD, and code lifecycle management. 

Data engineers can manage code lifecycle and scheduling across development, staging, and production execution environments without having to manage any infrastructure. 

And lastly, data analysts can contribute to existing pipelines or manage their own by developing, testing, and version controlling SQL pipelines from a web interface. 

What customers are saying

“As a company with 1000+ collaborators, we used to struggle with a lack of governance and standards to manage our BigQuery data,” says Lucas Rolim, Director of Data & Analytics at Hurb. “Dataform provides our data team a common interface to adopt software development best practices such as versioning, code reviews and commit history.”

“Before we started using Dataform, we used an in-house system to transform our data which was struggling to scale to meet our needs,” says Neil Schwalb, Data Engineering Manager at Intuit Mailchimp. “After adopting Dataform and more recently Dataform in Google Cloud we’ve been able to speed up and scale our data transformation layer to 300+ tables across large volumes of data. The Google Cloud-Dataform integration has also sped up our development workflow by enabling faster testing, clearer logging, and broader accessibility.”

“Over the last few years the demand for Data Science, Machine Learning and AI has grown massively at OVO on our Path to Zero [carbon emissions]. We grew fast, and each team of Data Scientists were tasked with building data and machine learning pipelines. It meant we could deploy new capabilities quickly, and drive real benefits for our customers, but it wasn’t scalable,” says Dr. Katie Russell, Data Director at OVO. “Adopting Dataform has allowed us to create consistency without sacrificing flexibility or pace of development. With Dataform we have sped up deployment processes, reduced quality issues and improved documentation and discoverability.”

How can I learn more?

You can get started with Dataform by visiting the website or reading the documentation.