GCP – Introducing BigQuery unified governance: universal, intelligent, and open

Data is the critical foundation for AI, yet a vast amount of data’s potential remains untapped. Why? Data quality remains a top barrier. To use enterprise data to drive analytics-driven decisions and build differentiated AI, businesses need to be able to find, understand, and trust their data assets. This requires effective data governance encompassing discovery, cataloging, metadata management, quality assurance, sharing, and access control.

The stakes are high. According to Gartner, “through 2026, those organizations that don’t enable and support their AI use cases through an AI-ready data practice will see over 60% of AI projects fail to deliver on business SLAs and be abandoned.”

At Google Cloud Next 25, we’re announcing BigQuery unified governance, powerful data governance capabilities that help enterprises keep pace with governance complexities. Data silos, fragmented metadata, and ambiguous ownership create significant risks and impede innovation. BigQuery unified governance provides services and tools organizations need to simplify data management and unlock actionable insights.

BigQuery’s built-in, intelligent governance simplifies data and AI management, helping organizations discover, understand, and leverage their assets, transforming governance from a burden into a powerful tool for data activation. Central to BigQuery governance is BigQuery universal catalog, a unified, AI-powered data catalog that natively integrates Dataplex, BigQuery sharing, security and metastore capabilities, bringing together business, technical, and runtime metadata.

BigQuery’s unified governance capabilities are:

1. Unified: BigQuery brings governance directly into the heart of your data-to-AI lifecycle, enabling discovery, understanding, governance, and utilization of your data assets and AI models. This gives data administrators, stewards, and custodians robust tools for metadata management and policy enforcement, providing end-to-end data-to-AI lineage, data profiling, insights, and secure sharing. And with the new universal semantic search, finding the right data is as simple as asking a question in natural language.

2. Intelligent: New governance capabilities powered by gen AI stand to revolutionize data management. By harnessing the power of large language models (LLMs), BigQuery universal catalog can help you uncover hidden relationships between BigQuery data assets, enable automated metadata curation and intelligent query recommendations at scale, automate governance, and democratize data-driven insights across the organization.

3. Open: BigQuery universal catalog insulates you from change with support for open storage standards such as Apache Iceberg, and a unified runtime metastore across SQL, open-source engines, and AI/ML. The BigQuery metastore, which is included in the BigQuery universal catalog, is Iceberg-compliant, enabling a multi-engine, multi-vendor architecture for governance and use of fully managed Iceberg data.

ANZ Bank, a multinational banking and financial services provider, uses the BigQuery universal catalog for comprehensive data governance, discovery, and observability.

“With BigQuery universal catalog, ANZ has significantly improved the reliability and trustworthiness of our data. The centralized data quality monitoring and automated validation features are increasing confidence and efficiency in critical business outputs and decisions based on accurate and consistent information. BigQuery governance has become a cornerstone of our data governance strategy, ensuring our data is not just available, but dependable.” Artur Kaluza, Head of Data Strategy and Transformation, Risk, ANZ

Noteworthy features

The new unified governance experience in BigQuery provides a centralized interface within the BigQuery UI for managing, securing, and sharing data and AI assets. In addition, we are introducing a wide range of key new features and capabilities across governance, sharing, and security.

Governance

1. Full-catalog search with semantic understanding (preview): Users can now discover data and AI resources across projects and data silos within BigQuery using full-catalog semantic search. This feature introduces natural-language search capabilities, making it easier for both technical and non-technical users to search the catalog.

2. Automated metadata curation (preview): BigQuery universal catalog can now automatically generate metadata for BigQuery tables, including table and column descriptions, improving data discovery and support gen AI applications.

3. AI-powered knowledge engine (preview): Users can efficiently discover hidden relationships within a dataset with automated entity-relationship visualization. By leveraging inferred relationships, BigQuery universal catalog generates suggestions for cross-table queries and natural language questions, getting new data teams up to speed fast on unfamiliar data assets.

4. Data products (preview): BigQuery data products allow data owners to create, share, and govern collections of data assets by use case, packaging and sharing them within and across organizations in a way that’s consistent, governed, and that follows security best practices.

5. Business glossary (GA): The BigQuery business glossary provides organizations with a shared understanding of their data. Customers can define and administer company terms, identify data stewards for these terms, and attach them to data asset fields, improving context, collaboration, and search.

6. Automatic at-scale cataloging of BigLake and object tables (GA): BigQuery universal catalog harvests up-to-date metadata for structured and unstructured data from Cloud Storage, and uses it to automatically create query-ready BigLake tables at scale.

7. Automated anomaly detection (preview): BigQuery universal catalog automates data anomaly detection to help you identify data errors, inconsistencies, and outliers in your data, reducing the time you spend identifying and resolving data issues.

Sharing

8. BigQuery sharing integration with Google Cloud Marketplace (preview): Data owners can monetize datasets in BigQuery sharing (formerly Analytics Hub) through Google Cloud Marketplace.

9. Stream sharing in BigQuery (GA): Curate and share valuable real-time streams with Pub/Sub topics in BigQuery sharing.

10. Stored procedure sharing in BigQuery (preview): Share SQL stored procedures and enable execution in the subscriber’s project without revealing the actual code. 

11. Query template sharing in BigQuery (preview): Customize, reuse, and restrict SQL queries in a clean room through publisher-defined query templates.

Security

12. Data policies on columns (preview): Create raw access and data-masking policies associated directly to a column and that can be reused across columns and tables.

13. Subquery support with row-level security (GA): BigQuery universal catalog now supports SQL subqueries in security access policy definitions, enabling row filtering without changing existing data models.

These built-in governance advancements within the BigQuery platform help organizations unlock the full potential of their data and AI initiatives. 

In addition to the innovation in BigQuery, we continue to partner with third-party catalog providers to complement their governance capabilities. For example, Collibra’s enterprise-wide governance for data and AI extends BigQuery universal catalog capabilities to provide end-to-end visibility, quality and stewardship across hybrid and multicloud environments. This partnership helps ensure more teams can discover and trust the data they need to do AI, no matter where it lives, accelerating and strengthening every use case.

By embedding governance into BigQuery and automating metadata management, BigQuery universal catalog is helping businesses move beyond the challenges of data silos and operational inefficiency, ultimately driving innovation and accelerating business impact. Ready to learn more? You can join several sessions covering the latest in BigQuery governance, sharing, and security featuring customer speakers:

1. What’s new in data and AI governance with Levi’s and Verizon | Apr 9, 4pm

2. Redefine data and AI governance in the unified BigQuery platform with Walmart and Box | Apr 11, 12:30pm

3. From chaos to confidence: Master governance in the age of AI with EDMC, ANZ Bank, Intesa Sanpaolo Group, and Google | Apr 9, 11am

4. Unlock the power of secure data sharing with BigQuery with Liveramp and Levi’s | Apr 9, 4pm

5. Data fabric on BigQuery: an architect’s perspective of why and how to do it with Virgin Media O2 | Apr 11, 8:30am