GCP – Expanding BigQuery Data Transfer Service with new connectors, features, and more

With BigQuery, our goal is to allow you to extract valuable insights from your data, regardless of how much there is, or where it’s from. A key part of how we do this is our BigQuery Data Transfer Service, which automates and streamlines data loading into BigQuery from a wide variety of sources.

As a fully managed service, BigQuery Data Transfer Service offers a variety of benefits:

• Simplicity: Eliminate the need for infrastructure management or complex coding. Whether you use the UI, API, or CLI, getting started with data loading is easy.

• Scalability: Used by tens of thousands of customers each month, Data Transfer Service easily handles massive data volumes and high numbers of concurrent users, accommodating demanding data transfer jobs.

• Security: Your data’s safety is paramount. Data Transfer Service employs robust security measures like encryption, authentication, and authorization. And as you’ll see below, we’ve significantly expanded its ability to support regulated workloads without compromising ease of use.

• Cost-effectiveness: Many first-party connectors, like those for Google Ads and YouTube, are provided at no cost. And for a growing list of third-party connectors, we offer consumption-based pricing that’s highly price-competitive, so you can unify your data cost-effectively.

Based on your feedback, we expanded the BigQuery Data Transfer Service connector ecosystem, enhancing security and compliance, and improving the overall user experience. Let’s dive into the latest updates.

Key feature updates

Expanded data connectivity

We are thrilled to announce that several highly-requested connectors are now generally available:

• Oracle: Integrate your key operational databases with BigQuery for enhanced analysis and reporting.

• Salesforce and ServiceNow: Build unified customer profiles and bring in your IT service management data to gain operational insights.

• Salesforce Marketing Cloud (SFMC) and Facebook Ads: Ingest your marketing and analytics data into BigQuery for comprehensive analysis and campaign optimization.

• Google Analytics 4 (GA4): A major milestone for your marketing analytics, now you can build production marketing analysis pipelines with GA4 data.

These new additions join the quickly growing list of existing connectors, including Amazon S3, Amazon Redshift, Azure Blob Storage, Campaign Manager, Cloud Storage, Comparison Shopping Service (CSS) Center, Display & Video 360, Google Ad Manager, Google Ads, Google Merchant Center, Google Play, MySQL, PostgreSQL, Search Ads 360, Teradata, YouTube Channel, and YouTube Content Owner.

New connectors in preview

We are also excited to launch new connectors in preview, further expanding our ecosystem:

• Stripe and PayPal: Ingest financial and transaction data into BigQuery for revenue analysis, refund tracking, and customer behavior insights.

• Snowflake (migration connector): Migrate your data from Snowflake with features like key pair authentication, auto schema detection, and support for migrating data residing on all three major clouds (Google Cloud, AWS, and Azure).

• Hive managed tables (migration connector): This connector supports Metadata and Tables migration for Hive and Iceberg from on-prem and self-hosted cloud Hadoop environments to Google Cloud. This lets you perform one-time migrations and synchronize incremental updates of Hive and Iceberg tables, with Iceberg tables being registered with BigLake metastore, and Iceberg and Hive tables registered with Dataproc Metastore.

Enhancements to existing connectors and platform capabilities

• Google Cloud Storage: We are excited to announce the GA of event-driven transfers. Now, your data transfers can trigger automatically the moment a new file arrives in your Cloud Storage bucket, for near-real-time data pipelines.

• Salesforce: CRM users get an efficiency boost with incremental ingestion now available in preview. Data Transfer Service now intelligently loads only new or modified records, saving time and compute resources. 

• SA360: The recently updated Search Ads 360 connector now includes full support for Performance Max (PMax) campaigns, so you can analyze data from Google’s latest campaign types.

• Google Ad Manager: We improved data freshness for the Google Ad Manager connector by rolling out incremental updates for DT files. Google Ad Manager adds the Google Ad Manager DT files into the Cloud Storage bucket. A transfer run then incrementally loads the new Google Ad Manager DT files from the Cloud Storage bucket into the BigQuery table without reloading files that have already been transferred.

• Google Ads: We launched custom reports for Google Ads GA, so you can use Google Ads Query Language (GAQL) queries in your transfer configuration to ingest custom Google Ads reports and fields beyond those available in the standard reports and fields.

• Oracle: We significantly enhanced the Oracle connector to support the ingestion of tables containing millions of records, ensuring that even your largest and most critical datasets can be transferred to BigQuery.

Enhanced security and compliance

To continue to meet your stringent security and compliance needs, we’re also investing in our infrastructure.

• Access transparency: Along with BigQuery, we’ve extended Data Transfer Service administrative access controls to customer-identifiable metadata. Administrative access controls (access transparency, access approval, and personnel controls) is a feature of Cloud services that gives customers real-time notifications of when, why, and how Google personnel access their user content. This new capability applies access transparency controls to reads of customer-defined attributes and any customer service configuration that may be used to identify the customer or customer workloads.

• EU Data Boundary: We are excited to announce GA of Data Transfer Service for EU Data Boundary and Sovereign Controls compliance programs in the EU, including EU regions support with Data Boundary with Access Justifications and Sovereign Controls by Partners. This enables customers to expand their workloads on Google Cloud in regulated markets.

• FedRAMP High: We successfully implemented the security controls required to launch Data Transfer Service into the FedRAMP High compliance regime. This will allow U.S. government, civilian agencies, and contractors to expand their adoption of FedRAMP High regulated workloads on Google Cloud.

• CJIS Compliance: We launched BigQuery Data Transfer Service for Criminal Justice Information Services (CJIS) compliance. Data Transfer Service now meets the security standards of the CJIS Security Policy, enabling U.S. state, local, and tribal law enforcement and criminal justice organizations to handle sensitive information using our service.

• Custom organization policies: We announced the GA of custom organization policies so you can allow or deny specific operations on Data Transfer Service transfer configurations, to help meet your organization’s compliance and security requirements.

• Regional endpoints: We enabled regional endpoints for the Data Transfer Service API. Regional endpoints are request endpoints that ensure requests are only processed if the resource exists in the specified location. This way, workloads can comply with data residency and data sovereignty requirements by maintaining data at rest and in transit within the specified location.

• Key tracking: You can now use key usage tracking to see which storage resources are protected by each of your Cloud KMS keys. For more information, learn how to view key usage.

• Proactive threat mitigation: We recently completed a detailed, proactive threat modeling exercise for the entire BigQuery Data Transfer Service. This in-depth review allowed us to identify and mitigate high-priority security risks, further hardening the platform against potential threats.

An intuitive and unified user experience

We’ve made significant investments to the BigQuery user experience to make data ingestion simpler and more intuitive.

The “Add Data” experience in the BigQuery UI now provides a single, simplified entry point to guide you through the data-loading process. Whether you’re a seasoned data engineer or a new analyst, this wizard-like workflow makes it easy to discover and configure transfers from any source, removing the guesswork and getting you to insights faster.

Finally, to further streamline the setup process, the BigQuery Data Transfer Service API is now enabled by default for new BigQuery projects. This removes a manual step, so that data transfer capabilities are immediately available to everyone getting started with BigQuery.

A new, consumption-based pricing model

As we graduate more third-party connectors from preview to GA, we  introduced a new pricing model that reflects their status as fully supported, production-ready services.

This new consumption-based model applies to our third-party SaaS and database connectors (e.g., Salesforce, Facebook Ads, Oracle, MySQL, and others) and takes effect only when a specific connector becomes generally available.

Key details of the model:

• Free in preview: All connectors remain completely free of charge during the preview phase. This allows you to test, experiment, and validate new integrations without any financial commitment.

• Competitive pricing: Pricing is highly competitive, to help you feel comfortable loading data from critical sources.

• Consumption-based: You are billed based on the compute resources consumed by your data transfers, measured in slot-hours.

This change allows us to continue investing in building a robust and scalable data transfer platform. For more detailed information, please visit the official BigQuery pricing page.

Looking ahead

The journey continues! We are committed to building features that streamline your data pipelines and unlock new levels of insight. As you can see from the extensive new list of connectors in preview, we are continuing to innovate rapidly in migration, marketing analytics, operational databases, and enterprise applications.

Experience the power of BigQuery Data Transfer Service for yourself. Simplify your data loading process and accelerate your time to insights. Want to stay informed about the BigQuery Data Transfer Service? Join our email group for future product announcements and updates at https://groups.google.com/g/bigquery-dts-announcements. 

We also encourage readers to share their feedback and file feature requests via the public issue tracker, specifically by Creating a new BigQuery issue.