GCP – Simplify data loading with new enhancements to BigQuery Data Transfer Service

BigQuery makes it easy for you to gain insights from your data, regardless of its scale, size or location. BigQuery Data Transfer Service (DTS) is a fully managed service that automates the loading of data into BigQuery from a variety of sources. DTS supports a wide range of data sources, including first party data sources from the Google Marketing Platform (GMP) such as Google Ads, DV360 and SA360 etc. as well as cloud storage providers such as Google Cloud Storage, Amazon S3 and Microsoft Azure.

BigQuery Data Transfer Service offers a number of benefits, including:

Ease of use: DTS is a fully managed service, so you don’t need to worry about managing infrastructure or writing code. DTS can be accessed via UI, API or CLI. You can get started with loading your data in a few clicks within our UI.Scalability: DTS can handle large data volumes and a large number of concurrent users. Today, some of our largest customers move petabytes of data per day using DTS.Security: DTS uses a variety of security features to protect your data, including encryption and authentication. More recently, DTS has added capabilities to support regulated workloads without compromising ease-of-use.

Recent feature updates

Based on customer feedback, we launched a number of key feature updates in the second half of 2023. These updates include broadening our support for Google’s first party data sources, enhancing the current portfolio of connectors to incorporate customer feedback and improve user experience, platform updates to support regulated workloads, as well as other important security features.

Let’s go over these updates in more detail.

Supporting new data sources

Display and Video Ads 360 (DV360) – A new DTS connector (in preview) allows you to ingest campaign config and reporting data into BigQuery for further analysis. This connector will be beneficial for customers who want to improve their campaign performance, optimize their budgets, and target their audience more effectively.Azure Blob Storage – The new DTS connector for Azure Blob Storage and Azure Data Lake Storage (now GA) completes our support for all the cloud storage providers. This allows customers to automatically transfer data from Azure Blob Storage into BigQuery on a scheduled basis.Search Ads 360 (SA360) – We launched a new DTS connector (in preview) to support the new SA360 API. This will replace the current (and now deprecated) connector for SA360. Customers are invited to try the new connector and prepare for migration (from the current connector). The new connector is built using the latest SA360 API that includes platform enhancements as documented here.

Enhancing existing connectors

Google Ads: Earlier in 2023, we launched a new connector for Google Ads that incorporated the new Google Ads API. It also added support for Performance Max campaigns.

Recently, we enhanced the connector with a feature (in preview) to enable customers to create custom reports that use a Google Ads Query Language (GAQL) query. Customers have to use their existing GAQL queries on configuring a transfer. With this feature, customers will have an optimized data transfer that pulls in just the data they need and at the same time bring in newer fields that might not be supported in a standard Google Ads transfer.

Additionally, we added support for Manager Accounts (MCC) with a larger number of accounts (8K) for our larger enterprise customers.YouTube Content Owners: Expanded support for YouTube Content Owner by increasing the coverage of CMS reports by adding 27 new financial reports. You can find the details of all the reports supported here.Amazon S3: Enabled hourly ingestion from Amazon S3 (vs. daily ingestion previously) to improve data freshness. This has been a consistent customer requested feature to improve the freshness of data from S3.

Enhanced security and governance

Your data is only useful if you can access it in a trusted and safe way. This is particularly important for customers running regulated workloads. To that end, we enabled:

Customer-managed encryption keys (CMEK) support for scheduled queries, Cloud Storage transfers, and Azure Blob Storage transfersService Account support for additional data sources: Campaign Manager, Teradata, Google Play, Amazon Redshift, Search Ads 360, and Display & Video 360.Data residency and access transparency controls for scheduled queries and Cloud Storage transfers.Enabled support for Workforce Identity Federation for all data sources except for YouTube Channel Transfers.

Looking forward

As we look to 2024, we are excited to continue this rapid innovation with new connectors for marketing analytics, operational databases and enterprise applications. At the same time, we look to evolve our portfolio of connectors with capabilities that improve the “freshness” of data, and help customers with AI-assisted transformations.

Try BigQuery Data Transfer Service today and streamline loading your data into BigQuery!