GCP – New in BigQuery BI Engine: faster insights across popular BI tools

Business analysts working with larger and larger data sets are finding traditional BI methods can’t keep up with their need for speed.  BigQuery BI Engine is designed to meet this need by accelerating that most popular dashboards and reports that connect to BigQuery.  With the freshest data available, your analysts can identify trends faster, reduce risk, match the pace of customer demand, even improve operational efficiency in an ever-changing business climate. 

Today, we are excited to announce that we are extending BigQuery BI Engine to work with any BI or custom dashboarding applications that require sub-second query response times. In this preview, BI Engine will work seamlessly with Looker and other popular BI tools such as Tableau and Power BI without requiring any change to the BI tools. 

“Our business users are hungry for data and expect a lot from our team, so as soon as we had the opportunity to turn on BI Engine for BigQuery and Looker, we did and the performance-matched to the speed of business expectations and greatly improved the load times for dashboards,” said Harish Ramachandraiah, Director of Engineering and Analytics at Sunrun.  

This fast in-memory analysis service allowed the marketing, sales, and operations business users to analyze large and complex datasets interactively in Looker and Data Studio. 

“It helped the development team too, with its simplified architecture for getting started quickly without managing complex, post-ETL transformations and data pipelines. We saw 40% performance increases and we have 1000+ users interacting with the data daily via Looker and Data Studio, so this was a very effective way to improve their data experiences, save them time, and positively impact the data culture at Sunrun,” said Kiran Manne,  Architect on the BI team at Sunrun.

BI Engine is now natively integrated into the BigQuery API. Users of any BI tool or custom applications that connect to BigQuery API via JDBC/ODBC, can now take advantage of BI Engine by reserving BI Engine capacity in their GCP projects and specifying the size of memory. 

Architecturally, we revamped BI Engine with a distributed in-memory execution engine for always-fast experiences. BI Engine utilizes modern techniques such as vectorized processing, advanced data encodings, adaptive caching and query plans tuned for in-memory execution. When queries exceed the size of a reservation, BI Engine seamlessly adapts into a mixed mode operation by using BigQuery Slots.

Customers connecting to BigQuery data through partner BI solutions will immediately see value from BI Engine;

“When a company runs on data, speed, agility and empowerment are critical to achieving competitive advantage. Combining Tableau’s ease of use and flexibility with the power and scale of Google’s BigQuery BI Engine will rapidly unlock the power of data for even more people,” said Francois Ajenstat, Chief Product Officer at Tableau Software.

“Our robust community includes some of the most modern companies in the world.  They value Open Source because of its flexibility, speed of innovation, and modern approach to business intelligence. This new functionality goes a long way in making sure that their queried data is always fast and always fresh!”  – Maxime Beauchemin, CEO at Preset, original creator of Apache Superset and Apache Airflow.

BigQuery BI Engine advantages:

• Super-fast: Match performance to the speed of business by reducing time to insights
  Today, it’s difficult to run reports fast enough to steer your business in a data-driven way by using operational, prescriptive business intelligence. Teams also struggle with sluggish dashboards and stale data. BI Engine provides sub-second query response time with minimal load times and improved concurrency for data stored in BigQuery and many customers have experienced significant query performance improvements using BI Engine. The preview version is optimized for aggregation, filtering, group-by and star joins query patterns. A full list of supported functions and operators can be found here.

• Simplified architecture: Get started quickly without managing complex data pipelines or servers
  Traditional BI systems require users to move data from data warehousing platforms to data marts or BI platforms to support fast interactive analysis. This typically requires complex ETL pipelines for data movement. The time required by these ETL jobs can delay your reporting and compromise the freshness of data for critical decision support systems. BI Engine performs in-place analysis within BigQuery. This eliminates the need to move data or to create complex data transformation pipelines.

• Smart tuning: Very few configuration settings
  BI Engine’s self-tuning design automatically tunes queries by moving data between BI Engine’s in-memory storage, the BigQuery storage to ensure optimal performance and load times for dashboards. BigQuery administrators can easily add and remove BI Engine memory capacity in increments of 1GB, by using the Cloud Console.

Getting started with BI Engine Preview

Access to this preview of BI Engine will be done via an allowlisting process. Simply submit the preview enrollment form with your project details. There is no cost to using BI Engine for the enrolled projects during the first 10 weeks of the preview phase. The no-cost period of the preview begins today and ends 10 weeks later at 11:59 PM on May 6th, 2021. After that date, you will be charged regular BI Engine reservation costs. See link for details. If you have any questions, comments and feedback, join this Google Group and post your questions.  During this preview, users are limited to a maximum of 100 GB of BI Engine capacity for each Google Cloud project. However, you can request an increase in reservation capacity.

Learn more about BI Engine Preview

To learn more about BI Engine’s internal architecture and see a demonstration with Looker and Tableau, check out our talk at Next ‘20 OnAir or register for our upcoming “Delivering fast and fresh data experiences with BigQuery BI Engine & Looker” in March. As always, feel free to read through our documentation for quickstarts, best practices, and how-to guides.