GCP – Real-time data for real-world AI with support for Apache Flink in BigQuery

Today’s organizations aspire to become “by-the-second” businesses, capable of adapting in real time to changes in their supply chain, inventory, customer behavior, and more. They also strive to provide exceptional customer experiences, whether it’s through a support interaction or an online checkout process. We believe that real-time intelligence should be accessible to all businesses, regardless of their size or budget and should be integrated into a unified data platform, so that everything works together. Today, we’re taking a big step toward helping businesses realize these aspirations, with BigQuery Engine for Apache Flink, now in preview. 

Introducing BigQuery Engine for Apache Flink: Familiar Flink, now serverless 

BigQuery Engine for Apache Flink provides a state-of-the art real-time intelligence platform, empowering customers to:

Use familiar streaming technologies on Google Cloud. BigQuery Engine for Apache Flink makes it easier to lift and shift existing streaming applications relying on open-source Apache Flink to Google Cloud, without rewriting code or relying on third-party services. Combined with Google Managed Service for Apache Kafka (now GA), it is easy to migrate and modernize your streaming analytics on Google Cloud.

Reduce operational burden. BigQuery Engine for Apache Flink is entirely serverless, reducing operational burden and allowing customers to focus on what they do best — innovate their businesses.

Bring real-time data to AI. Enterprise developers experimenting with gen AI are looking for a well-integrated and scalable streaming platform that’s based on familiar technologies — Apache Flink and Apache Kafka — and that they can combine with Google’s differentiated AI/ML capabilities in BigQuery.

BigQuery Engine for Apache Flink arrives during a time when Google Cloud customers are leveraging many innovations in real-time analytics, including BigQuery continuous queries, which enables customers to analyze incoming data in BigQuery in real time using SQL, and Dataflow Job Builder, which helps customers define and deploy a streaming pipeline using a visual UI.  

With BigQuery Engine for Apache Flink, our streaming portfolio now spans SQL-based easy streaming with BigQuery continuous queries, popular open-source Flink and Kafka platforms, and advanced multimodal data streaming with Dataflow, including support for Iceberg. These capabilities are integrated with BigQuery, which connects your data with industry leading AI, including Gemini, Gemma and open models.

New AI capabilities unlocked when your data is real-time

As we look ahead, it’s clear that generative AI has reignited interest in the potential of data-driven insights and experiences. AI, especially generative AI, is most effective when it has access to the latest context. If you’re a retailer, you can combine historical purchase data with real-time interactions to personalize shopping experiences for your customers. If you’re a financial services company, you can use up-to-the-second transactions to refine your fraud detection model. Real-time data connected to AI means fresh data for training models, real-time user assistance with Retrieval Augmented Generation (RAG), and real-time predictions and inferences for your business applications, including integrating small models like Gemma into your streaming pipelines.  

We are taking a platform approach to introduce capabilities across the board so that, no matter what specific streaming architecture you need, or which streaming engine you prefer, you have the ability to leverage real-time data for your gen AI use cases. Features such as Dataflow enrichment transforms, support for Vertex AI text-embeddings, the RunInference transform, distributed counting in Bigtable, and many others make the task of building real-time AI applications easier than ever.   

We are very excited to get these capabilities into your hands and continue giving you more flexibility and choice when it comes to making your unified data and AI platform operate in real-time data. Learn more about BigQuery Engine for Apache Flink and get started using it today in the Google Cloud console.