Cloud

The first incarnation of search indexes in BigQuery focused on fast and efficient lookups on STRING data elements, either in standalone STRING scalar columns, or within an ARRAY, STRUCT, or JSON column. Our previous blog posts showcased the orders-of-magnitude performance gains achievable when utilizing indexes with the SEARCH function and other functions and operators.

Today, we are announcing the public preview of numeric search indexes, which enables optimized lookups on INT64 and TIMESTAMP data types. With this change, the EQUAL(=) and IN operations on these data types can utilize search indexes to reduce byte scans for improved performance. So now your lookups for account IDs or transactions IDs or log timestamps can get faster and cheaper.

In this blog, we demonstrate the gains on real data, showcasing index creation and queries on a 100TB log table called log_table that contains Google Cloud Logging data for an internal Google test project.

The base table details are as follows:

The table has the following columns of interest:

jsonPayload: type JSON

This jsonPayload has a leaf field named threadId of type JSON number.

sourceLocation: type RECORD (or STRUCT) with two sub-fields of interest:

file: type STRING, containing the name of the file producing the log entry

line: type INT64, containing the line number in the file where the log entry was produced. 

Using search indexes

By default, a search index is created for the STRING data only. If you want to index INT64 or TIMESTAMP, you need to provide them in the index option called data_types. In the following example, all data of type STRING and INT64 in the log_table table will be indexed.

JSON field search

In this first example, we want to search for log entries that have the thread ID 12104 in the JSON payload.

We compare between having a search index and having no index. Given that log entries with this thread ID are very rare, the results show dramatic improvements on all three metrics:

STRUCT nested field search

In the second example, we count how many log entries are produced from a certain line of code (line 813 in the file borg/borgletlib/borgletlib.cc).

Note that sourceLocation.file is a STRING field. A search index on only STRING data type can already help improve the query performance as shown below. However, with indexing on also the INT64 data type, the performance can be further improved.

Do I need indexing if I use partitioning/clustering?

While partitioning and clustering can optimize filtering and lookups, they have certain limitations. For instance, partitioning can only be done on a single column, and clustering allows up to four columns per table. However, clustering is most effective when filtering on the first clustering column, as subsequent columns often provide minimal pruning power. Furthermore, both partitioning and clustering are limited to top-level columns.

Search indexes on INT64/TIMESTAMP complement these BigQuery features by enabling lookup optimizations on any number of columns. In addition,  as demonstrated above, they cover struct nested fields, array elements, and JSON leaf fields.

This feature is currently in preview. For more information, refer to Optimize with numeric predicates.

Today, we’re announcing two new solar power purchase agreements in Japan that bring us closer to our goal to run on 24/7 carbon-free energy on every grid where we operate by 2030. These Power Purchase Agreements (PPAs) with Itochu’s partner Clean Energy Connect and Shizen Energy are our first in the country, and together they will add a combined 60 megawatts (MW) of new solar energy capacity to the Japanese grid. This will not only support our data centers in the region, but also align with Japan’s clean energy ambitions.

We are expanding our clean energy portfolio in the Asia-Pacific region

Our PPA with Clean Energy Connect, a partner of Itochu Corporation, involves constructing a network of roughly 800 small-scale solar plants across multiple grid regions in Japan. This novel, distributed approach is a creative solution to the challenge of limited land availability for large-scale solar projects in the country. It will generate a significant 40 MW of clean energy to support our operations in Japan.

The PPA with Shizen Energy, a leading renewable energy company in Japan, focuses on the development of a 20 MW utility-scale solar project situated in the same power grid as our recently opened data center in Inzai City, Chiba prefecture.

Through these agreements, we will procure the renewable energy generated from these solar farms across Japan, along with the associated energy attribute certificates. This will significantly reduce our carbon footprint in the region. 

A collaborative effort for a sustainable future

These projects are expected to be fully operational within four years and underscore our commitment to invest nearly $690 million (nearly 100 billion yen) into sustainable infrastructure in Japan.

Signing these PPAs is just the beginning of our decarbonization journey in Japan. We aim to continue our efforts in the region by collaborating with local partners and exploring even more innovative solutions to accelerate the country’s clean energy transition. To learn more about Google data centers, visit google.com/datacenters. To learn more about our sustainability work, visit sustainability.google.

Google Cloud customers are increasingly moving their generative AI workloads from proof-of-concept into production, and are now seeing real-world business impact from their AI investments. Many of these customers have worked with Google Cloud Consulting to apply AI in important and helpful ways. For example, Bristol Myers Squibb developed a new AI-powered interface to help its clinical study teams more easily find important information and generate documents, and Palo Alto Networks launched several new AI tools that utilize Gemini to streamline and enhance user experience in its copilots, improving productivity of security practitioners.

Moving these workloads into production requires a deep understanding of generative AI systems design, large language model architectures, prompt engineering, evaluation, and much more. Now, we’re bringing Google’s expertise in these areas to our customers at-scale, with the launch of a new service offering: Generative AI Ops. This new offering, delivered by either Google Cloud Consulting or via our comprehensive partner ecosystem, will help organizations mature their gen AI prototypes into production-grade solutions and provide support in important areas like security, model tuning and feedback, and optimization. 

With the launch of Generative AI Ops, Google Cloud now offers customers both an open and optimized technology stack for building AI, and a comprehensive set of services to support customers at every stage of their AI transformations — from exploration to production.

The new Generative AI Ops services offering moves customers through the steps required to make AI applications production-ready. These include:

Prompt engineering, design, and optimization: Designing well-optimized prompts is important to ensure models can provide high-quality outputs and build user trust. Using best practices for prompt engineering, and techniques such as ReAct, retrieval augmented generation (RAG), and chain of thought, Google Cloud Consulting can help customers build solutions to improve the performance of their gen AI applications and the outputs of models. Importantly, different models are often suited to different use cases, and each of these models may require different prompting structures. Our expert teams will also help customers apply the right model to the right use case, and to apply the right prompting technique to the right model.
Performance and system evaluation: Successfully putting AI into production requires constant evaluation and feedback to improve the performance of models and applications. This services offering helps customers design and deploy an evaluation framework tailored to their applications, and build mechanisms for automated evaluation metrics using tools like AutoSxS and GenAI Eval, human evaluation, as well as hybrid approaches.
Model optimization and continuous tuning: Once a framework for performance and system evaluation is in place, gen AI applications and models still require continuous tuning and optimization. Gen AI Ops provides solutions and managed services for optimizing and tuning models based on human feedback and benchmarking. This includes improving system architecture and model selection, reducing latency and costs, and incorporating the latest APIs and and available tools to orchestrate and build AI agents using LangChain or DIY orchestrators to ensure applications run optimally.
Monitoring and observability: Having a robust monitoring solution in place is critical to ensuring AI applications are production-ready. Google Cloud Consulting can help customers build observability solutions to constantly monitor the operations and performance of their gen AI applications on a wide variety of factors, like model accuracy and hallucination, latency, throughput, hardware utilization, model drift, traffic, and costs. 
Business integration and testing: It is critical that a customers’ applications and models perform well in real-world scenarios and integrate well with their business processes. Google Cloud Consulting can help customers through the careful planning required to achieve this, including setting up a scalable and secure environment on Google Cloud, designing APIs to efficiently manage interactions with various models, and implementing rigorous unit, integration, and load testing to evaluate their models’ performance under various conditions.

Train and enable customer teams

On top of the business planning and technical steps required to bring AI applications into production, training and team enablement are also critical priorities for customers wanting to see success in their cloud deployments. Through the Google Cloud Skills Boost Platform, Google Cloud offers a broad range of trainings, hands-on labs, bootcamps, and coursework to help upskill teams on generative AI, to ensure that customer teams can build, deploy, use, and manage new AI applications.

Get started

Ready to learn more? Discover how Google Cloud Consulting can help you learn, build, operate and succeed.

Today, Google announced new investments in digital infrastructure and security initiatives designed to increase digital connectivity, accelerate economic growth, and deepen resilience across Africa.

Expanding connectivity in Africa with Umoja

To help increase the reach and reliability of digital connectivity for Africa, today we’re announcing Umoja, the first ever fiber optic route to directly connect Africa with Australia.

Anchored in Kenya, the Umoja cable route will pass through Uganda, Rwanda, Democratic Republic of the Congo, Zambia, Zimbabwe, and South Africa, including the Google Cloud region, before crossing the Indian Ocean to Australia. Umoja’s terrestrial path was built in collaboration with Liquid Technologies to form a highly scalable route through Africa, including access points that will allow other countries to take advantage of the network. 

Umoja, which is the Swahili word for unity, joins Equiano in an initiative called Africa Connect. Umoja will enable African countries to more reliably connect with each other and the rest of the world. Establishing a new route distinct from existing connectivity routes is critical to maintaining a resilient network for a region that has historically experienced high-impact outages.   

We are grateful for the partnership from leaders across Africa and Australia to deliver Africa Connect to people, businesses, and governments in Africa and around the world. 

“Access to the latest technology, supported by reliable and resilient digital infrastructure, is critical to growing economic opportunity. This is a meaningful moment for Kenya’s digital transformation journey and the benefits of today’s announcement will cascade across the region.” – Meg Whitman, U.S. Ambassador to Kenya 

“I am delighted to welcome Google’s investment in digital connectivity, marking a historic milestone for Kenya, Africa, and Australia. The new intercontinental fiber optic route will significantly enhance our global and regional digital infrastructure. This initiative is crucial in ensuring the redundancy and resilience of our region’s connectivity to the rest of the world, especially in light of recent disruptions caused by cuts to sub-sea cables. By strengthening our digital backbone, we are not only improving reliability but also paving the way for increased digital inclusion, innovation, and economic opportunities for our people and businesses.” – H.E. Dr. William S. Ruto, President of the Republic of Kenya

“Diversifying Australia’s connectivity and supporting digital inclusion across the globe are both incredibly important objectives, and Google’s Umoja cable will help to do just that. Australia welcomes Google’s investment and congratulates all those involved in undertaking this crucial initiative.” – Hon Michelle Rowland MP, Australian Minister for Communications

“Africa’s major cities including Nairobi, Kampala, Kigali, Lubumbashi, Lusaka, and Harare will no longer be hard-to-reach endpoints remote from the coastal landing sites that connect Africa to the world. They are now stations on a data superhighway that can carry thousands of times more traffic than currently reaches here. I am proud that this project helps us deliver a digitally connected future that leaves no African behind, regardless of how far they are from the technology centers of the world.” – Strive Masiyiwa, Chairman and founder of Liquid  

Extending our collaboration

In addition to today’s infrastructure announcement, Google will sign a Statement of Collaboration with Kenya’s Ministry of Information Communications and The Digital Economy to accelerate joint efforts in cybersecurity, growing data-driven innovation, digital upskilling, and responsibly and safely deploying AI for societal benefits.

Strengthening cybersecurity with cloud-powered solutions 

As part of the collaboration, Google Cloud and Kenya are announcing that they intend to work together on strengthening Kenya’s cybersecurity. The Department of Immigration & Citizen Services is evaluating Google Cloud’s CyberShield solution and Mandiant expertise to strengthen the defense of its eCitizen platform. CyberShield enables governments to build enhanced cyberthreat capabilities, protect web-facing infrastructure, and helps teams develop skills and processes that drive effective security operations.

Realizing the benefits of digital transformation

Google has long recognized the critical role investments in secure technology infrastructure have on connecting communities, expanding education, and driving healthy economic development within Africa and around the world.  

Since Google opened our first Sub-Saharan Africa office in Nairobi in 2007, we have partnered with governments from countries across Africa on numerous digital initiatives. In 2021, we committed to invest $1 billion in Africa over five years to support a range of efforts, from improved connectivity to investment in startups, to help boost Africa’s digital transformation. Since then, Google has invested more than $900 million in the region, and we expect to fulfill our commitment by 2026. The collaboration introduced this week is the latest step towards delivering on our broader commitment to support Africa’s digital transformation, continued economic growth, and innovation. 

Supporting economic growth: Between 2021 and 2023, third party estimates show that Google’s products and services provided more than $30 billion of economic activity across Sub-Saharan Africa. Africa’s internet economy has the potential to grow to $180 billion by 2025 – 5.2% of the continent’s GDP, according to a report by the International Finance Corporation. Investments like Umoja, coupled with developing talent who can benefit from, and add to, this growing digital economy, will help ensure citizens can access government services and critical information, while enabling businesses to thrive and generate durable economic growth for the local economy.  

Skilling: Our training and certification initiatives help entrepreneurs get more out of the web, using digital technologies to build and sustain businesses and, in doing so, help generate durable economic growth for the local economy. For example, the Google Hustle Academy, a five-day bootcamp launched in 2022 focusing on subjects including leadership, business strategy, and e-commerce, has supported the growth of more than 3,500 small businesses in Kenya.

AI innovations created in Africa, for Africa: Through our AI Research Centers in Ghana and Nairobi, as well as the Product Development Center in Kenya, Google is continuing to build products and services to help tackle challenges across the continent. For example, in Kenya, Google partnered with Jacaranda Health to improve maternal health outcomes with expanded access to ultrasounds. Google is also working with Kenyan health organizations, including IntelliSOFT, Ona, and Medtronic Labs, to enhance the interoperability of digital health solutions. In addition, Google is running workshops with Kabarak University as part of Google’s efforts to support digital health innovation in Kenya.

Google is as committed as ever to partnering with communities, businesses, and governments in Africa to help foster even more innovation across the continent, and we are excited about this next chapter for Kenya and the region.