Cloud

Tis the season for learning new skills! Get ready for 12 Days of Learning, a festive digital advent calendar packed with courses, hands-on labs, videos, and community opportunities—all designed to boost your generative AI expertise. Discover a new learning resource on Google Cloud’s social channels every day for twelve days this December. 

Before you start: Get no-cost access to generative AI courses and labs

Join the Innovators community to activate 35 monthly learning credits in Google Cloud Skills Boost at no cost. Use these credits to access courses and labs throughout the month of December—and beyond! 

Ready to get started? Review all of the resources below.

Get festive with generative AI foundations

Learn how to use gen AI in your day-to-day work. These resources are designed for developers looking to gain foundational knowledge in gen AI.

1. A Developer’s Guide to LLMs: In this 10-minute video, explore the exciting world of large language models (LLMs). Discover different AI model options, analyze pricing structures, and delve into essential features.

2. Responsible AI: Fairness & Bias: This course introduces the concepts of responsible AI and shares practical methods to help you implement best practices using Google Cloud products and open source tools.

3. Gemini for end-to-end SDLC: This course explores how Google Cloud’s Gemini AI can assist in all stages of the software development lifecycle, from building and debugging web applications to testing and data querying. The course ends with a hands-on lab where you can build practical experience with Gemini.

4. Responsible AI for Developers: Interpretability & Transparency: This course introduces AI interpretability and transparency concepts. Learn how to train a classification model on image data and deploy it to Vertex AI to serve predictions with explanations.

5. Introduction to Security in the World of AI: This course equips security and data protection leaders with strategies to securely manage AI within their organizations. Bring these concepts to life with real-world scenarios from four different industries.

Cozy up with gen AI use cases

Launch these courses and labs to get more in-depth, hands-on experience with generative AI, from working with Gemini models to building agents and applications. 

1. Build Generative AI Agents with Vertex AI and Flutter: Learn how to develop an app using Flutter and then integrate the app with Gemini. Then use Vertex AI Agent Builder to build and manage AI agents and applications.

2. Machine Learning Ops (MLOps) for Generative AI: In this course, learn how to overcome MLOps challenges and use generative AI to streamline processes.

3. Boost productivity with Gemini in BigQuery: This course provides an overview of features to assist in the data-to-AI workflow, including data exploration and preparation, code generation, workflow discovery, and visualization. 

4. Build Generative AI Apps with Firebase Genkit: Learn how to integrate gen AI features into your applications using Firebase Genkit—from prototyping to production. 

5. Website Modernization with Generative AI on Google Cloud: Transform your website experiences with gen AI with this hands-on course. Learn how to build a generative search experience in an interactive lab.

6. Work with Gemini models in BigQuery: Through a practical use case involving customer relationship management, learn how to solve a real-world business problem with Gemini models. Plus, receive step-by-step guidance through coding solutions using SQL queries and Python notebooks.

Get a jump-start on your New Year’s resolutions with AI Skills Quest

Get an early start on your learning goals by signing up for AI Skills Quest, a monthly learning challenge that puts gen AI on your resume with verifiable skill badges. When you sign up, choose your path based on your level of knowledge:

• Beginner/Intermediate cohort: Learn fundamental AI concepts, prompt design, and Gemini app development techniques in Vertex AI, plus other Gemini integrations in various technical workflows.

• Advanced cohort: Already know the basics, but want to add breadth and depth to your AI skills? Sign up for the Advanced path to learn advanced AI concepts like RAG and MLOps.

Ready to ring in the new year with new skills? Find more generative AI learning content on Google Cloud Skills Boost.

2024 has been a landmark year for Google Earth Engine, marked by significant advancements in platform management, cloud integration, and core functionality. With increased interoperability between Google Cloud tools and services, and Earth Engine, we’ve unlocked powerful new workflows and use cases for our users.  Here’s a round up of this year’s top Earth Engine launches, many of which were highlighted in our Geo for Good 2024 summit.

Management: Streamlining Workflows

Earlier this year, we launched the new Earth Engine Overview page in the Cloud Console, serving as a centralized hub for Earth Engine resources, allowing you to manage Earth Engine from the same console used to manage and monitor other Cloud services. 

In this console, we also introduced a new Tasks page, allowing you to view and monitor Earth Engine export and import tasks alongside usage management and billing. The Tasks page provides a useful set of fields for each task, including state, runtime, and priority. Task cancellation is also easier than ever with single or bulk task cancellation in this new interface. 

As we deepen Earth Engine’s interoperability across Google Cloud, we’ll be adding more information and controls to the Cloud Console so that you can further centralize the management of Earth Engine alongside other services.

Integrations: deepening cloud interoperability

Earth Engine users can integrate with a number of cloud services and tools to enable advanced solutions requiring custom machine learning and robust data analytics. This year, we launched a set of features that improved existing interoperability, making it easier to both enable and deploy these solutions.  

Vertex AI integration
Using Earth Engine with Vertex AI enables use cases that require deep learning, such as crop classification. You can host a model in Vertex AI and get predictions from within the Earth Engine Code Editor. This year, we announced a major performance improvement to our Vertex Preview connector, which will give you more reliability and more throughput than the current Vertex connector.

Earth Engine access
To ensure all Earth Engine users can take advantage of these new integration improvements and management features, we’ve also transitioned all Earth Engine users to Cloud projects. With this change, all Earth Engine users can now leverage the power and flexibility of Google Cloud’s infrastructure, security, and growing ecosystem of tools to drive forward the science, research, and operational decision making required to make the world a better place.

Security: enhancing control

This year we launched Earth Engine support for VPC Service Controls – a key security feature that allows organizations to define a security perimeter around their Google Cloud resources. This new integration, available to customers with professional and premium plans, provides enhanced control over data, and helps prevent unauthorized access and data exfiltration. With VPC-SC, customers can now set granular access policies, restrict data access to authorized networks and users, and monitor and audit data flows, ensuring compliance with internal security policies and external regulations.

Platform: improving performance

Zonal Statistics
Computing statistics about regions of an image is a core Earth Engine capability. We recently launched a significant performance improvement to batch zonal statistics exports in Earth Engine. We’ve optimized the way we parallelize zonal statistics exports, such as exports that generate statistics for all regions in a large collection. This means that you will get substantially more concurrent compute power per batch task when you use ReduceRegions().

With this launch, large-scale zonal statistics exports are running several times faster than this time last year, meaning you get your results faster, and that Earth Engine can complete even larger analyses than ever. For example, you can now calculate the average tree canopy coverage of every census tract in the continental United States at 1 meter scale in 7 hours. Learn more about how we sped up large-scale zonal statistics computations in our technical blog post.

Python v1
Over the last year, we’ve focused on ease-of-use, reliability, and transparency for Earth Engine Python. The client library has moved into an open-source repository at Google which means we can sync changes to GitHub immediately, keeping you up-to-date on changes between releases. We are also sharing  pre-releases, so you can see and work with Python library candidate releases before they come out. We have a static loaded client library, which makes it easier to build on our Python library and better testing and error messaging. We’ve also continued making progress on improving geemap and integrations like xee.

With all of these changes, we’re excited to announce that the Python Client library is now ‘v1’, representing the maturity of Earth Engine Python. Check out this blog post to read more about these improvements and see how you can take full advantage of Python and integrate it into Google’s Cloud tooling.

COG-backed asset improvements
If you have data stored in Google Cloud Storage (GCS), in Cloud-Optimized GeoTIFF (COG) format, you can easily use it in Earth Engine via Cloud Geotiff Backed Earth Engine Assets, improving the previous experience requiring a single file GeoTIFF, where all bands have the same projection and type.

Now you can create an Earth Engine asset backed by multiple GeoTiffs, which may have different projections, different resolutions, and different band types–and Earth Engine will take care of these complexities for you. There are also major performance improvements to the previous feature: Cloud GeoTiff backed assets now have similar performance to native Earth Engine assets. In addition, If you want to use your GCS COGs elsewhere, like open source pipelines or other tools, the data is stored once and you can use it seamlessly across products.

Looking forward to 2025

We’re excited to see Earth Engine users leverage more advanced tools, stronger security, and seamless integrations to improve sustainability and climate resilience. In the coming year, we’re looking forward to further deepening cloud interoperability, making it easier to develop actionable insights and inform sustainability decision-making through geospatial data.

Conventional fraud detection methods have a hard time keeping up with increasingly sophisticated criminal tactics. Existing systems often rely on the limited data of individual institutions, and this hinders the detection of intricate schemes that span multiple banks and jurisdictions. 

To better combat fraud in cross-border payments, Swift, the global provider of secure financial messaging services, is working with Google Cloud to develop anti-fraud technologies that use advanced AI and federated learning.  

In the first half of 2025, Swift plans to roll out a sandbox with synthetic data to prototype learning from historic fraud, working with 12 global financial institutions, with Google Cloud as a strategic partner. This initiative builds on Swift’s existing Payment Controls Service (PCS), and follows a successful pilot with financial institutions across Europe, North America, Asia and the Middle East.

The partnership: Google Cloud and Swift 

Google Cloud is collaborating with Swift — along with technology partners including Rhino Health and Capgemini — to develop a secure, privacy-preserving solution for financial institutions to combat fraud. This innovative approach uses federated learning techniques, combined with privacy-enhancing technologies (PETs), to enable collaborative intelligence without compromising proprietary data.

Rhino Health will develop and deliver the core federated learning platform, and Capgemini will manage the implementation and integration of the solution.

“Swift is in a unique position in the financial industry – a trusted and cooperative network that is integral to the functioning of the global economy. As such, we are ideally placed to lead collaborative, industry-wide efforts to fight fraud. This exploration will help the community validate whether federated learning technology can help financial institutions stay one step ahead of bad actors through sharing of fraud labels, and in turn enabling them to provide an enhanced cross-border payments experience to their customers,” said Rachel Levi, head of artificial intelligence, Swift. 

“At Google Cloud, we are committed to empowering financial institutions with cutting-edge technology to combat the evolving threat of fraud. Our collaboration with Swift exemplifies the transformative potential of federated learning and confidential computing. By enabling secure collaboration and knowledge sharing without compromising data privacy, we are fostering a safer and more resilient financial ecosystem for everyone,” said Andrea Gallego, Managing Director, global GTM incubation, Google Cloud,  

The challenge: Traditional fraud detection is falling behind

The lack of visibility across the payment lifecycle creates vulnerabilities that can be exploited by criminals. A collaborative approach to fraud modeling offers significant advantages over traditional methods in combating financial crimes. To be effective, this approach requires data sharing across institutions, which is often restricted because of privacy concerns, regulatory requirements, and intellectual property considerations.

The solution: Federated learning

Federated learning offers a powerful solution for collaborative AI model training without compromising privacy and confidentiality. Instead of requiring financial institutions to pool their sensitive data, the model training occurs within financial institutions on decentralized data.

Here’s how it works for Swift: 

1. A copy of Swift’s anomaly detection model is sent to each participating bank. 

2. Each financial institution trains this model locally on their own data.

3. Only the learnings from this training — not the data itself — are transmitted back to a central server for aggregation, managed by Swift. 

4. The central server aggregates these learnings to enhance Swift’s global model. 

This approach significantly minimizes data movement and ensures that sensitive information remains within each financial institution’s secure environment. 

Core benefits of the federated learning solution

By using federated learning solutions, financial institutions can achieve substantial benefits, including:  

• Shared intelligence: Financial institutions work together by sharing information on fraudulent activities, patterns, and trends, which creates a much larger and richer decentralized data pool than any single institution could gather alone.

• Enhanced detection: The collaborative global model can identify complex fraud schemes that might go unnoticed by individual institutions, leading to improved detection and prevention.

• Reduced false positives: Sharing information helps refine fraud models, leading to more accurate identification of genuine threats and fewer false alarms that disrupt both legitimate activity and the customer experience.

• Faster adaptation: The collaborative approach allows for faster adaptation to new fraud trends and criminal tactics. As new threats emerge, the shared knowledge pool helps all participants quickly adjust their models and their fraud prevention tools.

• Network effects: The more institutions participate, the more comprehensive the data pool becomes, creating a powerful network effect that strengthens fraud prevention for everyone involved.

For widespread adoption, federated learning must seamlessly integrate with existing financial systems and infrastructure. This allows financial institutions to easily participate and benefit from the collective intelligence without disrupting their operations.

Architecting the global fraud AI solution

The initial scope remains a synthetic data sandbox centered on prototyping learning from historic payments fraud. The platform allows multiple financial institutions to train a robust fraud detection model while preserving the confidentiality of their sensitive transaction data. It uses federated learning and confidential computing techniques, such as Trusted Execution Environments (TEEs), to enable secure, multi-party machine learning without training data movement.

There are several key components to this solution:

• Federated server in TEE execution environment: A secure, isolated environment where a federated learning (FL) server orchestrates the collaboration of multiple clients by first sending an initial model to the FL clients. The clients perform training on their local datasets, then send the model updates back to the FL server for aggregation to form a global model.

• Federated client: Executes tasks, performs local computation and learning with local dataset (such as  data from an individual financial institution), then submits results back to FL server for secure aggregation.

• Bank-specific encrypted data: Each bank holds its own private, encrypted transaction data that includes historic fraud labels. This data remains encrypted throughout the entire process, including computation, ensuring end-to-end data privacy.

• Global fraud-based model: A pre-trained anomaly detection model from Swift that serves as the starting point for federated learning.

• Secure aggregation: Using a Secure Aggregation protocol to compute these weighted averages would ensure that the server learns only the historic fraud labels from participating financial institutions, but not exactly which  financial institution, thereby preserving the privacy of each participant in the federated learning process.

• Global anomaly detection trained model and aggregated weights: The improved anomaly detection model, along with its learned weights, is securely exchanged back to the participating financial institutions. They can then deploy this enhanced model locally for fraud detection monitoring on their own transactions.

We’re seeing more enterprises adopt federated learning to combat global fraud, including global consulting firm Capgemini. 

“Payment fraud stands as one of the greatest threats that undermines the integrity and stability of the financial ecosystem, with its impact acutely felt upon some of the most vulnerable segments of our society,” said Sudhir Pai, chief technology and innovation officer, Financial Services, Capgemini. 

“This is a global epidemic that demands a collaborative effort to achieve meaningful change. Our application of federated learning is grounded with privacy-by-design principles, leveraging AI to pioneer secure aggregation and anonymization of data which is of primary concern to large financial institutions. The potential to apply our learnings within a singular global trained model across other industries will ensure we break down any siloes and combat fraud at scale,” he said. 

“We are proud to support Swift’s program in partnership with Google Cloud and Capgemini,” said Chris Laws, chief operating officer, Rhino. “Fighting financial crime is an excellent example of the value created from the complex multi-party data collaborations enabled by federated computing, as all parties can have confidence in the security and confidentiality of their data.”

Building a safer financial ecosystem, together

This effort to fight fraud collaboratively will help build a safer and more secure financial ecosystem. By harnessing the power of federated learning and adhering to strong principles of data privacy, security, platform interoperability, confidentiality, and scalability, this solution has the potential to redefine how we combat fraud in the age of fragmented globalized finance and demonstrates a commitment to building a more resilient and trustworthy financial world.

A few months back, we kicked-off Network Performance Decoded, a series of whitepapers sharing best practices for network performance and benchmarking. Today, we’re dropping the second installment – and this one’s about some of those pesky performance limiters you’re bound to run into. In this blog, we’re giving you the inside scoop on how to tackle these issues head-on. 

First up: A Brief Look at Network Performance Limiters

This whitepaper busts the myth that network interface card speed is the be-all and end-all of network performance. Sure, high Mbit/s or Gbit/s sounds great, but packet size and offloading techniques can make a huge difference in how efficiently your network actually runs. Here’s a sneak peek:

• Mbit/s isn’t everything: It’s not just about raw speed. How you package your data (packet size) seriously impacts throughput and CPU usage.

• Bigger packets, better results: Larger packets mean less overhead per packet, which translates to better throughput and less strain on your CPU.

• Offload for a TCP boost: TCP Segmentation Offload (TSO) and Generic Receive Offload (GRO) let your network interface card do some of the heavy lifting, freeing up your CPU and giving your TCP throughput a nice bump — even with smaller packets.

• Watch out for packet-per-second limits: Smaller packets can sometimes hit a bottleneck because of how many packets your system can handle per second.

• At a constant bitrate, bigger packets are more efficient: Even with a steady bitrate, larger packets mean fewer packets overall, which leads to less CPU overhead and a more efficient network.

Get a handle on these concepts and you’ll be able to fine-tune your network for better performance and efficiency, no matter the advertised speed.

Next: A Brief Look at Round Trip Time

This whitepaper dives into TCP Round Trip Time (RTT) — a key network performance metric. You’ll learn how it’s measured, what can throw it off, and how to use that info to troubleshoot network issues like a pro. We’ll show you how the receiving application’s behavior can mess with RTT measurements, and call out some important nuances to consider. For example, TCP RTT measurements do not include the time TCP may spend resending lost segments, which your applications see as latency. Lastly, we’ll show how you can use tools like netperf (also included in our PerfKit Benchmarker toolkit) to get an end-to-end picture.

Finally: A Brief Look at Path MTU Discovery

Last but not least, this whitepaper breaks down Path MTU discovery, a process that helps prevent IP fragmentation. Understanding how networks handle packet sizes can help you optimize your network setup, avoid those frustrating fragmentation issues, and troubleshoot effectively. We’ll even walk you through common problems — like those pesky ICMP blocks leading to large packets being dropped without any notification to the sender — and how to fix them. Plus, you’ll learn the difference between Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS) — knowledge that’ll come in handy when you’re configuring your network and troubleshooting packet size problems.

Stay tuned!

These resources are part of our mission to create an open, collaborative space for network performance benchmarking and troubleshooting. The examples might be from Google Cloud, but the ideas apply everywhere – regardless of where your workloads may be running. You can find all our whitepapers (past, present, and future) on our webpage. Keep an eye out for more!

It’s been more than two and a half years since we introduced AlloyDB for PostgreSQL, our 100% PostgreSQL-compatible database that offers superior performance, availability, and scale. AlloyDB reimagines PostgreSQL with Google’s cutting-edge technology. It includes a scale-out architecture, built-in analytics, and AI/ML-powered management for a truly modern data experience, and is fully managed so you can focus on your application.

PostgreSQL has long been a favorite among developers for its flexibility, reliability, and robust feature set. AlloyDB brings PostgreSQL to the next level with faster performance, stronger functionality, better migration options, and smarter AI capabilities. 

As 2024 comes to a close, it felt like a great time to celebrate with a snazzy AlloyDB overview video and summarize the AlloyDB’s key benefits in an infographic. Whether you’re new to the product or have tried it already, take a look to make sure you’re taking advantage of every benefit. You can also download our in-depth AlloyDB e-book for a deeper dive.

Ready to dive deeper?

Want to learn more about how AlloyDB can revolutionize your PostgreSQL experience? Download our in-depth AlloyDB e-book and discover the transformative ways AlloyDB is redefining what’s possible. You’ll uncover:

• How AlloyDB delivers superior transactional performance at half the cost

• Why AlloyDB is the best database service for building gen AI apps

• The flexibility of running AlloyDB anywhere, on any platform

• How AI-driven development and operations can simplify your database journey

• The power of real-time business insights with AlloyDB’s columnar engine

The future of PostgreSQL is here, and it’s built for you. Start building your next great app with a 30-day AlloyDB free trial.

Think about your favorite apps – the ones that deliver instant results from massive amounts of data. They’re likely powered by vector search, the same technology that fuels generative AI.

Vector search is crucial for developers who need to build applications that are lightning-fast, handle massive datasets, and remain cost-effective, even with huge spikes in traffic. But building and deploying this technology can be a real challenge, especially for gen AI applications that demand incredible flexibility, scale, and speed. In a previous blog post, we showed you how to create production-ready AI applications with features like easy filtering, automatic scaling, and seamless updates. 

Today, we’ll share how Vertex AI’s vector search is tackling these challenges head-on. We’ll explore real-world performance benchmarks demonstrating incredible speed and scalability – all in a cost-effective way.

How does Vertex AI vector search work? 

Imagine you own a popular online store: to keep shoppers happy, your search engine needs to instantly sift through millions of products and deliver relevant results, even during peak shopping seasons. Vector search is a technique for finding similar items within massive datasets. It works by converting data, like text or images, into numerical representations called embeddings. These embeddings capture the semantic meaning of the data, allowing for more accurate and relevant search results. 

For example, imagine your customers are searching for a “navy blue dress shirt.” A keyword search might miss products labeled “midnight blue button-down,” even though they’re essentially the same. Vector search does a better job of surfacing the right products because it uses embeddings to understand the relationships between words and concepts.

You can use it for a wide range of applications, like the e-commerce example shared above, or as a retrieval-augmented generation (RAG) system for generative AI agents, where it grounds responses in your data or recommendation systems that deliver personalized suggestions based on user preferences.

As Xun Wang, Chief Technology Officer of Bloomreach, recently said, “Bloomreach has made the strategic decision to replace OpenAI with Google Vertex AI Embeddings and Vertex AI vector search. Google’s platform delivers clear advantages in performance, scalability, reliability and cost optimization. We’re confident this move will drive significant benefits and we’re thrilled to embark on this new partnership.”

Real-world impact of Vertex AI’s vector search

Our customers are achieving remarkable results with vector search. Here are four standout ways this technology is helping them build high-performance gen AI apps.

#1: The fastest vector search for highly responsive applications

To meet customer expectations, fast response times are critical across search, recommendation systems, and gen AI applications. Studies have consistently found that faster response times directly contribute to an increase in revenue, conversion, and retention.

Vector search is engineered for incredibly low latency at high quality, while maintaining cost-effectiveness.  In our testing, vector search was able to maintain ultra-low latency (9.6ms at P95) and high recall (0.99) while scaling up to 5K queries per second on a dataset of one billion vectors. By achieving such low latencies, Vertex AI vector search ensures that users receive fast, relevant responses, no matter how large the dataset or how many parallel requests hit the system.

As Yuri M. Brovman from eBay wrote in a recent blog post, “[eBay’s vector search] hit a real-time read latency of less than 4ms at 95%, as measured server-side on the Google Cloud dashboard for vector search”.

#2: Massively scalable for all application sizes

Another important consideration in production-ready applications is the ability of your application to support growing data sizes and user bases. 

This means it can easily accommodate sudden spurts in demand, making it massively scalable for applications of any size. Vertex AI vector search can scale up to support billions of embeddings and hundreds of thousands of queries per second while maintaining ultra low latency.

#3: Up to 4X more cost effective

Vertex AI vector search not only maintains performance at scale, but it is also 4x more cost effective than competing solutions, especially for high performance applications. With Vertex AI vector search’s ANN index, you will need significantly less compute for fast and relevant results at scale.

^{Vertex AI vector search’s real-world benchmarks of public datasets by using 2 replicas of n2d machines. Latency was measured at the provided QPS; vector search can scale up beyond this throughput by adding additional replicas.}

#4: It’s highly configurable for all application types

In some scenarios, developers might be interested in trading-off latency for higher recall (or vice versa). For example, an e-commerce website might prioritize speed for quick product suggestions, while a research database might prioritize comprehensive results even if it takes slightly longer. Vector search enables tuning these parameters and hitting higher recall or higher latency, to match business needs. 

Additionally, vector search supports auto-scaling – and when load on the deployment increases, it scales to maintain performance. We measured auto-scaling and found that vector search was able to maintain consistent latency with high recall, as QPS increased from 1K to 5K.

Developers can also increase the number of replicas in order to handle higher throughput, as well as pick different machine types to balance cost and performance. This flexibility makes vector search suitable for a wide range of applications beyond semantic search, including recommendation systems, chatbots, multimodal search, anomaly detection, and image similarity matching.  

Going further with hybrid search

Dense embedding-based semantic search, while excellent at understanding meaning and context, has a weak point: it cannot find items that the embedding model can’t make sense of. Items like product numbers, company’s internal codenames or newly coined terms, aren’t found by semantic search because the embedding model doesn’t understand their meanings. 

With Vertex AI vector search’s hybrid search, building this type of sophisticated search engine is no longer a daunting task. Developers can easily create a single index that incorporates both dense and sparse embeddings, representing semantic meaning and keyword relevance respectively. This streamlined approach allows for rapid development and deployment of high-performance search applications, fully customized to meet specific business needs.

As Nicolas Presta, Sr. Engineering Manager at Mercado Libre wrote, “Most of our successful sales start with a search, so it is important that we give precise results that best match a user’s query. These complex searches are getting better with the addition of the items retrieved from vector search, which will ultimately increase our conversion rate. Hybrid search will unlock more opportunities to uplevel our search engine so that we can create the best customer experience while improving our bottom line.” – Nicolas Presta, Sr. Engineering Manager at Mercado Libre.