GCP – Inside Mercado Libre’s multi-faceted Spanner architecture

Mercado Libre, the e-commerce and fintech pioneer of Latin America, operates at a staggering scale, demanding an infrastructure that’s not just resilient and scalable, but also a catalyst for rapid innovation. While our use of Spanner for foundational consistency and scale is known, a deeper dive reveals a sophisticated, multi-layered strategy. Spanner is not just a database here; it’s a core engine powering our internal developer platform, diverse data models, advanced analytics loops, intelligent features, and even our roadmap for next-generation AI applications.

This blog explores the technical underpinnings of how Mercado Libre leverages Spanner in concert with our internal innovations like the Fury platform, achieving significant business impact and charting a course for an AI-driven future.

The dual challenge: internet-scale operations and developer velocity

Mercado Libre faces the classic challenges of internet-scale services: keeping millions of daily financial transactions safe, making it easy for developers to build apps, and maintaining near-perfect uptime. The solution required a database powerful enough for the core and an abstraction layer elegant enough for broad developer adoption.

Fury: Mercado Libre’s developer gateway

At the heart of Mercado Libre’s strategy is Fury, our in-house middleware platform. Fury is designed to abstract away the complexities of various backend technologies, providing developers with standardized, simplified interfaces to build applications. 

• Abstraction & Standardization: Fury allows development teams to focus on business logic rather than the nuances of distributed database management, schema design for specific engines, or optimal connection pooling.

• Spanner as the Reliable Core: Spanner is an always-on, globally consistent, multi-model database with virtually unlimited scale. By designating Spanner as a choice within Fury, Mercado Libre ensures that applications built on the platform using Spanner  inherit its best features – they stay consistent globally, scale without breaking, and rarely go down.

Spanner – the versatile backbone

Through Fury, Spanner empowers Mercado Libre’s developers with remarkable versatility. Some apps need complex transactions, others need fast lookups. Spanner handles both, which means teams can use just one system:

1. Relational prowess for complex transactions: For sophisticated transactional workloads like order management, payments, and inventory systems, Spanner’s relational capabilities (SQL, ACID transactions, joins) remain critical. 

2. High-performance key-value store: Many modern applications require fast point lookups and simple data structures. While Spanner isn’t Mercado Libre’s default backend for typical key-value workloads, there are specific applications running large scale non-relational, KV-style workloads on the Spanner.

Spanner’s foundational architecture — TrueTime for global consistency and automated sharding for effortless scaling — makes it an ideal candidate to reliably serve both these access patterns through the Fury platform.

Handling peak demand

Mercado Libre’s Spanner instances demonstrate significant processing capacity, handling around 214K queries per second (QPS) and 30K transactions per second (TPS). To manage this substantial workload, the Spanner infrastructure dynamically scales to over 400 nodes (by 30%), highlighting the robust and elastic nature of the underlying system in accommodating high-demand scenarios. This level of throughput and scalability is critical for maintaining the performance and reliability of Mercado Libre’s services during its busiest times.

Turning data into action

Mercado Libre builds a dynamic data ecosystem around Spanner, leveraging advanced analytics to feed insights directly back into operational systems.

They achieve real-time analytics by combining Spanner Data Boost with BigQuery Federation. Data Boost isolates analytical queries, preventing them from impacting critical transactional performance. This allows for powerful, large-scale analytics to run directly on fresh Spanner data within BigQuery, integrating seamlessly with other data sources.

Insights from BigQuery, such as customer segmentations or fraud scores, are then actioned via Reverse ETL, feeding directly back into Spanner. This enriches operational data, enabling immediate action by frontline applications like serving personalized content or performing real-time risk assessments.

Furthermore, Spanner Change Streams coupled with Dataflow drive crucial service integrations. By capturing real-time data modifications from Spanner, they establish robust pipelines. These enable loading changes into BigQuery for analytics or streaming them to services like Fury Stream for real-time consumption, ensuring low-latency data propagation and enabling event-driven architectures across their systems.

The impact: cost savings, agility, and future-proofing

The strategic adoption of Spanner, amplified by internal platforms like Fury and sophisticated data workflows, has yielded significant benefits for Mercado Libre:

• Significant cost savings & low total cost of ownership: The combination of Spanner’s managed nature (reducing manual sharding, maintenance, and maintenance work), efficient resource utilization, and the abstraction provided by Fury has led to a lower Total Cost of Ownership and substantial cost savings.

• Business impact & agility: Developers, freed from infrastructure complexities by Fury and empowered by Spanner’s versatile capabilities, can deliver new features and applications faster. The reliability of Spanner underpins critical business operations, minimizing disruptions.

• Low operational overhead: Automated scaling, sharding, and maintenance in Spanner significantly reduce the human effort required to manage large-scale database infrastructure.

Building for AI:  Next-generation applications on Spanner

Looking ahead, Mercado Libre is exploring Spanner to support more AI workloads.

Spanner’s characteristics make it an ideal foundation:

• Consistent state management: Critical for AI systems that need to maintain and reliably update their state context.

• Scalable memory/knowledge store: Ability to store and retrieve vast amounts of data for AI system memory, logs, and contextual information.

• Transactional operations: Enabling AI systems to perform reliable actions that interact with other systems.

• Integration with analytics & Machine Learning (ML): The existing data loops and ML.PREDICT capabilities can enrich AI systems with real-time insights and intelligence.

Spanner provides the transactional foundation  these sophisticated, AI applications will require.

Conclusion: A Unified, Intelligent Data Foundation

Mercado Libre’s adoption of Spanner demonstrates how to use a powerful, globally consistent database not just for its core capabilities, but as a strategic enabler for developer productivity, operational efficiency, advanced analytics, and future AI ambitions. Through their Fury platform, they’ve simplified access to Spanner’s capabilities, allowing it to serve as a flexible foundation for both relational and non-relational needs. The integration with BigQuery via Data Boost demonstrates a comprehensive approach to building an intelligent, data-driven enterprise. As Mercado Libre builds AI applications, Spanner is set to continue its role as the consistent and scalable foundation for their next wave of innovation.

Learn more

• Discover how Spanner can transform your business.

• Get started on Spanner today with a 90 day free trial instance.