Amazon S3 is often referred to as the "backbone" of the internet, but few understand the sheer scale of its operations. Originally launched on March 14, 2006, with a simple premise of providing "infinite" storage, the service has grown to manage over 500 trillion objects and hundreds of exabytes of data. To visualize this scale, if you stacked all of S3's hard drives on top of each other, they would reach the International Space Station and back. What started as a rudimentary architecture of six microservices has evolved into a complex ecosystem of over 300 highly decoupled microservices, a process engineers describe as "rebuilding the plane in flight".

At the core of S3's value proposition is its incredible durability guarantee of 99.999999999% (eleven nines). Achieving this without the prohibitive cost of tripling storage hardware required a shift from simple replication to Erasure Coding (EC). S3 utilizes algorithms based on Reed-Solomon codes, specifically a 5+4 design where data is split into fragments and parity shards. This allows the system to tolerate the simultaneous loss of an entire Availability Zone and an additional node in another zone without losing data, maintaining resilience against correlated failures.

A pivotal moment in S3's history occurred in December 2020, when the service migrated from an eventual consistency model to strong consistency for all requests. Previously, the system prioritized availability, meaning a listing of objects might not immediately reflect a recent write. The engineering team solved this by implementing a new replicated journal data structure and a cache coherency protocol with failure allowances. Remarkably, this massive architectural upgrade was rolled out with no downtime and at no additional cost to customers, despite the increased infrastructure requirements.

To handle the massive throughput of millions of requests per second, S3 employs intelligent horizontal scaling. The system partitions the index based on object key prefixes. Initially, keys reside in a single partition, but as traffic grows, S3 automatically splits these partitions to support higher throughput—up to 3,500 write or 5,500 read requests per second per prefix. This "evolvability" allows S3 to treat scale as an advantage, where performance actually improves as the system grows larger and workloads become more decorrelated.

Under the hood, the physical storage layer has also undergone a transformation. While historically built in C++, new storage nodes utilize "ShardStore," a storage engine written in Rust to ensure memory safety and high concurrent performance. ShardStore manages data in units called "extents" and uses Log-Structured Merge (LSM) trees to map shard identifiers to physical locations. To guarantee correctness at this scale, AWS employs "lightweight formal methods," using mathematical models to verify code logic and prevent concurrency bugs before they ever reach production.

In 2024, S3 evolved beyond simple object storage into an active data platform for AI and analytics. The introduction of Amazon S3 Tables provides native support for the Apache Iceberg format, automating maintenance tasks like compaction. Furthermore, S3 launched support for Vectors, a new primitive designed to store semantic embeddings for AI applications, allowing for high-dimensional similarity searches directly within the storage layer.

Security in S3 is not just about permissions but about mathematical certainty. The service uses automated reasoning tools like Zelkova to translate IAM and bucket policies into logical formulas, verifying that access controls behave exactly as intended. This approach has been extended to ensure that generative AI applications built on AWS do not violate safety policies, treating security as a provable correctness problem rather than just a configuration setting.

Ultimately, S3 is described by its leadership as a "living, breathing organism" that continues to change shape based on how the world uses data. From its origins as a simple "bag of bits" to a sophisticated system supporting SQL queries and vector search, S3 demonstrates that in hyperscale distributed systems, the simplicity of the user interface hides one of the most complex and robust engineering feats in history.

Sources

• Amazon S3 - Wikipedia & AWS re:Invent 2024 announcements
• From Bits to Tables: The Evolution of S3 Storage
• S3-like Object Storage - Book Notes
• The History of Amazon S3 - CloudSee Drive
• AWS re:Invent 2024 announcements and keynote updates
• AWS re:Invent 2024 - The Candid Startup
• How Amazon S3 Stores 100 Trillion Objects Without Losing One
• Our Origins - AWS & Erasure Coding sources
• Optimizing Performance Of Amazon S3
• Amazon S3 Update – Strong Read-After-Write Consistency
• Import from Amazon S3 Vectors to OpenSearch Serverless
• ShardStore: A "new storage backend for Amazon S3" on 40k lines of Rust
• Cloud Storage Deep Dive (Part 2) & S3 Tables announcements
• How does Amazon S3 ensure data Durability and Availability?
• Erasure Coding — This is how cloud providers store multiple copies
• Understanding erasure coding
• What is Amazon S3? & Automated Reasoning sources
• Amazon S3 Strong Consistency
• Deep dive on Amazon S3 (PDF)
• Amazon S3 Metadata now supports metadata for all your S3 objects
• Millions of Tiny Databases - USENIX
• How automated reasoning helps Amazon S3 innovate at scale
• Erasure Coding | System Design - AlgoMaster.io
• Erasure Coding Chapter - AlgoMaster.io
• How Amazon S3 Stores 100 Trillion Objects Without Losing One (Ajit Singh)
• Request Routing and Performance (Partition Keys)
• Better Key Design & Partitioning
• How S3 is built - Transcript (Milan, VP of Data & Analytics at AWS)
• Stack of drives to ISS (Transcript)
• S3 Origins & Eventual Consistency (Transcript)
• Eventual Consistency rationale (Transcript)
• S3 Basic Architecture (Put/Get) (Transcript)
• Replicated Journal & Strong Consistency implementation (Transcript)
• Failure Allowance & Cache Coherency (Transcript)
• No cost for Strong Consistency (Transcript)
• Automated Reasoning & Formal Methods (Transcript)
• Proofing correctness on check-ins (Transcript)
• Math saves you at scale (Transcript)
• Correlated Failures (Transcript)
• S3 Tables & SQL (Transcript)
• S3 Vectors & Embeddings (Transcript)
• Vectors as a new primitive (Transcript)
• Scale is to your advantage (Transcript)
• Product Shape & Coherency (Transcript)
• S3 as a living organism (Transcript)