Enterprise

How Amazon’s S3 jumpstarted the cloud revolution

Amazon's first real web service brought us everything from Pinterest to coronavirus vaccines. Fifteen years later, insiders tell Protocol how it grew to store more than 100 trillion objects.

The Amazon Spheres are part of Amazon's Seattle headquarters.

The Spheres at Amazon headquarters are an architectural instantiation of the cloud.

Photo: MissMushroom/Unsplash

In late 2005, Don Alvarez was just another software entrepreneur struggling to get a new business off the ground when a friend working at Amazon invited him to check out a secret project that would change the world.

Alvarez's startup, FilmmakerLive, was designing online collaboration applications for creative professionals and faced a common problem for that time: storage. Tech startups were just starting to claw their way back from the excesses of the dot-com era, and buying expensive hardware was a risky bet for a startup. Buy too little and your site crashes. Buy too much and you go broke. For the chaotic life of a startup, that was a risky bet.

He was skeptical about what he could learn about movie collaboration from an ecommerce company, but took the friend up on his offer.

"Rudy Valdez blew my mind," Alvarez told Protocol. Valdez was then the head of business development for AWS, which at that time offered only a handful of basic services. He gave Alvarez, now director of engineering for Mural, a taste of Amazon's first and arguably most fundamental product: S3, a cloud-based object storage service.

S3, or Simple Storage Service, made its debut 15 years ago this weekend. It would be years before "the cloud" became one of the most disruptive forces in the history of enterprise computing. Amazon didn't even use the term when it announced S3 on March 14, 2006. But the storage service's launch instantly solved some very tricky problems for entrepreneurs like Alvarez, and would come to change the way all businesses thought about buying information technology.

Startups like Pinterest, Airbnb and and Stripe flocked to AWS in the coming years, and older companies like Netflix — then a DVD-mailing operation — also took the plunge to retool their operations for the internet.

"Amazon was putting infinite disk space in the hands of every startup at an incredibly low and pay-for-what-you-need price point, there was nothing like that," Alvarez said. "The second piece was that their API was so simple that i could just pick it up and build something useful in it, in the first 24 hours of using an unreleased, unannounced product."

S3 is now a vital cog in the AWS machine, which generated more than $45 billion in revenue last year. It has evolved in many different directions over the last 15 years, yet has kept a set of design principles drawn up by a team led by Allan Vermeulen, Amazon's chief technical officer during the earliest days of AWS, at the heart of its strategy.

"We knew what [customers] wanted to do then," Mai-Lan Tomsen Bukovec, vice president for AWS Storage and the current head of S3, told Protocol. "But we also knew that applications would evolve, because our customers are incredibly innovative, and what they're doing out there in all the different industries is going to change every year."

Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage. Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage.Photo: Amazon Web Services

Building for flexibility

"When people think bigger and faster in computers, they think of this," said Vermeulen during an interview in 2014, drawing a line in the air up and to the right. But storage technology has evolved differently, he said, over a period of long plateaus followed by sharp increases in capabilities: "It's the difference between driving my Tesla and flying my airplane."

S3 was one of those sharp breaks from the status quo. It was a godsend for developers like Alvarez, who no longer had to worry about buying and maintaining pricey storage hardware just to do business.

"There was nothing that we had access to that provided anything remotely like what S3 could do," Alvarez said. "I felt like somebody had just given me the keys to the candy store."

Like much of AWS, S3 was born from Amazon's experience building and scaling Amazon.com, which taught it a lot of hard lessons about the limits and possibilities of distributed computing.

"A forcing function for the design was that a single Amazon S3 distributed system must support the needs of both internal Amazon applications and external developers of any application. This means that it must be fast and reliable enough to run Amazon.com's websites, while flexible enough that any developer can use it for any data storage need," AWS said in the original launch press release for S3 in 2006.

In the early days of the cloud, performance and reliability were a huge concern. And those concerns were especially fraught when it came to data, which even 15 years ago was understood to be one of the most important assets in a company's arsenal.

"When we launched S3 15 years ago, S3 had eight microservices, and we have well over 300 now." Tomsen Bukovec said, referring to the then-novel software development practice of breaking up large chunks of interdependent code into smaller, independent services.

Building around microservices allowed AWS to decentralize points of failure for S3 while also creating a system designed to acknowledge that distributed cloud services will fail on occasion, and that such failures shouldn't take the entire system down.

It also allowed the company to layer on future enhancements without having to disturb the core pieces of the system: AWS now claims that S3 offers "11 9s" of reliability, or an astonishing 99.999999999% uptime that exceeds self-managed storage equipment by a large margin. (Other cloud storage vendors have matched this standard.)

S3 began life as a holding pen for simple web elements like images and video that website operators would pull down from AWS to your browser when you loaded a page. Over time, as companies became more comfortable with cloud storage, they started putting all kinds of data in S3.

And that's when things started to get a little messy.

Amazon Web Services's booth at the Microsoft PDC event in Los Angeles in 2008.Photo: D. Begley/Flickr

Plugging leaky buckets

If you look back at any number of security incidents over the past several years, a large number of them can be attributed to "leaky buckets," referring to the core unit of S3 storage. These incidents happen to other cloud providers as well, but given AWS's market share it's a problem the company has had to deal with on many, many occasions.

AWS operates under a "shared responsibility" model for security: AWS will prevent anyone from physically accessing its servers or infiltrating its network, but customers are expected to protect their accounts to a reasonable extent. In other words, you can't blame the rental car company if someone steals your laptop from the back seat of an unlocked vehicle.

Yet time and time again, cloud customers have left sensitive data belonging to their own customers in unprotected storage buckets open to anyone who can find them, which is easier than you might think. It's just one example of how AWS has had to evolve some of its core products to meet customers where they are, especially later-arriving customers accustomed to accessing everything they need from private, internal networks.

"In a business application world, you don't need to have access outside the company, or really outside a group of users within the business," Tomsen Bukovec said. But it was clear that AWS needed to do more to help its customers help themselves, which led to the development of tools like Block Public Access that could lock down all storage buckets associated with a corporate account.

It was also clear to outsiders in the fast-growth early days of AWS that Amazon's famous "two-pizza teams" were "both a strength and a weakness," Alvarez said.

"It enabled every one of those services to rocket forward at a speed none of those competitors could match. And in the early days, it meant there was a lot less consistency [and] that was hard to puzzle through and manage," he said, noting that the experience has improved over time.

Additional security tools have followed that let customers scan their accounts for unauthorized access from the public internet, or assign different levels of access to people with different roles within a company.

"Where we're seeing customers go with their migrations is that they often have hundreds of buckets and lots and lots of [different] roles," Tomsen Bukovec said of the newcomers to the cloud who seem most prone to these mistakes. "When we think about what to build to help customers secure the perimeter of their AWS resource, we think about how they would like to audit and how they would like to control" access to their storage resources inside S3.


Hospitalman Cierrajaye Santella, assigned to Naval Hospital Bremerton and Navy Medicine Readiness and Training Command Bremerton, prepares to administer the Moderna coronavirus vaccine. Moderna used AWS in the COVID-19 vaccine's development.Photo: U.S. Navy

Getting to 100 trillion

S3 continued to evolve in the years following its debut, and it also got a lot cheaper: By the time AWS got around to having its first major re:Invent developer conference in 2012, one of the major announcements from that week was a 24% to 28% percent reduction in S3 storage prices, the 24th such price cut the company had made up to that point.

Those price cuts were possible because AWS was able to upgrade the underlying S3 service on the fly, as Alyssa Henry, then vice president of AWS Storage Services, explained during a keynote address in 2012.

S3 was originally designed to hold 20 billion objects in storage, but it grew more quickly than anyone had anticipated, hitting 9 billion objects within the first year. The company upgraded the underlying storage service with more capacity in mind without any disruption to the original S3 customers, and By 2012 it had scaled to 1 trillion objects in storage, and by 2020, 100 trillion.

"What's really cool about this is customers didn't have to do anything: You didn't have to go out buy the next upgrade — v2 of Amazon S3; you didn't have to do the migration yourself; you just got it all for free, it just worked, things just got better," Henry, who is now executive vice president and head of Square's Seller unit, said at the 2012 event. "That's one of the differences with the cloud versus how traditional IT has been done."

A similar upgrade rolled out just last year, when AWS introduced strong consistency across S3.

Consistency is a data-storage concept that can rattle your brain a bit the first time it shows up; older storage systems such as the original S3 were designed around "eventual consistency," meaning that a storage service wouldn't always be able to tell you right away if a new piece of data had settled into its designated storage bucket, but it would catch up before long.

Now that modern applications move much faster, however, anything that makes a query to a storage service really needs to know the exact, current list of available data to perform at the expected level. So over the last couple of years, AWS rebuilt S3 around strong consistency principles, which other cloud providers offer but were able to roll out against a much smaller user base.

"That is a very complicated engineering problem," Tomsen Bukovec said, and it was one of the stand-out announcements from the re:Invent 2020 among the geekier set of AWS users.

As they head into a new decade, Tomsen Bukovec and her team are looking at ways to make it easier to do machine learning on top of S3 data, and to improve the performance and capabilities of data lakes that allow for fine-grained analysis of internal and customer data among AWS users.

In fact, the Moderna vaccine for COVID-19 was developed with the help of a S3 data lake, Tomsen Bukovec said.

"We have this unique view that we built up over 15 years of usage, where we can determine what our customers are trying to do, and how we can build [S3] in such a way that it keeps true to that simple, cost-effective, secure, durable, reliable and highly-performant storage," she said.

Podcasts

Should startups be scared?

Stock market turmoil is making VCs skittish. Could now be the best time to start a company?

yellow sticky notes on gray wall
Photo by Startaê Team on Unsplash

This week, we break down why Elon Musk is tweeting about the S&P 500's ESG rankings — and why he might be right to be mad. Then we discuss how tech companies are failing to prevent mass shootings, and why the new Texas social media law might make it more difficult for platforms to be proactive.

Then Protocol's Biz Carson, author of the weekly VC newsletter Pipeline, joins us to explain the state of venture capital amidst plunging stocks and declining revenues. Should founders start panicking? The answer might surprise you.

Keep Reading Show less
Caitlin McGarry

Caitlin McGarry is the news editor at Protocol.

Sponsored Content

Why the digital transformation of industries is creating a more sustainable future

Qualcomm’s chief sustainability officer Angela Baker on how companies can view going “digital” as a way not only toward growth, as laid out in a recent report, but also toward establishing and meeting environmental, social and governance goals.

Three letters dominate business practice at present: ESG, or environmental, social and governance goals. The number of mentions of the environment in financial earnings has doubled in the last five years, according to GlobalData: 600,000 companies mentioned the term in their annual or quarterly results last year.

But meeting those ESG goals can be a challenge — one that businesses can’t and shouldn’t take lightly. Ahead of an exclusive fireside chat at Davos, Angela Baker, chief sustainability officer at Qualcomm, sat down with Protocol to speak about how best to achieve those targets and how Qualcomm thinks about its own sustainability strategy, net zero commitment, other ESG targets and more.

Keep Reading Show less
Chris Stokel-Walker

Chris Stokel-Walker is a freelance technology and culture journalist and author of "YouTubers: How YouTube Shook Up TV and Created a New Generation of Stars." His work has been published in The New York Times, The Guardian and Wired.

Inside the Crypto Cannabis Club

As crypto crashes, an NFT weed club holds on to the high.

The Crypto Cannabis Club’s Discord has 23,000 subscribers, with 28 chapters globally.

Photo: Nat Rubio-Licht/Protocol

On a Saturday night in downtown Los Angeles, a group of high strangers gathered in a smoky, colorful venue less than a mile from Crypto.com Arena. The vibe was relaxed but excited, and the partygoers, many of whom were meeting each other for the very first time, greeted each other like old friends, calling each other by their Discord names. The mood was celebratory: The Crypto Cannabis Club, an NFT community for stoners, was gathering to celebrate the launch of its metaverse dispensary.

The warmth and belonging of the weed-filled party was a contrast to the metaverse store, which was underwhelming by comparison. But the dispensary launch and the NFTs required to buy into the group are just an excuse: As with most Web3 projects, it’s really about the community. Even though crypto is crashing, taking NFTs with it, the Crypto Cannabis Club is unphased, CEO Ryan Hunter told Protocol.

Keep Reading Show less
Nat Rubio-Licht

Nat Rubio-Licht is a Los Angeles-based news writer at Protocol. They graduated from Syracuse University with a degree in newspaper and online journalism in May 2020. Prior to joining the team, they worked at the Los Angeles Business Journal as a technology and aerospace reporter.

Fintech

Privacy and harassment could spoil Grindr’s Wall Street romance

As it pursues a long-held goal of going public, the gay dating app has to confront its demons.

Grindr may finally be a public company.

Illustration: woocat/iStock/Getty Images Plus; Protocol

Grindr's looking for more than just a hookup with Wall Street. Finding a stable relationship may be tough.

The location-based dating app favored by gay men was a pioneer, predating Tinder by three years. It’s bounced from owner to owner after founder Joel Simkhai sold it in 2018 for $245 million. A SPAC merger could be the answer, but businesses serving the LGBTQ+ community have had trouble courting investors. And Grindr has its own unique set of challenges.

Keep Reading Show less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol, covering breaking news. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

Climate

The minerals we need to save the planet are getting way too expensive

Supply chain problems and rising demand have sent prices spiraling upward for the minerals and metals essential for the clean energy transition.

Critical mineral prices have exploded over the past year.

Photo: Andrey Rudakov/Bloomberg via Getty Images

The newest source of the alarm bells echoing throughout the renewables industry? Spiking critical mineral and metal prices.

According to a new report from the International Energy Agency, a maelstrom of rising demand and tattered supply chains have caused prices for the materials needed for clean energy technologies to soar in the last year. And this increase has only accelerated since 2022 began.

Keep Reading Show less
Lisa Martine Jenkins

Lisa Martine Jenkins is a senior reporter at Protocol covering climate. Lisa previously wrote for Morning Consult, Chemical Watch and the Associated Press. Lisa is currently based in Brooklyn, and is originally from the Bay Area. Find her on Twitter ( @l_m_j_) or reach out via email (ljenkins@protocol.com).

Latest Stories
Bulletins