Protocol | Enterprise

How Amazon’s S3 jumpstarted the cloud revolution

Amazon's first real web service brought us everything from Pinterest to coronavirus vaccines. Fifteen years later, insiders tell Protocol how it grew to store more than 100 trillion objects.

The Amazon Spheres are part of Amazon's Seattle headquarters.

The Spheres at Amazon headquarters are an architectural instantiation of the cloud.

Photo: MissMushroom/Unsplash

In late 2005, Don Alvarez was just another software entrepreneur struggling to get a new business off the ground when a friend working at Amazon invited him to check out a secret project that would change the world.

Alvarez's startup, FilmmakerLive, was designing online collaboration applications for creative professionals and faced a common problem for that time: storage. Tech startups were just starting to claw their way back from the excesses of the dot-com era, and buying expensive hardware was a risky bet for a startup. Buy too little and your site crashes. Buy too much and you go broke. For the chaotic life of a startup, that was a risky bet.

He was skeptical about what he could learn about movie collaboration from an ecommerce company, but took the friend up on his offer.

"Rudy Valdez blew my mind," Alvarez told Protocol. Valdez was then the head of business development for AWS, which at that time offered only a handful of basic services. He gave Alvarez, now director of engineering for Mural, a taste of Amazon's first and arguably most fundamental product: S3, a cloud-based object storage service.

S3, or Simple Storage Service, made its debut 15 years ago this weekend. It would be years before "the cloud" became one of the most disruptive forces in the history of enterprise computing. Amazon didn't even use the term when it announced S3 on March 14, 2006. But the storage service's launch instantly solved some very tricky problems for entrepreneurs like Alvarez, and would come to change the way all businesses thought about buying information technology.

Startups like Pinterest, Airbnb and and Stripe flocked to AWS in the coming years, and older companies like Netflix — then a DVD-mailing operation — also took the plunge to retool their operations for the internet.

"Amazon was putting infinite disk space in the hands of every startup at an incredibly low and pay-for-what-you-need price point, there was nothing like that," Alvarez said. "The second piece was that their API was so simple that i could just pick it up and build something useful in it, in the first 24 hours of using an unreleased, unannounced product."

S3 is now a vital cog in the AWS machine, which generated more than $45 billion in revenue last year. It has evolved in many different directions over the last 15 years, yet has kept a set of design principles drawn up by a team led by Allan Vermeulen, Amazon's chief technical officer during the earliest days of AWS, at the heart of its strategy.

"We knew what [customers] wanted to do then," Mai-Lan Tomsen Bukovec, vice president for AWS Storage and the current head of S3, told Protocol. "But we also knew that applications would evolve, because our customers are incredibly innovative, and what they're doing out there in all the different industries is going to change every year."

Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage. Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage.Photo: Amazon Web Services

Building for flexibility

"When people think bigger and faster in computers, they think of this," said Vermeulen during an interview in 2014, drawing a line in the air up and to the right. But storage technology has evolved differently, he said, over a period of long plateaus followed by sharp increases in capabilities: "It's the difference between driving my Tesla and flying my airplane."

S3 was one of those sharp breaks from the status quo. It was a godsend for developers like Alvarez, who no longer had to worry about buying and maintaining pricey storage hardware just to do business.

"There was nothing that we had access to that provided anything remotely like what S3 could do," Alvarez said. "I felt like somebody had just given me the keys to the candy store."

Like much of AWS, S3 was born from Amazon's experience building and scaling Amazon.com, which taught it a lot of hard lessons about the limits and possibilities of distributed computing.

"A forcing function for the design was that a single Amazon S3 distributed system must support the needs of both internal Amazon applications and external developers of any application. This means that it must be fast and reliable enough to run Amazon.com's websites, while flexible enough that any developer can use it for any data storage need," AWS said in the original launch press release for S3 in 2006.

In the early days of the cloud, performance and reliability were a huge concern. And those concerns were especially fraught when it came to data, which even 15 years ago was understood to be one of the most important assets in a company's arsenal.

"When we launched S3 15 years ago, S3 had eight microservices, and we have well over 300 now." Tomsen Bukovec said, referring to the then-novel software development practice of breaking up large chunks of interdependent code into smaller, independent services.

Building around microservices allowed AWS to decentralize points of failure for S3 while also creating a system designed to acknowledge that distributed cloud services will fail on occasion, and that such failures shouldn't take the entire system down.

It also allowed the company to layer on future enhancements without having to disturb the core pieces of the system: AWS now claims that S3 offers "11 9s" of reliability, or an astonishing 99.999999999% uptime that exceeds self-managed storage equipment by a large margin. (Other cloud storage vendors have matched this standard.)

S3 began life as a holding pen for simple web elements like images and video that website operators would pull down from AWS to your browser when you loaded a page. Over time, as companies became more comfortable with cloud storage, they started putting all kinds of data in S3.

And that's when things started to get a little messy.

Amazon Web Services's booth at the Microsoft PDC event in Los Angeles in 2008.Photo: D. Begley/Flickr

Plugging leaky buckets

If you look back at any number of security incidents over the past several years, a large number of them can be attributed to "leaky buckets," referring to the core unit of S3 storage. These incidents happen to other cloud providers as well, but given AWS's market share it's a problem the company has had to deal with on many, many occasions.

AWS operates under a "shared responsibility" model for security: AWS will prevent anyone from physically accessing its servers or infiltrating its network, but customers are expected to protect their accounts to a reasonable extent. In other words, you can't blame the rental car company if someone steals your laptop from the back seat of an unlocked vehicle.

Yet time and time again, cloud customers have left sensitive data belonging to their own customers in unprotected storage buckets open to anyone who can find them, which is easier than you might think. It's just one example of how AWS has had to evolve some of its core products to meet customers where they are, especially later-arriving customers accustomed to accessing everything they need from private, internal networks.

"In a business application world, you don't need to have access outside the company, or really outside a group of users within the business," Tomsen Bukovec said. But it was clear that AWS needed to do more to help its customers help themselves, which led to the development of tools like Block Public Access that could lock down all storage buckets associated with a corporate account.

It was also clear to outsiders in the fast-growth early days of AWS that Amazon's famous "two-pizza teams" were "both a strength and a weakness," Alvarez said.

"It enabled every one of those services to rocket forward at a speed none of those competitors could match. And in the early days, it meant there was a lot less consistency [and] that was hard to puzzle through and manage," he said, noting that the experience has improved over time.

Additional security tools have followed that let customers scan their accounts for unauthorized access from the public internet, or assign different levels of access to people with different roles within a company.

"Where we're seeing customers go with their migrations is that they often have hundreds of buckets and lots and lots of [different] roles," Tomsen Bukovec said of the newcomers to the cloud who seem most prone to these mistakes. "When we think about what to build to help customers secure the perimeter of their AWS resource, we think about how they would like to audit and how they would like to control" access to their storage resources inside S3.


Hospitalman Cierrajaye Santella, assigned to Naval Hospital Bremerton and Navy Medicine Readiness and Training Command Bremerton, prepares to administer the Moderna coronavirus vaccine. Moderna used AWS in the COVID-19 vaccine's development.Photo: U.S. Navy

Getting to 100 trillion

S3 continued to evolve in the years following its debut, and it also got a lot cheaper: By the time AWS got around to having its first major re:Invent developer conference in 2012, one of the major announcements from that week was a 24% to 28% percent reduction in S3 storage prices, the 24th such price cut the company had made up to that point.

Those price cuts were possible because AWS was able to upgrade the underlying S3 service on the fly, as Alyssa Henry, then vice president of AWS Storage Services, explained during a keynote address in 2012.

S3 was originally designed to hold 20 billion objects in storage, but it grew more quickly than anyone had anticipated, hitting 9 billion objects within the first year. The company upgraded the underlying storage service with more capacity in mind without any disruption to the original S3 customers, and By 2012 it had scaled to 1 trillion objects in storage, and by 2020, 100 trillion.

"What's really cool about this is customers didn't have to do anything: You didn't have to go out buy the next upgrade — v2 of Amazon S3; you didn't have to do the migration yourself; you just got it all for free, it just worked, things just got better," Henry, who is now executive vice president and head of Square's Seller unit, said at the 2012 event. "That's one of the differences with the cloud versus how traditional IT has been done."

A similar upgrade rolled out just last year, when AWS introduced strong consistency across S3.

Consistency is a data-storage concept that can rattle your brain a bit the first time it shows up; older storage systems such as the original S3 were designed around "eventual consistency," meaning that a storage service wouldn't always be able to tell you right away if a new piece of data had settled into its designated storage bucket, but it would catch up before long.

Now that modern applications move much faster, however, anything that makes a query to a storage service really needs to know the exact, current list of available data to perform at the expected level. So over the last couple of years, AWS rebuilt S3 around strong consistency principles, which other cloud providers offer but were able to roll out against a much smaller user base.

"That is a very complicated engineering problem," Tomsen Bukovec said, and it was one of the stand-out announcements from the re:Invent 2020 among the geekier set of AWS users.

As they head into a new decade, Tomsen Bukovec and her team are looking at ways to make it easier to do machine learning on top of S3 data, and to improve the performance and capabilities of data lakes that allow for fine-grained analysis of internal and customer data among AWS users.

In fact, the Moderna vaccine for COVID-19 was developed with the help of a S3 data lake, Tomsen Bukovec said.

"We have this unique view that we built up over 15 years of usage, where we can determine what our customers are trying to do, and how we can build [S3] in such a way that it keeps true to that simple, cost-effective, secure, durable, reliable and highly-performant storage," she said.

Theranos’ investor pitches go on trial

Prosecutors in the Elizabeth Holmes fraud case are now highlighting allegations the company sought to mislead investors.

The fresh details of unproven claims made about the viability of Theranos' blood tests and efforts to conceal errors when demonstrating testing equipment added to the evidence against Holmes, who is accused of fraud in her role leading the company.

Photo: David Paul Morris/Bloomberg via Getty Images

The Theranos trial continued this week with testimony from Daniel Edlin, a former product manager at the blood-testing startup, and Shane Weber, a scientist from Pfizer. Their testimonies appeared to bolster the government's argument that Holmes intentionally defrauded investors and patients.

The fresh details about audacious and unproven claims made about the viability of Theranos' blood tests and efforts to conceal errors when demonstrating testing equipment added to the evidence against Holmes, who is accused of fraud in her role leading the company.

Keep Reading Show less
Aisha Counts

Aisha Counts (@aishacounts) is a reporting fellow at Protocol, based out of Los Angeles. Previously, she worked for Ernst & Young, where she researched and wrote about the future of work, emerging technologies and startups. She is a graduate of the University of Southern California, where she studied business and philosophy. She can be reached at acounts@protocol.com.

The way we work has fundamentally changed. COVID-19 upended business dealings and office work processes, putting into hyperdrive a move towards digital collaboration platforms that allow teams to streamline processes and communicate from anywhere. According to the International Data Corporation, the revenue for worldwide collaboration applications increased 32.9 percent from 2019 to 2020, reaching $22.6 billion; it's expected to become a $50.7 billion industry by 2025.

"While consumers and early adopter businesses had widely embraced collaborative applications prior to the pandemic, the market saw five years' worth of new users in the first six months of 2020," said Wayne Kurtzman, research director of social and collaboration at IDC. "This has cemented collaboration, at least to some extent, for every business, large and small."

Keep Reading Show less
Kate Silver

Kate Silver is an award-winning reporter and editor with 15-plus years of journalism experience. Based in Chicago, she specializes in feature and business reporting. Kate's reporting has appeared in the Washington Post, The Chicago Tribune, The Atlantic's CityLab, Atlas Obscura, The Telegraph and many other outlets.

Protocol | Policy

8 takeaways from states’ new filing against Google

New details have been unsealed in the states' antitrust suit against Google for anticompetitive behavior in the ads market.

Google is facing complaints by government competition enforcers on several fronts.

Photo: Drew Angerer/Getty Images

Up to 22%: That's the fee Google charges publishers for sales on its online ad exchanges, according to newly unredacted details in a complaint by several state attorneys general.

The figure is just one of the many details that a court allowed the states to unveil Friday. Many had more or less remained secrets inside Google and the online publishing industry, even through prior legal complaints and eager public interest.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

Protocol | Workplace

This tech founder uses a converted Sprinter van as an office on wheels

The CEO of productivity startup Rock likes to work on the road. Here's how he does it — starting with three different WiFi hotspots.

Kenzo Fong, founder and CEO of the 20-person productivity software startup Rock, has been working out of his converted Mercedes-Benz Sprinter van since the pandemic began.

Photo: Kenzo Fong/Rock

Plenty of techies have started companies in garages. Try running a startup from a van.

In San Francisco, one software company founder has been using a converted Mercedes-Benz Sprinter van — picture an Amazon delivery vehicle — as a mobile office.

Keep Reading Show less
Allison Levitsky
Allison Levitsky is a reporter at Protocol covering workplace issues in tech. She previously covered big tech companies and the tech workforce for the Silicon Valley Business Journal. Allison grew up in the Bay Area and graduated from UC Berkeley.
Protocol | Policy

Most Americans want AI regulation — and they want it yesterday

In a poll, people said they wanted to see artificial intelligence technologies develop in the U.S. — alongside rules governing their use.

U.S. lawmakers have only just begun the long process of regulating the use of AI.

Photo: Louis Velazquez/Unsplash

Nearly two-thirds of Americans want the U.S to regulate the development and use of artificial intelligence in the next year or sooner — with half saying that regulation should have begun yesterday, according to a Morning Consult poll. Another 13% say that regulation should start in the next year.

"You can thread this together," Austin Carson, founder of new nonprofit group SeedAI and former government relations lead for Nvidia, said in an email. "Half or more Americans want to address all of these things, split pretty evenly along ideological lines."

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

ai
Latest Stories