Source Code: Your daily look at what matters in tech.

enterpriseprotocol | enterpriseauthorTom KrazitNoneAre you keeping up with the latest cloud developments? Get Tom Krazit and Joe Williams' newsletter every Monday and Thursday.d3d5b92349
×

Get access to Protocol

Will be used in accordance with our Privacy Policy

I’m already a subscriber
Protocol | Enterprise

How Amazon’s S3 jumpstarted the cloud revolution

Amazon's first real web service brought us everything from Pinterest to coronavirus vaccines. Fifteen years later, insiders tell Protocol how it grew to store more than 100 trillion objects.

The Amazon Spheres are part of Amazon's Seattle headquarters.

The Spheres at Amazon headquarters are an architectural instantiation of the cloud.

Photo: MissMushroom/Unsplash

In late 2005, Don Alvarez was just another software entrepreneur struggling to get a new business off the ground when a friend working at Amazon invited him to check out a secret project that would change the world.

Alvarez's startup, FilmmakerLive, was designing online collaboration applications for creative professionals and faced a common problem for that time: storage. Tech startups were just starting to claw their way back from the excesses of the dot-com era, and buying expensive hardware was a risky bet for a startup. Buy too little and your site crashes. Buy too much and you go broke. For the chaotic life of a startup, that was a risky bet.

He was skeptical about what he could learn about movie collaboration from an ecommerce company, but took the friend up on his offer.

"Rudy Valdez blew my mind," Alvarez told Protocol. Valdez was then the head of business development for AWS, which at that time offered only a handful of basic services. He gave Alvarez, now director of engineering for Mural, a taste of Amazon's first and arguably most fundamental product: S3, a cloud-based object storage service.

S3, or Simple Storage Service, made its debut 15 years ago this weekend. It would be years before "the cloud" became one of the most disruptive forces in the history of enterprise computing. Amazon didn't even use the term when it announced S3 on March 14, 2006. But the storage service's launch instantly solved some very tricky problems for entrepreneurs like Alvarez, and would come to change the way all businesses thought about buying information technology.

Startups like Pinterest, Airbnb and and Stripe flocked to AWS in the coming years, and older companies like Netflix — then a DVD-mailing operation — also took the plunge to retool their operations for the internet.

"Amazon was putting infinite disk space in the hands of every startup at an incredibly low and pay-for-what-you-need price point, there was nothing like that," Alvarez said. "The second piece was that their API was so simple that i could just pick it up and build something useful in it, in the first 24 hours of using an unreleased, unannounced product."

S3 is now a vital cog in the AWS machine, which generated more than $45 billion in revenue last year. It has evolved in many different directions over the last 15 years, yet has kept a set of design principles drawn up by a team led by Allan Vermeulen, Amazon's chief technical officer during the earliest days of AWS, at the heart of its strategy.

"We knew what [customers] wanted to do then," Mai-Lan Tomsen Bukovec, vice president for AWS Storage and the current head of S3, told Protocol. "But we also knew that applications would evolve, because our customers are incredibly innovative, and what they're doing out there in all the different industries is going to change every year."

Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage. Mai-Lan Tomsen Bukovec runs Amazon S3 and AWS Storage.Photo: Amazon Web Services

Building for flexibility

"When people think bigger and faster in computers, they think of this," said Vermeulen during an interview in 2014, drawing a line in the air up and to the right. But storage technology has evolved differently, he said, over a period of long plateaus followed by sharp increases in capabilities: "It's the difference between driving my Tesla and flying my airplane."

S3 was one of those sharp breaks from the status quo. It was a godsend for developers like Alvarez, who no longer had to worry about buying and maintaining pricey storage hardware just to do business.

"There was nothing that we had access to that provided anything remotely like what S3 could do," Alvarez said. "I felt like somebody had just given me the keys to the candy store."

Like much of AWS, S3 was born from Amazon's experience building and scaling Amazon.com, which taught it a lot of hard lessons about the limits and possibilities of distributed computing.

"A forcing function for the design was that a single Amazon S3 distributed system must support the needs of both internal Amazon applications and external developers of any application. This means that it must be fast and reliable enough to run Amazon.com's websites, while flexible enough that any developer can use it for any data storage need," AWS said in the original launch press release for S3 in 2006.

In the early days of the cloud, performance and reliability were a huge concern. And those concerns were especially fraught when it came to data, which even 15 years ago was understood to be one of the most important assets in a company's arsenal.

"When we launched S3 15 years ago, S3 had eight microservices, and we have well over 300 now." Tomsen Bukovec said, referring to the then-novel software development practice of breaking up large chunks of interdependent code into smaller, independent services.

Building around microservices allowed AWS to decentralize points of failure for S3 while also creating a system designed to acknowledge that distributed cloud services will fail on occasion, and that such failures shouldn't take the entire system down.

It also allowed the company to layer on future enhancements without having to disturb the core pieces of the system: AWS now claims that S3 offers "11 9s" of reliability, or an astonishing 99.999999999% uptime that exceeds self-managed storage equipment by a large margin. (Other cloud storage vendors have matched this standard.)

S3 began life as a holding pen for simple web elements like images and video that website operators would pull down from AWS to your browser when you loaded a page. Over time, as companies became more comfortable with cloud storage, they started putting all kinds of data in S3.

And that's when things started to get a little messy.

Amazon Web Services's booth at the Microsoft PDC event in Los Angeles in 2008.Photo: D. Begley/Flickr

Plugging leaky buckets

If you look back at any number of security incidents over the past several years, a large number of them can be attributed to "leaky buckets," referring to the core unit of S3 storage. These incidents happen to other cloud providers as well, but given AWS's market share it's a problem the company has had to deal with on many, many occasions.

AWS operates under a "shared responsibility" model for security: AWS will prevent anyone from physically accessing its servers or infiltrating its network, but customers are expected to protect their accounts to a reasonable extent. In other words, you can't blame the rental car company if someone steals your laptop from the back seat of an unlocked vehicle.

Yet time and time again, cloud customers have left sensitive data belonging to their own customers in unprotected storage buckets open to anyone who can find them, which is easier than you might think. It's just one example of how AWS has had to evolve some of its core products to meet customers where they are, especially later-arriving customers accustomed to accessing everything they need from private, internal networks.

"In a business application world, you don't need to have access outside the company, or really outside a group of users within the business," Tomsen Bukovec said. But it was clear that AWS needed to do more to help its customers help themselves, which led to the development of tools like Block Public Access that could lock down all storage buckets associated with a corporate account.

It was also clear to outsiders in the fast-growth early days of AWS that Amazon's famous "two-pizza teams" were "both a strength and a weakness," Alvarez said.

"It enabled every one of those services to rocket forward at a speed none of those competitors could match. And in the early days, it meant there was a lot less consistency [and] that was hard to puzzle through and manage," he said, noting that the experience has improved over time.

Additional security tools have followed that let customers scan their accounts for unauthorized access from the public internet, or assign different levels of access to people with different roles within a company.

"Where we're seeing customers go with their migrations is that they often have hundreds of buckets and lots and lots of [different] roles," Tomsen Bukovec said of the newcomers to the cloud who seem most prone to these mistakes. "When we think about what to build to help customers secure the perimeter of their AWS resource, we think about how they would like to audit and how they would like to control" access to their storage resources inside S3.


Hospitalman Cierrajaye Santella, assigned to Naval Hospital Bremerton and Navy Medicine Readiness and Training Command Bremerton, prepares to administer the Moderna coronavirus vaccine. Moderna used AWS in the COVID-19 vaccine's development.Photo: U.S. Navy

Getting to 100 trillion

S3 continued to evolve in the years following its debut, and it also got a lot cheaper: By the time AWS got around to having its first major re:Invent developer conference in 2012, one of the major announcements from that week was a 24% to 28% percent reduction in S3 storage prices, the 24th such price cut the company had made up to that point.

Those price cuts were possible because AWS was able to upgrade the underlying S3 service on the fly, as Alyssa Henry, then vice president of AWS Storage Services, explained during a keynote address in 2012.

S3 was originally designed to hold 20 billion objects in storage, but it grew more quickly than anyone had anticipated, hitting 9 billion objects within the first year. The company upgraded the underlying storage service with more capacity in mind without any disruption to the original S3 customers, and By 2012 it had scaled to 1 trillion objects in storage, and by 2020, 100 trillion.

"What's really cool about this is customers didn't have to do anything: You didn't have to go out buy the next upgrade — v2 of Amazon S3; you didn't have to do the migration yourself; you just got it all for free, it just worked, things just got better," Henry, who is now executive vice president and head of Square's Seller unit, said at the 2012 event. "That's one of the differences with the cloud versus how traditional IT has been done."

A similar upgrade rolled out just last year, when AWS introduced strong consistency across S3.

Consistency is a data-storage concept that can rattle your brain a bit the first time it shows up; older storage systems such as the original S3 were designed around "eventual consistency," meaning that a storage service wouldn't always be able to tell you right away if a new piece of data had settled into its designated storage bucket, but it would catch up before long.

Now that modern applications move much faster, however, anything that makes a query to a storage service really needs to know the exact, current list of available data to perform at the expected level. So over the last couple of years, AWS rebuilt S3 around strong consistency principles, which other cloud providers offer but were able to roll out against a much smaller user base.

"That is a very complicated engineering problem," Tomsen Bukovec said, and it was one of the stand-out announcements from the re:Invent 2020 among the geekier set of AWS users.

As they head into a new decade, Tomsen Bukovec and her team are looking at ways to make it easier to do machine learning on top of S3 data, and to improve the performance and capabilities of data lakes that allow for fine-grained analysis of internal and customer data among AWS users.

In fact, the Moderna vaccine for COVID-19 was developed with the help of a S3 data lake, Tomsen Bukovec said.

"We have this unique view that we built up over 15 years of usage, where we can determine what our customers are trying to do, and how we can build [S3] in such a way that it keeps true to that simple, cost-effective, secure, durable, reliable and highly-performant storage," she said.

Protocol | Fintech

Marqeta turns to a fintech outsider

Randy Kern, a Salesforce and Microsoft veteran, is taking a plunge into the payments world.

Randy Kern is joining Marqeta after decades at Microsoft and Salesforce.

Photo: Marqeta

Marqeta has just named a new chief technology officer. And it's an eyebrow-raising choice for a critical post as the payments powerhouse faces new challenges as a public company.

Randy Kern, who joined Marqeta last month, is a tech veteran with decades of engineering and leadership experience, mainly in enterprise software. He worked on Microsoft's Azure and Bing technologies, and then went on to Salesforce where he last served as chief customer technology officer.

Keep Reading Show less
Benjamin Pimentel

Benjamin Pimentel ( @benpimentel) covers fintech from San Francisco. He has reported on many of the biggest tech stories over the past 20 years for the San Francisco Chronicle, Dow Jones MarketWatch and Business Insider, from the dot-com crash, the rise of cloud computing, social networking and AI to the impact of the Great Recession and the COVID crisis on Silicon Valley and beyond. He can be reached at bpimentel@protocol.com or via Signal at (510)731-8429.

As President of Alibaba Group, I am often asked, "What is Alibaba doing in the U.S.?"

In fact, most people are not aware we have a business in the U.S. because we are not a U.S. consumer-facing service that people use every day – nor do we want to be. Our consumers – nearly 900 million of them – are located in China.

Keep Reading Show less
J. Michael Evans
Michael Evans leads and executes Alibaba Group's international strategy for globalizing the company and expanding its businesses outside of China.
Protocol | Policy

What can’t Jonathan Kanter do?

Biden's nominee to lead the DOJ's antitrust section may face calls to remove himself from issues as weighty as cracking down on Google and Apple.

DOJ antitrust nominee Jonathan Kanter's work as a corporate lawyer may require him to recuse himself from certain cases.

Photo: New America/Flickr

Jonathan Kanter, President Joe Biden's nominee to run the Justice Department's antitrust division, has been a favorite of progressives, competitors to Big Tech companies and even some Republicans due to his longtime criticism of companies like Google.

But his prior work as a corporate lawyer going after tech giants may require him to recuse himself from some of the DOJ's marquee investigations and cases, including those involving Google and Apple.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

Protocol | Enterprise

Couchbase plots escape from middle of database pack with $200M IPO

The company has to prove it can beat larger rivals like MongoDB, as well as fast-growing competitors like Redis Labs, not to mention the big cloud companies.

Couchbase celebrates its initial public offering on the Nasdaq market.

Photo: Nasdaq

At first glance, Couchbase appears to be stuck in the middle of the cloud database market, flanked by competitors with more traction and buzz. But fresh off a $200 million IPO Thursday, CEO Matt Cain relished the opportunity ahead to prove why his company can beat out rivals the market considers more valuable.

The NoSQL database provider's public offering helped propel Couchbase to a $1.2 billion valuation. But unlike one of the last big data-related IPOs, market leader Snowflake's historic debut on the public markets last December, Couchbase has some work to do to differentiate itself.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

People

SPACs are so Q1 and other takeaways from a disorienting year in IPOs

Amid the frenzy of tech IPOs this year, a few surprising discoveries stand out.

Through it all, the house always wins.

Image: CSA Images/Getty Images

2021 is shaping up to be a disorienting year for tech IPOs. The first six months brought us the Alex Rodriguez SPAC, an $85 billion Coinbase debut and a mysterious delay in the Robinhood S-1 filing that was ultimately cleared up when the firm paid a token fine.

Amid the recurring frenzy, it's easy to slip into a familiar pattern of analysis: Wait for an S-1 to drop, react to the financial disclosures, then see whether the stock "pops" after its trading debut. By the time one stock starts trading, several tantalizing new S-1s are already up for inspection. The problem with this cycle is that it stops too early: A stock's opening-day pop only really reflects the extent to which a few overworked investment bankers underestimated investor demand. A pop makes for headlines. It doesn't make a company.

Keep Reading Show less
Hirsh Chitkara
Hirsh Chitkara (@ChitkaraHirsh) is a researcher at Protocol, based out of New York City. Before joining Protocol, he worked for Business Insider Intelligence, where he wrote about Big Tech, telecoms, workplace privacy, smart cities, and geopolitics. He also worked on the Strategy & Analytics team at the Cleveland Indians.
Latest Stories