What we talk about when we talk about outages

The Big Story

Good luck with DIY internet

When something goes wrong on the internet, it's clear that most people still don't understand what it takes to bring you the internet.

Fastly's 49-minute outage inspired some odd reporting on Tuesday, with a series of dark headlines and stories popping up in consumer-facing publications. It's not clear how many enterprise infrastructure companies one can fit in a hand, but the suggestion appears to be that it's a small number, and that that's a Bad Thing.

• "The outage has raised questions about relying on a handful of companies to run the vast infrastructure that underpins the internet," according to the BBC, which neither cited interlocutors nor pointed to any other sort of alternative architecture on which we can rely.
• Wired's story was headlined "How an Obscure Company Took Down Big Chunks of the Internet," seemingly defining "obscure" as "a company we don't know but one that all these Fortune 500 companies apparently use."
• To Wired, "the incident offers a stark reminder of how fragile and interconnected internet infrastructure can be, especially when so much of it hinges on a handful of companies that operate largely outside of public awareness."

There are far more than a "handful" of companies that keep the internet running nearly 24/7 around the world amid a huge surge in demand for their services; Fastly is actually a much smaller player in the CDN market compared to Akamai. And businesses can layer redundant options across multiple cloud service providers if they want to maximize uptime, which is difficult and expensive, but possible.

A very different series of reactions spread across Twitter from the people who actually built the modern internet and understand what it takes to keep it running, the people we profiled earlier this year in our oral history of the #hugops movement.

• This group was impressed at how quickly Fastly responded to a very weird bug, "a bug that could be triggered by a specific customer configuration under specific circumstances," Fastly wrote in its incident report.
• Nathaniel Gleicher, head of security policy at Facebook, probably summed up this reaction best: "All things being equal, a 49 min time-to-remediate is pretty quick for a sneaky little misconfig bug. Lots of stories about the breakdown, but bugs (and hacks) will always happen — nothing is perfect."

There's an incredibly important point here about modern internet infrastructure that always gets lost in the wake of a prominent outage.

• No business is forced to use third-party cloud service providers to reach their customers; if you don't want to rely on "a handful" of companies, Dell and HPE will be happy to sell you servers, storage and networking gear.
• But the last time that strategy was in vogue was known as the "dot-com era," and there will be no going back to the days when Myspace was considered a revolutionary web site.
• Even as companies start to consider once again rolling some of their own infrastructure, building and maintaining everything you need to operate on the internet is a capital-intensive undertaking that also requires expensive, specialized talent. Even Amazon is a Fastly customer.
• That's exactly why demand for cloud computing exploded over the last 15 years: It enabled a surge of innovation from startups and other companies that would have struggled mightily to build internet businesses against the capital requirements of reliable DIY infrastructure.

So what are we really talking about when we talk about outages?

• It's safe to say that right now, lots of people don't trust the big tech companies that have gleefully participated in what dot-com era investor John Doerr called "the greatest legal accumulation of wealth in history."
• And after a pandemic year during which we all depended on the internet more than ever, more people notice when things inevitably go wrong.
• It's more than fair to demand maximum uptime from service providers, but the internet will never be available 100 percent of the time.
• So what lies beneath the surface of these discussions is an uneasy sense of disconnection people feel when the internet goes down, which is a problem that can't be solved by manifesting six more CDN vendors or (gulp) building government-run internet infrastructure.

The internet will always have chokepoints, and software will always have bugs. But networks and software systems are dramatically more resilient than the day Doerr made his first tech investment.

• If you're really worried about the internet after a hiccup at Fastly prevented the world from consuming internet content for an entire 49 minutes — the rough length of one episode of "This Is Us" on a streaming service — you should look at the concentration of Tier 1 internet service providers.

— Tom Krazit

A MESSAGE FROM INTEL AND MICROSOFT AZURE

Trust, but verify. It's a good guideline for many things, but essential in data security. That need to confirm the reliability of data got easier recently when Microsoft rolled out Azure Confidential Ledger, which provides cryptographic evidence that blockchain ledgers have not been tampered with.

Learn more

This Week On Protocol

AI-as-a-service: The next generation of SaaS companies will need to work closely with data companies to truly unlock the potential of artificial intelligence in their services, Databricks CEO Ali Ghodsi told Protocol's Joe Williams during a virtual event Monday. "Wherever you have software, that software can become more intelligent," he said.

Apple's cloud: It was WWDC week for Apple developers, and while most of the attention was focused on a new version of iOS, Apple also unveiled an interesting new cloud service. Xcode Cloud will allow developers to build, test and deploy their applications using Apple infrastructure, one of the first major cloud services the company has released since embarking on a cloud-talent hiring spree in late 2019.

Five Questions For...

Tim Tully, Partner, Menlo Ventures

What was your first tech job?

My first tech job was at JavaSoft, a division of Sun Microsystems responsible for the Java programming language. I worked with the sales engineering team to develop application prototypes of new aspects of Java, such as JDBC. Over the years that spilled into other things like working on Java inside of the Mozilla project.

What was the first computer that got you excited about technology?

It was my Apple II. I played video games on that until I had blisters. The games were on 5.25 inch floppies and magnetic cassette tapes. I also had a key upgrade at the time: an RF modulator add-on that provided output to a small television I had on a cart.

If Protocol gave you $1 billion to start a new enterprise tech company from scratch today, what would you do?

I'd start a company to completely abstract away the major cloud service providers so that moving my application between them was 100% seamless. To a certain extent, that can happen with tech like containers and container orchestration, but there are many ancillary dependencies and services that software teams wind up wiring into the application that makes portability very, very difficult, such as identity management, load balancing and more.

What's your favorite pastime that doesn't involve a screen?

Snowboarding. I like being outside with the cold air, taking in the terrain, the travel associated with it, the gear and obviously charging down a mountain.

How can enterprise tech improve its current status around diversity, equity and inclusion?

I've seen empirically that it can be done by doing a few things. One, setting diversity hiring goals for organizations, which can have a real impact. Second, it's important to improve the processes in order to hit those goals. For example, at Splunk we had a "two in the pool" policy that required two people out of the hiring panel to have diverse backgrounds themselves (the NFL's Rooney Rule), or for there to be at least one diverse candidate in the candidate pool.

Around the Enterprise

• The multicloud era is in full swing, according to Flexera's latest cloud infrastructure report: 76% of respondents are using AWS, while 73% are using Microsoft Azure.
• Meanwhile, Microsoft ranks ahead of its rival within the partner community according to a CRN survey, which attributed the results to the maturity of Microsoft's partner programs.
• European regulators may treat AWS as a separate entity for tax purposes, given that the entire Amazon operation is far less profitable.
• Google Cloud is moving YouTube workloads over to its cloud servers, which might seem overdue but also underscores how tricky it can be to move infrastructure services that aren't broken.
• UiPath beat Wall Street expectations for its first quarter as a public company, although its stock fell in after-hours trading.
• Will virtual events be as popular post-pandemic? LinkedIn seems to think so: It's betting that Hopin's success with its virtual events platform will continue into 2022.
• A plot to bomb an AWS data center was hatched by a man who traveled from Texas to Washington, D.C. for the Jan. 6 insurrection attempt. Seth Aaron Pendley pleaded guilty to charges regarding his plans to blow up the data center in Northern Virginia.
• Google plans to build a new undersea cable running between the East Coast and Argentina, the 16th cable it has helped finance.
• The pandemic reprioritized cloud strategies for all kinds of organizations. But Nextgov reports that state and local governments are having different degrees of success with the transition.

A MESSAGE FROM INTEL AND MICROSOFT AZURE

Trust, but verify. It's a good guideline for many things, but essential in data security. That need to confirm the reliability of data got easier recently when Microsoft rolled out Azure Confidential Ledger, which provides cryptographic evidence that blockchain ledgers have not been tampered with.

Learn more