The Big Story

Lights out in Menlo Park

Facebook more or less disappeared from the internet for a good chunk of Monday. It's not just that its website went down, it's that the company seemed for hours to have been deleted entirely. It was so bad that the domain facebook.com appeared to be for sale. (Jack Dorsey publicly inquired about the price.)

• Employees said they couldn't badge into Facebook offices or access their email.They turned instead to Discord, Zoom and other platforms to keep in touch. And users who couldn't get on Facebook flocked elsewhere; Twitter and Signal both reported huge spikes in usage yesterday.
• "*Sincere* apologies to everyone impacted by outages of Facebook powered services right now," Facebook CTO Mike Schroepfer later tweeted — because there was no other way to communicate. "We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible."

The outage was the result of a fairly straightforward mistake. There were some initial rumors and conspiracy theories (coming a day after Frances Haugen outed herself as a whistleblower and a day before a Congressional hearing, how could there not be?), but the truth appears to be much more routine. It involves BGP, the tech that helps networks communicate, and Facebook basically "took away the map telling the world's computers how to find its various online properties," as Brian Krebs put it.

• "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication," Facebook's Santosh Janardhan wrote on the company's blog last night. "This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt."
• The problem was simple, but the fix was complicated. "The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations," Janardhan said, "complicating our attempts to quickly diagnose and resolve the problem."
• Cloudflare published a detailed blog post about what happened, showing a quick burst of BGP updates just before the outage began. "With those withdrawals," Cloudflare's Tom Strickx and Celso Martinho wrote, "Facebook and its sites had effectively disconnected themselves from the internet."

The internet, of course, absolutely loved the whole thing. The #deleteFacebook hashtag worked, declared seemingly everybody. There were MySpace jokes galore. Twitter weighed in. So did God. Others, though, saw bigger takeaways from the outage:

• "Facebook, WhatsApp, and Instagram all going down at the same time sure seems like an easily-understandable and publicly-popular example of why breaking up a certain monopoly into at least three pieces might not be a bad idea," Edward Snowden tweeted.
• Many people used the outage as a chance to appeal for a decentralized internet (and in some cases, try to sell people on their blockchain-based social app.) "Kudos to @facebook for giving us a very real demonstration of why the move to a decentralised Web 3 is necessary and, indeed, inevitable," Polkadot founder and Ethereum co-creator Gavin Wood tweeted.

Seeing an internet without Facebook was eye-opening, both for the businesses that rely on the company to reach customers and share information, and for the people — particularly in the developing world — who rely on it to communicate with loved ones. It was also a reminder that the internet's infrastructure is still fragile and complex and always barely holding together.

• By about 5 p.m. ET, Facebook appeared to be starting to recover. By the end of the work day on the West coast, things were back to normal.
• At the end of it all, Mark Zuckerberg weighed in. "Facebook, Instagram, WhatsApp and Messenger are coming back online now," he posted on his newly functioning Facebook page. "Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about."

The big lesson here? Don't put all your eggs in one basket. Everything at Facebook runs on a single system, from its platforms to its status pages to its in-office security systems, so breaking one thing broke everything. And when the servers need to be reset, you really don't want to rely on the servers to let you into the building. Expect Facebook to make changes, and others to make sure this doesn't happen to them next.

— David Pierce (email | twitter)

A MESSAGE FROM FACEBOOK

We've more than quadrupled our safety and security teams to 40,000 in the last 5 years to stop bad actors and remove illicit content. It's working: In just the past few months, we took down 1.7 billion fake accounts & 7.1 million terrorism-related posts. But our work to reduce harmful and illicit content on our platforms is never done. We're working to help you connect safely.

Learn more

People Are Talking

Jony Ive described what he misses most about Steve Jobs, who died 10 years ago today:

• "Steve's last words to me were that he would miss talking together … I miss Steve desperately and I will always miss not talking with him."

Frances Haugen is expected to tell lawmakers today that Facebook has no oversight:

• "When we figured out cars were safer with seat belts, the government took action … I implore you to do the same here."

Danish lawmaker Christel Schaldemose wants a Facebook probe after Haugen's claims:

• "The Facebook Files — and the revelations that the whistleblower has presented to us — underscores just how important it is that we do not let the large tech companies regulate themselves."

Profits shouldn't overrule trust or safety, Marc Benioff said:

• "You may not pay right away (& you may be able to deceive), but eventually — you will pay. Truth always wins!"

While trying to get an FTC antitrust suit tossed, Facebook spokesperson Chris Sgro said the company has plenty of competition:

• "The FTC's fictional market ignores the competitive reality: Facebook competes vigorously with TikTok, iMessage, Twitter, Snapchat, LinkedIn, YouTube and countless others."

Making Moves

Mike Park is leaving Twitter. Park is the company's VP of product, and Tony Haile will take over the role.

GlobalFoundries is eyeing a U.S. IPO. In the United States, the Abu-Dhabi-owned semiconductor firm has plants in New York State and Burlington, Vermont.

Ashkan Soltani will take charge of the California Privacy Protection Agency. He played a big role in writing the California Consumer Privacy Act and the California Privacy Rights Act.

Alex Hungate is joining Grab as COO. Hungate has been the head of in-flight catering services company SATS for eight years.

Margo Stern is heading to Peloton to lead content design. Stern worked on content strategy at Google, Twitter and Facebook and most recently led content design at Level.

Udemy filed for an IPO. Its revenue grew 56% last year amid the pandemic-driven shift to remote learning, according to its paperwork.

In Other News

• Facebook misled investors about a shrinking user base among younger people, according to a complaint filed with the SEC by Frances Haugen.
• An ex-Tesla employee was awarded $130 million in damages. A jury found that Owen Diaz was subjected to a racially hostile work environment.
• Captain Kirk is going to space … for real. Actor William Shatner will board Blue Origin's crewed spaceflight next week alongside three other passengers, including Planet Labs co-founder Chris Boshuizen.
• "Squid Game" is getting everyone back on Netflix. Its app topped App Store rankings for the first time in years.
• TikTok should meet with Connecticut educators and parents, the state's attorney general said. William Tong said a viral challenge on the platform to "slap a teacher" could harm teachers, and that TikTok should promise reform.
• On Protocol: Trust in tech is tanking, according to a new study. The industry was one of the most trusted for a solid six years, but it's now one of the least trusted sectors, just ahead of the pharmaceutical industry, health insurance and energy.
• Some workers at Citizen are unionizing. Dozens of employees filed a petition with the NLRB, but the company doesn't support their unionization efforts.
• Amazon is celebrating Black Friday early. The company is introducing "Black Friday-worthy" deals to get more people shopping earlier, while big retailers try to ensure people get their holiday gifts on time.
• A Texas man was sentenced for plotting to blow up an Amazon data center. Seth Aaron Pendley was given 10 years in federal prison for the plan, which he said was meant to "f*** up the Amazon servers."

One More Thing

Meet Matt Aimonetti

On the surface, Matt Aimonetti looks like just another person working on the metaverse at a large tech company (Microsoft). But if you look at his track record, he's done a lot in the music industry and takes on a bit of work outside engineering.

Aimonetti founded Splice, a platform for music creators, and has said he owes the music world for his professional career. In between leaving Splice and joining Microsoft, he even worked on a project with Grimes and launched a newsletter on streaming recommendations. He's one of those people who may work at Big Tech, but hasn't let go of his old passions.

We're featuring tech-industry creators and leaders we think you might like here every Tuesday. If you have folks you think everyone should know about, send them our way!

A MESSAGE FROM FACEBOOK

We've invested $13 billion in teams and technology over the last 5 years to enhance safety.It's working: In just the past few months, we took down 1.7 billion fake accounts to stop bad actors from doing harm. But there's more to do. We're working to help you connect safely.

Learn more