Disaster recovery is not an option

The Big Story

Up in smoke

This week served as an unfortunate reminder for a lot of companies that backups and disaster-recovery plans are not "nice-to-haves" in the modern economy.

First it was the massive Exchange hack, which has compromised tens of thousands of email servers that, even in the Slack and Microsoft Teams era, are probably how most companies run their businesses. And then another nightmare scenario unfolded Tuesday night when a data center at OVH, the European cloud provider, caught fire and was destroyed, forcing it to shut down all activity in its Strasbourg, France region.

Enterprise tech customers can't control security flaws in the products they buy, and they certainly can't control fires at remote data centers. But they can control how they manage and protect their corporate data and user data, both of which are priceless.

The Exchange hack is expected to get very real in the next several days: Security experts believe that the hackers behind the exploits will activate back doors placed on those servers before Microsoft was able to release a rare out-of-cycle patch for the flaws.

• Even if Exchange users find and disable one of those back doors, there might be several more: CISA released a list of tips for companies running their own Exchange servers to use to try and contain the damage.
• However, "companies whose Exchange servers give way to even a remote chance for spying should probably rebuild their mail servers completely. The web shell is simply the first of many possible backdoors the attacker might have installed," argued Nicholas Weaver of the University of California, Berkeley.
• However, that might require them to restore a backup that predates the exploit, which could cause problems for weeks or months of emails.

But the solution here isn't just to migrate to the cloud. You'd be forgiven for wondering: Why would anyone bother to manage their own Exchange server, when you can just let Microsoft or Google run them? But things can go wrong in the cloud too.

• We still don't know the full extent of the damage to OVH's Strasbourg site, but one data center was completely destroyed by fire and water, and another damaged. Thousands of servers were probably lost, and two other data centers at the site remained shut down throughout Wednesday.
• Likewise, no cause has been released as of this writing, but the SBG2 data center consumed by the fire appeared to consume 2 megawatts of power, roughly the same amount of electricity used by over 1,000 homes.
• Facepunch, a UK-based developer of a game called Rust, "confirmed a total loss of the affected EU servers" running in Strasbourg, erasing some player data for good.
• It could take several days to restore any type of service from Strasbourg depending on the condition of the site, according to OVH, which could be a big problem for companies that built applications around that region's servers.

There are lessons here for all enterprise customers. OVH is a niche player in cloud computing, even in Europe. But disasters — natural or man-made — will happen at large buildings that consume incredible amounts of electricity, and security events like the Exchange flaw will happen much more frequently.

• Companies that need to protect vital customer data, or remain online no matter what, need to build their applications around multiple regions, or at least across availability zones offered by major cloud providers to limit the fallout from any one disaster.
• Moving email to the cloud will solve a lot of security problems, but it's a big project that needs careful planning, not a panicked rush job in the wake of a hack.
• In a real-time world, nightly backups and monthly patch cycles probably won't cut it.
• Regulators can help here by reexamining the type of data that needs to be kept on-premises in certain industries, and whether or not cloud solutions offer sufficient protection.

Disaster recovery can't just be a document that sits in some untouched corner of a Google Drive only to surface during a frantic search amid a crisis; it needs to be a core part of every company's IT strategy.

• Build an easier-to-use, mostly-foolproof corporate backup and recovery system, and enterprise customers (not to mention venture capitalists) will beat a path to your door.

— Tom Krazit

JOIN US

Section 230 of the Communications Decency Act is the most-discussed and least-understood law governing the modern internet. This event will delve into the future of Section 230 and how to change the law without compromising the internet as we know it. Join Protocol's Emily Birnbaum and Issie Lapowsky in conversation with Senator Mark Warner. This event is presented by Internet Association.

RSVP for this event.

This Week On Protocol

Cashing in: The passage of the COVID-19 relief bill this week will see about $1 billion flow into the Technology Modernization Fund, which does exactly what you think a fund with that name would do. It's a smaller windfall than hoped for by both the tech industry and government reformers, according to Protocol's Emily Birnbaum.

Five Questions For...

Hubert Palan, Founder and CEO of Productboard

What's the best piece of advice you could give to someone starting their first tech job?

Make sure to understand the technology, but realize that even more important is deep knowledge of customer needs. Learn about psychology and nurture your empathy.

Pick one piece of consumer or business software (that isn't sold by your company) that you can't live without.

My whole house runs on Google Home and I enjoy the convenience and control over all aspects of our household (even though usability could be much better!).

What was the first computer that made you realize the power of computing and connectivity?

My mom was a developer and in the late '80s, together with three other families, we bought an IBM PC AT knock-off and I dialed into the first [Bulletin Board Service]. This was before the World Wide Web was invented. It was magical.

What was the biggest reason for the success of cloud computing over the past decade?

Cloud computing eliminated high entry barriers into the software world. The cost of building a software service declined dramatically and modularization eliminated the need to build everything from scratch which further sped up software delivery times.

What will be the biggest challenge for cloud computing over the coming decade?

I expect regulatory challenges for the dominant players. I foresee heightened competition and fragmentation of services with specialized services for every little niche. I am curious how humanity will be able to adapt to the constantly accelerating rate of innovation.

Around the Enterprise

• A scary hack into internet-connected surveillance cameras sold by Verkada allowed one group to download a ton of video footage from inside offices belonging to companies such as Tesla and Cloudflare as well as hospitals and schools.
• Oracle's prediction of lower earnings guidancefor the upcoming quarter did not endear it to Wall Street, and to make its Wednesday afternoon worse, its network infrastructure went down.
• Zapier has only raised $1.3 million in funding to build out its app automation startup, but it's now worth $5 billion.
• Data centers aren't dead, and a group of former data center executives just raised $250 million through a SPAC, likely to invest in a big data center operator, according to Data Center Knowledge.
• MongoDB reported a 38% jump in revenue, a smaller-than-expected loss, and raised its revenue guidance to hit the enterprise earnings trifecta.
• Salesforce is pulling out of a planned office dealnear its San Francisco headquarters and plans to sublease more space in another building near Salesforce Tower after the pandemic emptied out downtown San Francisco last year.
• Microsoft formally retracted a quantum-computing paper that was the basis for some of its quantum research after realizing it had made some calculation errors.
• Cisco named Marianna Tessel to its board of directors. Tessel is currently CTO at Intuit and was formerly an executive at Docker.
• It's amazing how relevant Microsoft Excel remains to modern business 36 years after it made its debut, and Packy McCormick has a nice long read on the original no-code tool.