The hack that shook Washington

The Big Story

SolarWinds, not all that fine

A cybersecurity nightmare is unfolding in the nation's capital as the fallout from one of the most brazen security breaches in recent memory continues to spread throughout several government agencies.

The internal networks belonging to no less than five government agencies, including the Defense and State Departments, were left wide open to a group of hackers believed to be working on behalf of Russia for several months this year, according to multiple reports and tacit confirmation of a breach from government officials earlier this week. This incident was especially scary because no one seems to know exactly how much data was accessed or stolen, and because those affected may never know.

The incident also highlights the vulnerability of the software supply chain, an important part of modern application development. Here's what we know:

• Sometime around spring 2020 (which was approximately 37 years ago), someone managed to infiltrate the software update systems belonging to SolarWinds, an enterprise software company used by many high-profile corporations and government agencies.
• Routine software updates for SolarWinds' Orion tool, which monitors the performance of customer applications, were compromised with malware that installed a backdoor allowing remote access to servers running Orion.
• (According to Reuters, which first broke the news of the breach, the company used the password "solarwinds123" on its update servers for a period of time last year, which is just amazing, if unrelated to this incident.)
• SolarWinds said that around 18,000 customers downloaded the affected updates earlier this year, and those customers included at least five different U.S. government agencies handling extremely sensitive data.
• FireEye, the first company to acknowledge the existence of the incident last week, published details on how other SolarWinds' customers running Orion can check their networks for signs of intrusion.

This isn't the first major cybersecurity event precipitated by a supply-chain attack: The NotPetya malware entered networks around the world through a vulnerability in tax-reporting software used by the government of Ukraine, as described in our profile of Microsoft CISO Bret Arsenault earlier this year.

• Cloud computing has allowed businesses to select software suppliers from an enormous number of providers over the last decade, rather than relying on a big package of software from a huge vendor or an internal development team to build everything needed to run a modern business on the internet.
• That only increases the attack surface that malicious hackers can exploit to get inside networks and systems.
• The SolarWinds incident is galling because the hackers were able to forge digital signatures that made their malware look like a SolarWinds-approved update, and because monitoring services — by definition — need to have access to a huge amount of internal software to do their job.
• So perversely, companies that promptly downloaded and installed vendor software updates were punished by this exploit.

The damage is probably limited to high-impact targets like the U.S. government agencies, security experts believe, due to the sophistication of this technique. That's in contrast to a more widespread event like NotPetya that took out corporate systems around the world.

• Still, it would appear that agents working on behalf of the Russian government had undetected access to the inner workings of the U.S. government for months.
• And as we know, the outgoing Trump administration has had a ... let's say, hands-off approach toward Russian cybersecurity activity over the last several years.
• Getting a handle on the scope of this breach will occupy the incoming Biden administration for several months, and a full accounting of the damage might be impossible.

But every hack is a wake-up call, presenting another opportunity to review the security of the software that runs the corporate world.

• Businesses expect that the software updates digitally signed by their vendors are legit, and if it gets easier to fake those credentials, expect to see more of these types of incidents.
• The malware appears to have been inserted during SolarWinds' software build process, which means everyone using continuous integration and continuous delivery tools to streamline the assembly process should double-check their security configurations.
• There are other essential enterprise tech tools, such as backup and recovery systems, that need to have widespread access to sensitive assets in order to do their jobs and could also be a vector for these types of incidents.

As for the lingering effects of the incident, which astounded even grizzled veterans of the last two decades of cybersecurity activity, The New York Times might have summed it up best:

• "Analysts said it was hard to know which was worse: that the federal government was blindsided again by Russian intelligence agencies, or that when it was evident what was happening, White House officials said nothing."

A MESSAGE FROM ZENDESK

Customer experience (CX) is having a moment. The pandemic has forced and accelerated the adoption of digital channels for many. This is great news. But customer expectations of the digital experience is high, and companies who are thoughtful about the customer experience and differentiate themselves will be winners.

Read more

This Week On Protocol

Punk rock: The release of Cyberpunk 2077 frustrated a lot of console gamers. But the game runs like a champ on Google Cloud's Stadia service, and Protocol's Seth Schiessel and Shakeel Hashim think this is a watershed moment for cloud gaming.

Congress goes quantum: A little-noticed provision in the massive military spending bill passed last week didn't sneak by Protocol's Emily Birnbaum, who reported that Congress authorized up to $10 billion in new annual spending on quantum computing, AI and 5G technologies over the next five years. Right now, that number is closer to $1.5 billion a year.

Righting the price: Coupa's spending-management tools are gaining traction as more companies embrace cloud services only to discover that older ways of tracking the budget just became obsolete. Protocol's Joe Williams spoke with CEO Rob Bernshteyn about the company's trajectory.

This week in AWS re:Invent

Amazon CTO Werner Vogels traditionally closes re:Invent with a deep-dive keynote into emerging technologies on AWS' radar. While it took us three weeks to get there in 2020, here's some of the major announcements from his address:

• Chaos engineering is a very interesting way to troubleshoot tech infrastructure, and AWS will provide chaos engineering as a service in a partnership with Grafana.
• Cloud users like simplicity, but not always: Sometimes they need access to the Linux command line for greater control of their environments. CloudShell is a new AWS service that does just that.
• Twitter announced that it plans to add AWS services alongside its own infrastructure and Google Cloud storage services, prompting hundreds of jokes from cloud users worried about not having a place to complain when AWS goes down.

Around the Cloud

• Google Cloud had a rough start to the week, enduring two separate outages affecting G Suite customers Monday and Tuesday.
• On the other hand, Google Cloud plans to beef up security with the hiring of its first CISO, tapping Phil Venables of Goldman Sachs, according to The Wall Street Journal.
• AWS released an amended complaint protesting the Department of Defense's decision to award the JEDI cloud contract to Microsoft. It claims (in a heavily-redacted court filing) that corrections to the Pentagon's cost-analysis calculations should make it the front runner.
• Oracle may be moving to Austin, Texas, but co-founder and CTO Larry Ellison plans to skip the taco lines in favor of the warm shores of Lanai, where he owns an astonishing 98% of the land.
• IBM and Red Hat kicked up a hornet's nest of angry Linux administrators over the last week by announcing that the current version of CentOS, a free version of Red Hat Linux, would be phased out.
• ClickUp raised a huge $100 million Series B round to value the office productivity software company at over $1 billion.
• If you're not following the funniest Simpsons-DevOps mashup account on Twitter, first read or listen to this interview from Corey Quinn with the protagonists behind @SimpsonsOps, then correct that mistake, neighborinos.

A MESSAGE FROM ZENDESK

Customer experience (CX) is having a moment. The pandemic has forced and accelerated the adoption of digital channels for many. This is great news. But customer expectations of the digital experience is high, and companies who are thoughtful about the customer experience and differentiate themselves will be winners.

Read more