Software supply chain security scares spread

The Big Story

Cover up

A new software supply-chain hack surfaced this week, making it clear that the massive SolarWinds hack was just a preview of what could await companies that depend on countless vendors for the software they need to operate in the 21st century.

The nature of the product involved in this attack — software development testing tools from a startup called Codecov — is especially troubling because by definition it requires access to sensitive assets on a company's network in order to accomplish its tasks. Codecov is a 35-person seed-stage company whose products are used by a number of high-profile companies, including IBM, HPE and RBC.

In the complex world of modern software development, there are dozens of moving parts, inside even smaller companies. And under pressure to continuously ship software, companies rely on tools that if compromised, could cause more problems than they solve.

• Codecov's tool allows customers to analyze code running on testing servers in an attempt to find potential errors when that new code is pushed to production.
• Everybody (well, almost everybody) tests their software before pushing it live, but it can be tricky to know when you've run enough tests to feel confident that code will work.
• But you can't run indefinite numbers of tests on new code, otherwise you'll never ship anything.
• So tools like Codecov run tests against new code and generate a result measured against internal performance standards that tells you how much of that code actually executed; the more code that executes, the more likely it is that the software will work.

That means Codecov needs access to its customers' software development assembly lines in order to run those tests. And someone determined to gain access to the internal networks of Codecov customers managed to infiltrate its tool.

• Codecov notified customers last week that it discovered its software had been compromised with a backdoor as early as January, but it wasn't clear until this week how widespread the damage might be.
• It's also not entirely clear how hackers gained access to Codecov, but once they did they "put extra effort into using Codecov to get inside other makers of software development programs, as well as companies that themselves provide many customers with technology services, including IBM," Reuters reported.
• IBM said it had not found any evidence of its own code being altered, but it's not clear if the hackers had used their access to IBM to breach customer networks.
• Still, the backdoor could have allowed the attackers to siphon loads of login credentials related to key enterprise tech services such as AWS or GitHub, because Codecov's tools would need to know where code was stored and running.

Software supply-chain hacks aren't entirely new, but the SolarWinds breach discovered last year underscored that big compromises at small vendors can have enormous effects.

• Startups like Codecov have thrived over the past several years because there's just no way even huge companies like IBM can make all the software tools they need themselves.
• And these attacks are especially damaging, because in order to use tools that sit at the heart of software development pipelines, customers have to trust them with so much sensitive information.

Codecov removed the backdoor from the latest edition of its tool and urged customers to upgrade and change their login credentials. The story of this supply chain hack is just beginning, and it won't be the last such attempt to infiltrate critical infrastructure.

— Tom Krazit

A MESSAGE FROM HEWLETT PACKARD ENTERPRISE

Global lockdown orders and the ongoing pandemic have disrupted supply chains, underscoring the need for greater agility and resilience. On the season premiere of The Element Podcast, leading experts from MIT and Hewlett Packard Enterprise explore how companies are rapidly transforming their supply chains and adapting for a more dynamic and digital future.

Learn more

This Week On Protocol

Data-center gigs: Don't miss this story about life for Google's data-center employees. Protocol's Anna Kramer talked to several current and former employees who, for the most part, like doing the work required to keep servers up and running — but they want a path to full-time employment.

Cloudy gamers: Last year looked like it was going to be the year of cloud gaming thanks to technical advances and the pandemic, but it hasn't quite worked out that way. This week's Protocol Gaming takes a look at how different cloud gaming models fared in 2020, and what might come next.

On the path: UiPath went public Wednesday, raising $1.35 billion for its robotic-process automation software services and ending the day valued at around $36 billion. Protocol's Joe Williams spoke with CFO Ashim Gupta, who acknowledged the intense interest in enterprise cloud stocks right now but said "the best way to combat hype is by progress."

Five Questions For...

David Friend, CEO, Wasabi Technologies

What was your first tech job?

I built the electronic music studio at Yale University as an undergraduate bursary job. This was an incredible experience for me as I'd later go on to co-found ARP Instruments which developed synthesizers used by Stevie Wonder, David Bowie and Led Zeppelin, and even helped Steven Spielberg communicate with aliens providing that legendary five-note communication in "Close Encounters of the Third Kind."

What was the first computer that made you realize the power of computing and connectivity?

My first software company, Computer Pictures Inc., was an early player in computer graphics. I built the Computer Pictures software on an Apple II computer. This was one of the first times I really experienced the extensive power of computing and is what really got me into the business of software.

What was the biggest reason for the success of cloud computing over the past decade?

Although I can only speak to cloud storage specifically, what I have noticed is that no one wants to own generic storage servers anymore. It is cheaper and more simple to store data in the cloud and it obviously comes with less capacity restrictions than on-premises storage. As more data moves to the cloud over time, we'll see the benefits of greater data accessibility on cloud computing as well.

What will be the biggest challenge for cloud computing over the coming decade?

There is a rapidly growing need for large amounts of cloud data storage. One of the challenges we notice, that should be avoided, is not getting locked into a single vendor. To get real value from the cloud, the cloud has to be a richer environment with multiple vendors focusing on parts of the ecosystem that they can do better than anyone else. The hardware world evolved this way, with dozens or hundreds of specialty vendors, and the cloud environment should do the same.

Will the pandemic usher in a new era of remote working, or will we all come back together when it is safe to do so?

I believe workplace operations will go back to 75% of the way they were pre-pandemic with employees going into the office three to four times a week. Although it is possible to complete the majority of their work from home, people still need some amount of face-to-face contact. It energizes us, makes us more creative and ultimately drives greater innovation.

Around the Enterprise

• Dish will attempt to catch up to the leading wireless carrier companies by building a brand-new 5G network on AWS.
• Signal's Moxie Marlinspike published details of security flaws in the Cellebrite phone-unlocking technology used to bypass Signal's own protections after claiming to have stumbled upon one of Cellebrite's devices that fell off a truck in San Francisco, which ... ¯\_(ツ)_/¯.
• AnandTech published a very detailed interview on Intel's latest server technology with Lisa Spelman, vice president of its Xeon and Memory Group.
• Thin clients are back: Microsoft is getting ready to launch a "Cloud PC" service this summer that would allow users of any device to access a corporate Microsoft-managed desktop environment running on Azure, according to ZDNet's Mary Jo Foley.
• IBM eked out an increase in quarterly revenue after a year of declines, and predicted more of the same for 2021.
• IPO timing can be a divisive issue for executives who want to stay private and board members who want a return on their investment, which appears to have been the case at AI startup DataRobot.
• Israel has selected Amazon and Google to build a "local cloud" for government use, promising that all data will stay inside the country's borders.
• Grafana became the latest enterprise vendor to introduce new, more restrictive licensing for an open-source project it developed, although a commercial agreement with AWS means that the cloud leader will continue to offer its own managed version of the project.
• Pulumi, one of the newer companies to offer a take on managing infrastructure with code, introduced the 3.0 version of its flagship service.
• Greg Kroah-Hartman, not amused by a second attempt from researchers at the University of Minnesota to submit intentionally buggy patches to Linux, banned all submissions to the Linux kernel from email addresses belonging to the Golden Gophers.

A MESSAGE FROM HEWLETT PACKARD ENTERPRISE

Global lockdown orders and the ongoing pandemic have disrupted supply chains, underscoring the need for greater agility and resilience. On the season premiere of The Element Podcast, leading experts from MIT and Hewlett Packard Enterprise explore how companies are rapidly transforming their supply chains and adapting for a more dynamic and digital future.

Learn more