The time bombs ticking inside the world’s software

All hands on deck

Be nice to any enterprise software developers you come across today; they did not have a good weekend.

Companies around the world are scrambling to limit the damage from one of the most consequential open-source software security vulnerabilities discovered in years, first publicly disclosed on Friday. A bug in a program called Log4j, which has been used in countless numbers of Java applications built over the last two decades, forced pretty much every company doing business on the internet to scrutinize their software in order to determine if they were vulnerable.

Why is this vulnerability so bad? Most security holes require a certain degree of expertise to exploit. The effort required with this one, which is being called “Log4Shell,” is trivial.

• Most software (and certainly all commercial software) maintains a log of all activity that happens while the software is running. This allows developers to go back and figure out what went wrong when users are having trouble.
• That activity includes keystrokes that users enter into web forms on a site.
• The Log4Shell bug allows an attacker to simply enter a carefully crafted string of characters in a web form that, once logged, directs the computer on which it is running to download malicious code.
• At that point, your computer is no longer yours.
• Malware designed to exploit the vulnerability was starting to spread Sunday night, installing (what else) crypto-mining software on affected systems. The next wave could be more damaging.

There are two further compounding problems. One is simply to do with scale: Java has been one of the most popular enterprise software programming languages for a very long time, and Log4j is one of the most popular logging tools used in Java applications. The other is to do with how software is built: Log4j has also been used in an enormous number of open-source software programs that often serve as a foundation for other software.

• Open-source software has led to an explosion in enterprise software innovation over the last two decades, but there’s an open secret in this world: Scores of popular and prominent open-source projects are maintained by a handful of people who aren’t necessarily paid to do that work.
• This prescient illustration from xkcd probably saw more traffic this weekend than it has in years.
• The last time enterprise tech faced such a crisis, when the Heartbleed vulnerability was discovered in OpenSSL in 2014, almost all the major enterprise tech companies agreed to contribute to a fund that would maintain the security of OpenSSL and other key open-source projects.
• The problem, however, is not a lack of money: There are so many open-source projects that have been used to build some of the most critical software in the world that simply identifying the ones that need support is an enormous challenge.

Ideally, open-source foundations could solve this: Apache or the Linux Foundation would organize efforts and pay maintainers to keep those open-source projects up to date. But that idea is also more complicated than it sounds.

• Once a developer is on a payroll, that developer is subject to the whims of the person who is cutting the check.
• Critics of open-source foundations already think they are too beholden to their corporate donors.
• But the current situation is untenable; the three maintainers in charge of Log4j have been working nonstop for over a week to patch the code without any compensation.
• “Now is the perfect time for Open Source maintainers to become legible to the big companies that depend on them—and that want to get more out of them—and send them five-to-six figure invoices,” said Filippo Valsorda, a Google employee who works on the Go programming language, on his personal blog (emphasis his).

This is an enormous, existential problem for a world that increasingly runs on software. And it once again highlights the software supply-chain security issues that have been top of mind all year following the SolarWinds attacks.

• Modern software applications have been assembled from dozens (if not hundreds) of pieces, many of which are open-source projects because everyone agreed long ago that it doesn’t make sense to reinvent the basic plumbing all software needs to work every time a new application is created.
• This ubiquity allowed software to flourish; the history of enterprise tech would be very different if developers had to license and pay someone for all that basic plumbing, or if they had to build it themselves.

But we’ve moved fast, and things are broken. It’s far past the time to find a solution to the open-source security problem, because the next time this happens, it will only be worse.

— Tom Krazit (email | twitter)

A MESSAGE FROM LEXMARK

Lexmark, a leading provider of printers and imaging equipment — one of the first IoT devices — understands the potential as well as the challenges better than most. We sat down with Lexmark CEO Allen Waugerman to discuss this major development, which he calls one of the most significant milestones in the company’s 30-year history.

Learn more

This week on Protocol

HashIPO: It was a good week for HashiCorp, which executed its long-expected IPO Thursday that valued the company at around $15 billion and minted two new billionaires in the process. I spoke with CEO Dave McJannet and co-founder and CTO Armon Dadgar about how the company got to this point, how long its cozy relationship with the cloud providers can last and why its future depends on managed services.

Practice what you preach: Protocol’s Biz Carson yelled at me in a Slack thread for suggesting that the increasing interest of venture capital firms in enterprise software tailored to their needs should be called “VCaaS,” but she doesn’t write this newsletter. It makes sense, however, that the companies that have gotten quite rich from the explosion in industry-focused enterprise would start to realize they need tools of their own.

Upcoming at Protocol

Gaming platforms have traditionally been defined by their hardware, from arcades to personal computers to home consoles — and now, mobile phones. But cloud gaming, the rise of AR/VR and the promise of the metaverse have begun to redefine the very nature of gaming platforms and revolutionize the nature of play.

Join Protocol’s Nick Statt next Tuesday, Dec. 14, at 10 a.m. PT / 1 p.m. ET for a virtual event discussing the future of our entertainment platforms with Frederic Descamps, CEO and co-founder of Manticore Games; Chris Mahoney, senior manager of central product development at Zynga; and Kellee Santiago, director of external publishing at Niantic. RSVP here.

A MESSAGE FROM LEXMARK

The trajectory for IoT market growth is exponential. For manufacturers, IoT will enable access to real-time data so they can immediately see what's happening across their fleet and act on those insights — in some cases, before customers even notice an issue. Yet all too often, companies buy the building blocks, but struggle with the construction.

Learn more

Around the enterprise

AWS released a root-cause analysisof last Tuesday’s major outage, and said it will be releasing a new status page early next year to give customers better information about in-progress outages.

Salesforce acquired MuleSoft three years ago to add application integration tech to its arsenal, but that division is struggling amid staff turnover and culture clashes, according to Bloomberg.

Oracle stock enjoyed a strong rise to close the week, thanks to better-than-expected quarterly earnings. And Larry Ellison said some stuff, as usual.

JEDI didn’t just get a new name. The process of choosing cloud vendors for the JWCC, the Pentagon’s new cloud project, is going to look very different this time around, according to Federal News Network; and not just because the DoD ditched its winner-take-all strategy.