Power

Critical systems buckled under coronavirus traffic. Could it have been avoided?

Unemployment websites and other services are bending and breaking under traffic pressure. What could be done to fix them?

Screenshot of the New Jersey Department of Labor & Workforce Development website

In New Jersey, the state's unemployment benefits site is asking applicants to log on at different times of the day based on the last four digits of their Social Security number to help mitigate demand.

Screenshot: New Jersey Department of Labor & Workforce Development

The coronavirus pandemic has sent shockwaves through systems across the web, perhaps none more so than the unemployment benefits websites that were not set up to deal with the sudden crush of traffic.

Tens of thousands of workers have been laid off or furloughed as cities, states and countries issue shelter-in-place ordinances. In the U.S. alone there's predicted to be around 2.25 million unemployment claims from last week, up from around 281,000 the week prior.


Get what matters in tech, in your inbox every morning. Sign up for Source Code.


The cascading effects on benefits systems, many of which are based on outmoded technologies designed for much lighter traffic loads, have been devastating. The problems could have been avoided — but experts say getting organizations of all kinds to embrace better technologies before it's too late can be challenging.

Major slowdowns

In some areas of New York, claims were up over 1,000%, according to a representative of the state's department of labor. As a result, the state's site has crashed for people looking to apply for benefits.

"To address this unprecedented increase, we have added server capacity, increased bandwidth, extended phone hours, and dedicated more than 700 staff members to address the influx," Deanna Cohen, the deputy director of communications at the department, told Protocol. From March 16 to 21, the state's site had 2,270,300 web hits and saw around a 400% increase in logins on some days, Cohen said. "Our representatives are doing the best they can to make sure everyone is served, even if it takes longer than usual," she added.

In Ohio, jobless claims shot up from under 5,000 to nearly 140,000 in a single week, and the state's system buckled under the pressure, and briefly went completely offline on March 23. "During previous downturns in the economy, claims came in waves as the recession worsened and industries began to shut down, whereas these claims came in all at once," Bret Crow, a representative for Ohio's Department of Job and Family Services, told Protocol. "This amount of claims in this short expanse of time would tax any online system."

Crow said the state has been "working around the clock" to add servers to deal with the additional traffic. The state's current unemployment website was built in 2004, back at a time before modern smartphones, and when a large slice of the American population still got their internet from AOL. Crow added that Ohio is working with Sagitec Solutions (a leading provider of unemployment systems that Crow said did not build the original site) to modernize the state's unemployment system. He was not optimistic about the short-term prospects. "Given the complexity of UI benefits, it will take a minimum of 24 months to configure the system to match our law," Crow said.

Tens of thousands of new applicants in the U.S. had a similar effect on benefits sites across the country. Pennsylvania, Oregon and New Jersey, among other states, all saw their employment and labor sites slow or crash under the unprecedented demand. In New Jersey, the state's unemployment benefits site is asking applicants to log on at different times of the day based on the last four digits of their Social Security number to help mitigate the crush. In Colorado and New York, the state is requesting citizens to apply on different days depending on the first letter of their last name.

And the problem is global: Australia's MyGov, a site for accessing government services, slowed to a crawl last week after thousands more people than usual attempted to apply for unemployment benefits. The government initially claimed it was a DDoS attack. In the U.K., reports suggest that thousands of people have been left waiting in a digital queue to access the Universal Credits benefits system.

The problem of systems not built for scale

For the most part, the backbone of the internet has held up admirably under increased traffic, and many modern sites are built atop cloud services that allow them to scale with demand. But sites that are built on older, more rigid software and hardware, like many unemployment sites, are struggling under the pressure, experts say.

"Any piece of hardware or any piece of software has some maximum capacity," said Josh Chessman, a senior director analyst at Gartner who covers the IT industry. "That's just the reality of life, it doesn't matter if it's your phone or a server or a mainframe — there's a limit to what it can do."

When building web applications, Chessman said, you'll have a plan in mind: say you're expecting to get 1,000 visitors, you may build it to be able to maintain 2,000 visitors, to give yourself a little leeway. "The problem comes when you're in that position, but all of a sudden you go from X to 500 X overnight," he added.

And it's not just the front-end architecture of these websites that's buckling. The database systems managing people's private data — things that companies and governments have been reluctant to move to the cloud — are also straining under unintended use.

"Very often these back-end systems are running on technology that could be 30, 40 or 50 or more years old," Chessman said of these sites that often struggle in high-stress situations. "You've put something in place that works really well, and you put it in place in 1970 or 1980 and the problem is if it's really working well, nobody wants to upgrade it because (a) why?, and (b), it means downtime."

Outside the world of benefits, other suddenly essential services, including some that should be technologically adept, have been stung, as well.

U.K. retailer Ocado, one of the early success stories in online groceries, experienced pretty much the exact issue Chessman outlined. Demand for grocery deliveries has skyrocketed as fewer and fewer people are able to leave their homes. To deal with the sheer weight of customers attempting to book deliveries, Ocado put up a digital queue page, much like with the Universal Credits system, where thousands of customers waited for hours to potentially get groceries. The company compared the demand to a "denial-of-service attack," had to pause new customer signups, and even suspended service temporarily to get through the backlog.

"At some point … just adding more servers isn't making it faster because of something else," Chessman said. "It could be communication between servers, back-end stuff — that's another issue that can happen and cause problems."

IT inertia vs. digital transformation

Across the board, experts agree that while it's easy to fault organizations for not adopting newer, more resilient technologies, it's understandable that administrators haven't leaped to upgrade systems that are fundamentally working most of the time. But in light of their current failures, it's possible to fix the flaws, given time, willingness — and cash.

"Things change. It's not like these systems were built poorly 10, 15, 20 years ago — they were just built [with] 10, 15, 20 years ago's knowledge," said Peter Klayman, head of strategy at Bottle Rocket, an agency that works with companies on digital transformation. "Now, if you know that you have cloud capacity, it changes the way our enterprise architecture practice approaches problems because they're able to architect it for scale."

Some companies have leaned into the digital transformation that has been taking place over the last decade, and are better-suited to be flexible in pressing times, Klayman said. He pointed to fast-food restaurants like Chick-Fil-A and Burger King, as well as sites like Netflix, which built their systems on cloud platforms that will allow them to scale as demand does.

But for companies looking to offer some new mobile solution — such as curbside pickup of food — or build out a robust digital experience for thousands of new users, executives should be wary of what's really going to be possible. Chick-Fil-A's app was in continual development for over three years, Klayman said, and isn't something you can just throw together. "You can't just stand that stuff up right now," he said.

Even businesses that want to take advantage of the coronavirus pandemic — such as food and delivery companies looking to add curbside drop-offs or scheduled deliveries into their apps — are going to likely struggle to get anything done quickly. "Everyone's running around with their hair on fire," Klayman said. "It's very difficult to get the appropriate number of stakeholders together to actually launch a new enterprise endeavor."

"That's quite a lot of logistical undertaking and change management to do at a time when everyone is stressed, when people are concerned about their jobs, when people are concerned about their health," Klayman added.

"If you are on these legacy applications, even small changes, it takes our clients months," Wipro Digital President Rajan Kohli told Protocol. In some companies it can take six approvals in six months just to get a small change made, Kohli said. "That's how complex the environment is: It's not that the people are slow — it is because it touches six systems on four different release cycles, and by the end of it, they have to wait six months to just to get that small change in."

But after the virus has started to recede and life starts to return to normal, should companies begin to invest in more durable, flexible systems? It really depends on what they do.


Get in touch with us: Share information securely with Protocol via encrypted Signal or WhatsApp message, at 415-214-4715 or through our anonymous SecureDrop.


"It's almost always a case of money can fix it, but it doesn't make sense to spend that money upfront or deal with it afterwards — or hope it never happens," Chessman said. But not every business is likely to see a cost-benefit analysis where the likelihood of another pandemic happening anytime soon means they should invest in their systems.

"I wouldn't say they should because why spend the money on something that you might need once every five years, 10 years versus an Amazon or somebody like that who is pretty critical to the country right now," Chessman said.

Entertainment

Beat Saber, Bored Apes and more: What to do this weekend

Don't know what to do this weekend? We've got you covered.

Images: Ross Belot/Flickr; IGBD; BAYC

This week we’re listening to “Harvest Moon” on repeat; burning some calories playing Beat Saber; and learning all about the artist behind the goofy ape pics that everyone (including Gwyneth Paltrow?) is talking about.

Neil Young: Off Spotify? No problem.

Neil Young removed his music from Spotify this week, but countless recordings are still available on YouTube, including this 1971 video of him performing “Heart of Gold” in front of a live studio audience, complete with some charming impromptu banter. And while you’re there, scroll down and read a few of the top-rated comments. I promise you won’t be disappointed.

'Archive 81': Not based on a book, but on a podcast!

Netflix’s latest hit show is a supernatural mystery horror mini-series, and I have to admit that I was on the fence about it many times, in part because the plot just often didn’t add up. But then the main character, Dan the film buff and archivist, would put on his gloves, get in the zone, and meticulously restore a severely damaged, decades old video tape, and proceed to look for some meaning beyond the images. That ritual, and the sentiment that we produce, consume and collect media for something more than meets the eye, ultimately saved the show, despite some shortcomings.

'Secrets of Sulphur Springs': Season 2 is out now

If you’re looking for a mystery that's a little more family-friendly, give this show about a haunted hotel, time travel, and kids growing up in a world that their parents don’t fully understand a try. Season 2 dropped on Disney+ this month, and it not only includes a lot more time travel mysteries, but even uses the show’s time machine to tackle subjects as serious as reparations.

The artist behind those Bored Apes

Remember how NFTs are supposed to generate royalties with every resale, and thus support artists better than any of their existing revenue streams? Seneca, the artist who was instrumental in creating those iconic apes for the Bored Ape Yacht Club, wasn’t able to share details about her compensation in this Rolling Stone profile, but it sure sounds like she is not getting her fair share.

Beat Saber: Update incoming

Years later, Beat Saber remains my favorite VR game, which is why I was very excited to see a teaser video for cascading blocks, which could be arriving any day now. Time to bust out the Quest for some practice time this weekend!

Correction: Story has been updated to correct the spelling of Gwyneth Paltrow's name. This story was updated Jan. 28, 2022.


Janko Roettgers

Janko Roettgers (@jank0) is a senior reporter at Protocol, reporting on the shifting power dynamics between tech, media, and entertainment, including the impact of new technologies. Previously, Janko was Variety's first-ever technology writer in San Francisco, where he covered big tech and emerging technologies. He has reported for Gigaom, Frankfurter Rundschau, Berliner Zeitung, and ORF, among others. He has written three books on consumer cord-cutting and online music and co-edited an anthology on internet subcultures. He lives with his family in Oakland.

Sponsored Content

A CCO’s viewpoint on top enterprise priorities in 2022

The 2022 non-predictions guide to what your enterprise is working on starting this week

As Honeywell’s global chief commercial officer, I am privileged to have the vantage point of seeing the demands, challenges and dynamics that customers across the many sectors we cater to are experiencing and sharing.

This past year has brought upon all businesses and enterprises an unparalleled change and challenge. This was the case at Honeywell, for example, a company with a legacy in innovation and technology for over a century. When I joined the company just months before the pandemic hit we were already in the midst of an intense transformation under the leadership of CEO Darius Adamczyk. This transformation spanned our portfolio and business units. We were already actively working on products and solutions in advanced phases of rollouts that the world has shown a need and demand for pre-pandemic. Those included solutions in edge intelligence, remote operations, quantum computing, warehouse automation, building technologies, safety and health monitoring and of course ESG and climate tech which was based on our exceptional success over the previous decade.

Keep Reading Show less
Jeff Kimbell
Jeff Kimbell is Senior Vice President and Chief Commercial Officer at Honeywell. In this role, he has broad responsibilities to drive organic growth by enhancing global sales and marketing capabilities. Jeff has nearly three decades of leadership experience. Prior to joining Honeywell in 2019, Jeff served as a Partner in the Transformation Practice at McKinsey & Company, where he worked with companies facing operational and financial challenges and undergoing “good to great” transformations. Before that, he was an Operating Partner at Silver Lake Partners, a global leader in technology and held a similar position at Cerberus Capital LP. Jeff started his career as a Manufacturing Team Manager and Engineering Project Manager at Procter & Gamble before becoming a strategy consultant at Bain & Company and holding executive roles at Dell EMC and Transamerica Corporation. Jeff earned a B.S. in electrical engineering at Kansas State University and an M.B.A. at Dartmouth College.
Workplace

Mental health at work is still taboo. Here's how to make it easier.

Tech leaders, HR experts and organizational psychologists share tips for how to destigmatize mental health at work.

How to de-stigmatize mental health at work, according to experts.

Illustration: Christopher T. Fong/Protocol

When the pandemic started, HR software startup Phenom knew that its employees were going to need mental health support. So it started offering a meditation program, as well as a counselor available for therapy sessions.

To Chief People Officer Brad Goldoor’s surprise, utilization of these benefits was very low, starting at about a 10% take rate and eventually weaning off. His diagnosis: People still aren’t fully comfortable opening up about mental health, and they’re especially not comfortable engaging with their employer on the topic.

Keep Reading Show less
Michelle Ma

Michelle Ma (@himichellema) is a reporter at Protocol, where she writes about management, leadership and workplace issues in tech. Previously, she was a news editor of live journalism and special coverage for The Wall Street Journal. Prior to that, she worked as a staff writer at Wirecutter. She can be reached at mma@protocol.com.

Boost 2

Can Matt Mullenweg save the internet?

He's turning Automattic into a different kind of tech giant. But can he take on the trillion-dollar walled gardens and give the internet back to the people?

Matt Mullenweg, CEO of Automattic and founder of WordPress, poses for Protocol at his home in Houston, Texas.
Photo: Arturo Olmos for Protocol

In the early days of the pandemic, Matt Mullenweg didn't move to a compound in Hawaii, bug out to a bunker in New Zealand or head to Miami and start shilling for crypto. No, in the early days of the pandemic, Mullenweg bought an RV. He drove it all over the country, bouncing between Houston and San Francisco and Jackson Hole with plenty of stops in national parks. In between, he started doing some tinkering.

The tinkering is a part-time gig: Most of Mullenweg’s time is spent as CEO of Automattic, one of the web’s largest platforms. It’s best known as the company that runs WordPress.com, the hosted version of the blogging platform that powers about 43% of the websites on the internet. Since WordPress is open-source software, no company technically owns it, but Automattic provides tools and services and oversees most of the WordPress-powered internet. It’s also the owner of the booming ecommerce platform WooCommerce, Day One, the analytics tool Parse.ly and the podcast app Pocket Casts. Oh, and Tumblr. And Simplenote. And many others. That makes Mullenweg one of the most powerful CEOs in tech, and one of the most important voices in the debate over the future of the internet.

Keep Reading Show less
David Pierce

David Pierce ( @pierce) is Protocol's editorial director. Prior to joining Protocol, he was a columnist at The Wall Street Journal, a senior writer with Wired, and deputy editor at The Verge. He owns all the phones.

Fintech

Robinhood's regulatory troubles are just the tip of the iceberg

It’s easiest to blame Robinhood’s troubles on regulatory fallout, but its those troubles have obscured the larger issue: The company lacks an enduring competitive edge.

A crypto comeback might go a long way to help Robinhood’s revenue

Image: Olena Panasovska / Alex Muravev / Protocol

It’s been a full year since Robinhood weathered the memestock storm, and the company is now in much worse shape than many of us would have guessed back in January 2021. After announcing its Q4 earnings last night, Robinhood’s stock plunged into the single digits — just below $10 — down from a recent high of $70 in August 2021. That means Robinhood’s valuation dropped more than 84% in less than six months.

Investor confidence won’t be bolstered much by yesterday’s earnings results. Total net revenues dropped to $363 million from $365 million in the preceding quarter. In the quarter before that, Robinhood reported a much better $565 million in net revenue. Net losses were bad but not quite as bad as before: Robinhood reported a $423 million net loss in Q4, an improvement from the $1.3 billion net loss in Q3 2021. One of the most shocking data points: Average revenue per user dropped to $64, down from a recent high of $137 in Q1 2021. At the same time, Robinhood actually reported a decrease in monthly active users, from 18.9 million in Q3 2021 to 17.3 million in Q4 2021.

Keep Reading Show less
Hirsh Chitkara

Hirsh Chitkara ( @HirshChitkara) is a is a reporter at Protocol focused on the intersection of politics, technology and society. Before joining Protocol, he helped write a daily newsletter at Insider that covered all things Big Tech. He's based in New York and can be reached at hchitkara@protocol.com.

Workplace

Asana’s productivity expert wants you to ditch the 30-minute meeting

Professional organizer Joshua Zerkel says, sometimes you just need to stop working.

Joshua Zerkel is one of the first Certified Professional Organizers through the National Association of Productivity and Organizing Professionals.

Photo: Asana

When he entered the workforce, Joshua Zerkel didn’t know that productivity consultants were a thing. He was always the organized kid, alphabetizing his comic books and grouping his toys by category (G.I. Joes could never mix with Transformers). But his connection to productivity really clicked when he started working in web design. He quickly became the go-to employee for tips on staying focused.

“There are a lot of great talents that designers have, but one of them is typically not project organization and time management,” Zerkel joked.

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Latest Stories
Bulletins