yesMike MurphyNone
×

Get access to Protocol

I’ve already subscribed

Will be used in accordance with our Privacy Policy

Power

Critical systems buckled under coronavirus traffic. Could it have been avoided?

Unemployment websites and other services are bending and breaking under traffic pressure. What could be done to fix them?

Screenshot of the New Jersey Department of Labor & Workforce Development website

In New Jersey, the state's unemployment benefits site is asking applicants to log on at different times of the day based on the last four digits of their Social Security number to help mitigate demand.

Screenshot: New Jersey Department of Labor & Workforce Development

The coronavirus pandemic has sent shockwaves through systems across the web, perhaps none more so than the unemployment benefits websites that were not set up to deal with the sudden crush of traffic.

Tens of thousands of workers have been laid off or furloughed as cities, states and countries issue shelter-in-place ordinances. In the U.S. alone there's predicted to be around 2.25 million unemployment claims from last week, up from around 281,000 the week prior.


Get what matters in tech, in your inbox every morning. Sign up for Source Code.


The cascading effects on benefits systems, many of which are based on outmoded technologies designed for much lighter traffic loads, have been devastating. The problems could have been avoided — but experts say getting organizations of all kinds to embrace better technologies before it's too late can be challenging.

Major slowdowns

In some areas of New York, claims were up over 1,000%, according to a representative of the state's department of labor. As a result, the state's site has crashed for people looking to apply for benefits.

"To address this unprecedented increase, we have added server capacity, increased bandwidth, extended phone hours, and dedicated more than 700 staff members to address the influx," Deanna Cohen, the deputy director of communications at the department, told Protocol. From March 16 to 21, the state's site had 2,270,300 web hits and saw around a 400% increase in logins on some days, Cohen said. "Our representatives are doing the best they can to make sure everyone is served, even if it takes longer than usual," she added.

In Ohio, jobless claims shot up from under 5,000 to nearly 140,000 in a single week, and the state's system buckled under the pressure, and briefly went completely offline on March 23. "During previous downturns in the economy, claims came in waves as the recession worsened and industries began to shut down, whereas these claims came in all at once," Bret Crow, a representative for Ohio's Department of Job and Family Services, told Protocol. "This amount of claims in this short expanse of time would tax any online system."

Crow said the state has been "working around the clock" to add servers to deal with the additional traffic. The state's current unemployment website was built in 2004, back at a time before modern smartphones, and when a large slice of the American population still got their internet from AOL. Crow added that Ohio is working with Sagitec Solutions (a leading provider of unemployment systems that Crow said did not build the original site) to modernize the state's unemployment system. He was not optimistic about the short-term prospects. "Given the complexity of UI benefits, it will take a minimum of 24 months to configure the system to match our law," Crow said.

Tens of thousands of new applicants in the U.S. had a similar effect on benefits sites across the country. Pennsylvania, Oregon and New Jersey, among other states, all saw their employment and labor sites slow or crash under the unprecedented demand. In New Jersey, the state's unemployment benefits site is asking applicants to log on at different times of the day based on the last four digits of their Social Security number to help mitigate the crush. In Colorado and New York, the state is requesting citizens to apply on different days depending on the first letter of their last name.

And the problem is global: Australia's MyGov, a site for accessing government services, slowed to a crawl last week after thousands more people than usual attempted to apply for unemployment benefits. The government initially claimed it was a DDoS attack. In the U.K., reports suggest that thousands of people have been left waiting in a digital queue to access the Universal Credits benefits system.

The problem of systems not built for scale

For the most part, the backbone of the internet has held up admirably under increased traffic, and many modern sites are built atop cloud services that allow them to scale with demand. But sites that are built on older, more rigid software and hardware, like many unemployment sites, are struggling under the pressure, experts say.

"Any piece of hardware or any piece of software has some maximum capacity," said Josh Chessman, a senior director analyst at Gartner who covers the IT industry. "That's just the reality of life, it doesn't matter if it's your phone or a server or a mainframe — there's a limit to what it can do."

When building web applications, Chessman said, you'll have a plan in mind: say you're expecting to get 1,000 visitors, you may build it to be able to maintain 2,000 visitors, to give yourself a little leeway. "The problem comes when you're in that position, but all of a sudden you go from X to 500 X overnight," he added.

And it's not just the front-end architecture of these websites that's buckling. The database systems managing people's private data — things that companies and governments have been reluctant to move to the cloud — are also straining under unintended use.

"Very often these back-end systems are running on technology that could be 30, 40 or 50 or more years old," Chessman said of these sites that often struggle in high-stress situations. "You've put something in place that works really well, and you put it in place in 1970 or 1980 and the problem is if it's really working well, nobody wants to upgrade it because (a) why?, and (b), it means downtime."

Outside the world of benefits, other suddenly essential services, including some that should be technologically adept, have been stung, as well.

U.K. retailer Ocado, one of the early success stories in online groceries, experienced pretty much the exact issue Chessman outlined. Demand for grocery deliveries has skyrocketed as fewer and fewer people are able to leave their homes. To deal with the sheer weight of customers attempting to book deliveries, Ocado put up a digital queue page, much like with the Universal Credits system, where thousands of customers waited for hours to potentially get groceries. The company compared the demand to a "denial-of-service attack," had to pause new customer signups, and even suspended service temporarily to get through the backlog.

"At some point … just adding more servers isn't making it faster because of something else," Chessman said. "It could be communication between servers, back-end stuff — that's another issue that can happen and cause problems."

IT inertia vs. digital transformation

Across the board, experts agree that while it's easy to fault organizations for not adopting newer, more resilient technologies, it's understandable that administrators haven't leaped to upgrade systems that are fundamentally working most of the time. But in light of their current failures, it's possible to fix the flaws, given time, willingness — and cash.

"Things change. It's not like these systems were built poorly 10, 15, 20 years ago — they were just built [with] 10, 15, 20 years ago's knowledge," said Peter Klayman, head of strategy at Bottle Rocket, an agency that works with companies on digital transformation. "Now, if you know that you have cloud capacity, it changes the way our enterprise architecture practice approaches problems because they're able to architect it for scale."

Some companies have leaned into the digital transformation that has been taking place over the last decade, and are better-suited to be flexible in pressing times, Klayman said. He pointed to fast-food restaurants like Chick-Fil-A and Burger King, as well as sites like Netflix, which built their systems on cloud platforms that will allow them to scale as demand does.

But for companies looking to offer some new mobile solution — such as curbside pickup of food — or build out a robust digital experience for thousands of new users, executives should be wary of what's really going to be possible. Chick-Fil-A's app was in continual development for over three years, Klayman said, and isn't something you can just throw together. "You can't just stand that stuff up right now," he said.

Even businesses that want to take advantage of the coronavirus pandemic — such as food and delivery companies looking to add curbside drop-offs or scheduled deliveries into their apps — are going to likely struggle to get anything done quickly. "Everyone's running around with their hair on fire," Klayman said. "It's very difficult to get the appropriate number of stakeholders together to actually launch a new enterprise endeavor."

"That's quite a lot of logistical undertaking and change management to do at a time when everyone is stressed, when people are concerned about their jobs, when people are concerned about their health," Klayman added.

"If you are on these legacy applications, even small changes, it takes our clients months," Wipro Digital President Rajan Kohli told Protocol. In some companies it can take six approvals in six months just to get a small change made, Kohli said. "That's how complex the environment is: It's not that the people are slow — it is because it touches six systems on four different release cycles, and by the end of it, they have to wait six months to just to get that small change in."

But after the virus has started to recede and life starts to return to normal, should companies begin to invest in more durable, flexible systems? It really depends on what they do.


Get in touch with us: Share information securely with Protocol via encrypted Signal or WhatsApp message, at 415-214-4715 or through our anonymous SecureDrop.


"It's almost always a case of money can fix it, but it doesn't make sense to spend that money upfront or deal with it afterwards — or hope it never happens," Chessman said. But not every business is likely to see a cost-benefit analysis where the likelihood of another pandemic happening anytime soon means they should invest in their systems.

"I wouldn't say they should because why spend the money on something that you might need once every five years, 10 years versus an Amazon or somebody like that who is pretty critical to the country right now," Chessman said.

People

Beeper built the universal messaging app the world needed

It's an app for all your social apps. And part of an entirely new way to think about chat.

Beeper is an app for all your messaging apps, including the hard-to-access ones.

Image: Beeper

Eric Migicovsky likes to tinker. And the former CEO of Pebble — he's now a partner at Y Combinator — knows a thing or two about messaging. "You remember on the Pebble," he asked me, "how we had this microphone, and on Android you could reply to all kinds of messages?" Migicovsky liked that feature, and he especially liked that it didn't care which app you used. Android-using Pebble wearers could speak their replies to texts, Messenger chats, almost any notification that popped up.

That kind of universal, non-siloed approach to messaging appealed to Migicovsky, and it didn't really exist anywhere else. "Remember Trillian from back in the day?" he asked, somewhat wistfully. "Or Adium?" They were the gold-standard of universal messaging apps; users could log in to their AIM, MSN, GChat and Yahoo accounts, and chat with everyone in one place.

Keep Reading Show less
David Pierce

David Pierce ( @pierce) is Protocol's editor at large. Prior to joining Protocol, he was a columnist at The Wall Street Journal, a senior writer with Wired, and deputy editor at The Verge. He owns all the phones.

Politics

In 2020, COVID-19 derailed the privacy debate

From biometric monitoring to unregulated contact tracing, the crisis opened up new privacy vulnerabilities that regulators did little to address.

Albert Fox Cahn, executive director of the Surveillance Technology Oversight Project, says the COVID-19 pandemic has become a "cash grab" for surveillance tech companies.

Photo: Lianhao Qu/Unsplash

As the coronavirus began its inexorable spread across the United States last spring, Adam Schwartz, senior staff attorney at the Electronic Frontier Foundation, worried the virus would bring with it another scourge: mass surveillance.

"A lot of really bad ideas were being advanced here in the U.S. and a lot of really bad ideas were being actually implemented in foreign countries," Schwartz said.

Keep Reading Show less
Issie Lapowsky
Issie Lapowsky (@issielapowsky) is a senior reporter at Protocol, covering the intersection of technology, politics, and national affairs. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University’s Center for Publishing on how tech giants have affected publishing. Email Issie.
People

The year our personal lives took center stage at work

2020's blurring of professional and personal boundaries exacerbated disparities, humanized leaders and put personal values front and center.

In 2020, the personal and the professional became inextricable at work.

Photo: Tom Werner/Getty Images

For those of us lucky enough to keep our jobs and privileged enough to be able to work from home, our whole selves were bared at work this year. Our homes and faces were blown up for virtual inspection. Our children's demands and crises filled our working hours, and our working mothers became schoolteachers and housewives, whether they wanted to or not. Our illnesses became vital public information, and our tragedies shared. Our work lives ate into our social lives until there was no boundary between them.

In 2020, the personal and the professional became inextricable at work. Remote work might be the most sexy 2020 trend, but for the CEOs and leaders I spoke with, the de-professionalization of work could be the most important effect on a personal level. It's the one that has caused the most harm to women in the workplace and destroyed work-life balance for basically everyone. It's also what has contributed to the majority of work-from-home Americans being more satisfied with their work lives than they were before, mostly because they feel more connected to their families, they're able to set their own schedules and they're more comfortable at home, according to a Morning Consult poll. While we can't know exactly how many and who will be going back to the office just yet, as long as there is some kind of flexible work schedule, people's personal lives will be part of their work lives and vice versa.

Keep Reading Show less
Anna Kramer

Anna Kramer is a reporter at Protocol (@ anna_c_kramer), where she helps write and produce Source Code, Protocol's daily newsletter. Prior to joining the team, she covered tech and small business for the San Francisco Chronicle and privacy for Bloomberg Law. She is a recent graduate of Brown University, where she studied International Relations and Arabic and wrote her senior thesis about surveillance tools and technological development in the Middle East.

Protocol | Enterprise

How Christian Klein’s reboot of SAP’s strategy is working out

The pandemic wasn't kind to the company. But the way it's working with the major COVID-19 vaccine makers is a model for what comes next.

Christian Klein became SAP's sole CEO in April.

Photo: Picture Alliance/Getty Images

Christian Klein took over as SAP's sole CEO in April. It wasn't an ideal time to take the helm of an organization that sells expensive enterprise software.

As the spread of COVID-19 forced corporations everywhere to cut costs, one of the first places they looked was IT budgets. Specifically, companies around the world trimmed spending on back-end products, such as those offered by SAP, many of which still run via on-premise data centers.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

Politics

Congress is unleashing the FTC on COVID-19 scammers

The second stimulus package will expand the FTC's authority to penalize companies promoting fake information and faulty products.

The FTC will finally have the authority to immediately fine scammers touting fake COVID-19 treatments and lies about pandemic government benefits.

Photo: Al Drago/Getty Images

The FTC will finally have the authority to fine scammers touting fake COVID-19 treatments and lies about pandemic government benefits, thanks to new language in the economic stimulus package set to pass on Monday.

COVID-related scams proliferating online have overwhelmed government agencies and cost Americans more than $145 million since the beginning of the pandemic — and the FTC has been almost powerless to stop them, thanks to limitations on the agency's authority.

Keep Reading Show less
Emily Birnbaum

Emily Birnbaum ( @birnbaum_e) is a tech policy reporter with Protocol. Her coverage focuses on the U.S. government's attempts to regulate one of the most powerful industries in the world, with a focus on antitrust, privacy and politics. Previously, she worked as a tech policy reporter with The Hill after spending several months as a breaking news reporter. She is a Bethesda, Maryland native and proud Kenyon College alumna.

Latest Stories