When COVID-19 hit the U.S. in March, businesses of all kinds were devastated, but grocery was arguably among the hardest-hit. The technology that gave grocery a thin but continuous profit margin for decades imploded under the weight of a global pandemic. The fact that the virus was first discovered in China and that China has become the global supplier for U.S. grocery, well, that didn't help.

The COVID situation befell grocers in two very distinct waves: The first wave was an overwhelming soaring of orders, driven by scared consumers who wanted to stock up and be ready. And the second wave happened a few weeks later, when all of those new sales — and a chunk of the old ones—disappeared

"COVID-19 was the equivalent of Thanksgiving happening every day for two weeks straight, with no predictability," said Stefan Kalb, the CEO of Shelf Engine, a demand forecasting firm that works with major grocery chains. "Produce was up four times, bakery seven times. When COVID hit, grocers thought, 'Oh, my! We're going to be rich.' Then, all of a sudden, the sales plummeted, all at the same time. Sales dropped below what the sales were before COVID."

"This is one of these black swan events that nobody could have foreseen," Christian Beckner, the National Retail Federation's senior director of retail technology and cybersecurity, told Protocol.

All of the software that was supposed to predict what shoppers wanted and all of the supply-chain systems that were designed to keep orders fulfilled stopped doing what they were supposed to do.

"The grocery industry is notoriously tech averse. COVID exposed all the cracks in the system," Arik Keller, CEO of wholesale ordering firm Mable, said. Although COVID "gave many large stores and distributors record sales, it came at the expense of the businesses getting burned out from trying to keep up."

That situation was made far worse as human nature kicked in. In short, human managers didn't believe the recommended orders and they overrode the systems.

"It wasn't so much that systems failed as much as they were overwhelmed and, in some cases early on, the humans didn't trust the systems," Greg Buzek, president of retail IT consulting firm IHL Group, said. "We had situations where the computer aided ordering system initiated an order for 40x the volume of a key category — say toilet paper or wipes — and a human being cancelled that order."

Intersecting nightmares

That wasn't the only instance where human nature interfered. Senior management at many grocery chains initially didn't grasp the enormity of the disaster and chose to believe some rosier estimations that COVID would go away quickly. Rather than start preparing for what could be a one- to two-year ordeal, many of these executives Protocol spoke with chose to believe it would only last two or three weeks.

From an IT perspective, grocery businesses have been a huge champion of just-in-time inventory. That worked wonderfully until COVID obliterated the supply chain and all manner of other technology that made it work. Grocers then started overordering. Indeed, they became so uncertain about the supply chain that they ordered as much as what they could and hoped it would all work out later. That's when it became clear that grocers — who Buzek said "spend far less on IT than any other industry, as a percentage of revenue" — had an ideal tech infrastructure for ideal conditions, but one that proved especially fragile when the unexpected happened.

"The entire grocery supply chain is still suffering due to COVID," Mable's Keller said. "This industry is run on 30-plus year-old, asynchronous platforms that don't talk to each other." Keller added that this leads to a replenishment process that works for large grocery chains, but less so for everyone else. "Integration between distributors, brands, and stores, to help determine order sizes, inventory and forecasting, don't really exist," he said.

"Most CAO [computer assisted ordering] systems are glorified trend line forecasts," Kalb said. "When there is high volatility in sales patterns, they will always fail."

The two weeks of high traffic quickly morphed into far less traffic as news reports across the country reported soaring infection rates and death counts. As shoppers were unsurprisingly afraid to enter the stores, grocery quickly looked to new distribution methods, such as curbside pickup and delivery, fulfilled through existing delivery services such as Instacart and DoorDash.

That transition, initially, did not go smoothly. Although grocery executives carefully prepared for the changes needed to properly implement curbside pickup, such as having personnel pick the groceries and pack the bags, many didn't think through the payment process. Even when the orders were from the identical customers, purchasing the same groceries from the same store with the same payment methods, moving the purchases from inside to just a few feet outside racked up massive increases in payment card charges.

To use payments industry jargon, the charges changed from card-present (CP) to the much more expensive card-not-present (CNP). To the grocery chains, it quickly added millions of dollars of charges that few of them had budgeted. The difference is the most striking with mobile payments: For both Apple Pay and Google Pay, using a mobile device to pay in-store means leveraging a card reader's NFC capabilities and it's charged at the lower CP rate, but if that same customer uses that same mobile payment to pre-pay a grocery pickup, it jumps to the higher CNP rate.

The only positive for grocers about curbside and delivery was that they could sparingly move less desirable produce. Typically, a small percentage of produce in-store won't sell, such as visibly over-ripe bananas or overly-bruised apples. Now that the stores were selecting and packing their own produce, they could periodically include some of the less perfect items. In a pandemic, consumers are less likely to complain about such things, especially if most of the produce looks good. That is not expected to last long.

Delivery also brought with it surprises. There are all of the normal issues that crop up in dealing with a third-party, such as who owns the data and who can pursue customers. But far worse, inventory and personnel shortages made those first weeks of deliveries almost impossible for many customers to place orders. Again, the designers of these systems never envisioned something as massive as COVID hitting all stores everywhere this hard.

Aaron Hageman, the CEO of Delivery Drivers, which provides delivery drivers to grocery chains, said the software wasn't designed for demand this high.

"The reservation systems were often not even being able to reserve a slot eight days out. They didn't have the technology to backlog orders that far. Historically, we rarely have grocery deliveries scheduled weeks out and these challenges became bottlenecks," Hageman said, adding that systems would simply say that no delivery was available. "The technology wasn't communicating and it wouldn't say when delivery would be available. I would have happily ordered groceries two weeks out if it would have let me."

The labor situation at the same time was a balancing act: Employees and potential hires were concerned about being inside with so many people, which made hiring more difficult. This was the case, even though the economy was cratering and unemployment mushroomed, which theoretically should have made mass hiring easier. Spiraling all around these grocers were other implications of a global pandemic wrestling with a global economy.

Jonathan Wright, an IBM managing partner who specializes in supply chain issues, saw an intersection of global nightmares at the start of COVID.

"When COVID first hit and Wuhan went into lockdown, we started to understand what it was like to have supply shocks. Across industries, organizations did not fully understand the extent their supply chains were linked to China and therefore vulnerable to supply shocks," Wright said. "Then as states and nations went into lockdown in March, we felt what it was like to have demand shocks. For the first time ever, we had significant demand and supply shocks at the same time.

"Looking back at historical data to forecast demand didn't work," Wright added. "Managing supply chains in reactive mode didn't work."

Another shock to the grocery nervous system was the budding move to e-commerce. Most grocery chains already had online ordering set up, but they accounted for a far smaller share of orders than what was ahead.

"Before COVID, online orders were relatively small (as a percent of sales) so you can afford those things to be relatively unprofitable," Jeff Orschell, the consumer retail leader for consulting firm EY, said. Online ordering sidestepped high-margin impulse purchases, food samples and other in-store-only enticements to buy.

How COVID reshaped the in-store grocery experience

The nature of COVID forced obvious changes in-store, with mask requirements and frequent cleanings. But attempts to enforce distancing in-store didn't generally succeed. "The limitations on the number of people in stores, they sounded impressive, but they didn't really take that many people out of the store," Orschell said.

Worse, shoppers tend to congregate in specific areas — often produce, dairy and meats — and even with a small number of people in the store, ten people crowding into a small section doesn't allow for social distancing. The only option is to have employees strictly monitor and order people to stay distanced, which few if any grocery chains could manage.

Hageman said that he was pleasantly surprised by Walmart's conduct, especially in COVID's early days. Concerned that truckers would be afraid to deliver to the most hard-hit areas, Walmart slowed down the truck loading process, instructing drivers to stay in their cabs while other workers unloaded the truck. Hageman said this was unusual, as truck drivers have historically helped unload.

Inside stores, hardly any U.S. grocery chains were using technology to help with social distancing. Samrat Sharma, a consultant specializing in retail at PwC, said he was unaware of any U.S. grocery chain that was using their existing mobile apps to help with contact tracing.

Sharma has, however, been watching mobile technology grocery trials in China and Singapore where "it beeps on the phone when (customers) get too close together." He said that making those social-distancing efforts work can be difficult. For example, a family shopping together would theoretically be able to violate social distancing rules as long as they form their own quarantine bubble, meaning that they are routinely near each other.

Even if the social-distancing apps work well, what are most grocers willing to do to enforce it? Just trying to enforce masks has provoked fistfights in the U.S., making grocers and other retailers hesitant to be aggressive so as to not endanger their employees.

Sharma noted that, from an IT standpoint, some good may indeed come out of this disaster. Grocery executives have traditionally "relied purely on historic data. In COVID, that failed," he said. "This has been a coming out party for data and analytics. Traditional insights have become not really relevant."

Where does grocery IT go from here? As COVID numbers soar again this winter and paint an unclear picture for 2021, grocery is likely to see a repeat of March's sales volatility. Lessons learned from March should make grocery chains better able to handle it, but much depends on the rest of the planet. If the global supply chain again can't keep up, chaos will likely return.

But technology and workforce improvements that started happening earlier this year for curbside and delivery will almost certainly take the edge off of some of that pain. Will IT alone save the day? Unlikely, but it will help.