Ransomware victims often find themselves in what feels like an impossible position: There are endless decisions to be made, but it's a race against the clock to save computer systems and data.

That's what New Orleans city officials learned firsthand in the early morning hours on a Friday in December. At 6 a.m., there was a blip of suspicious activity on the city's cybersecurity alert system — not a definite sign of attack, but an early warning of what would come, the city's chief information officer Kim Walker LaGrue recounted in an interview with Protocol. At about 9 a.m., when many city employees started arriving to work, there was an uptick in reports of unusual behavior on their computers.

Sign up for Protocol newsletters

Get what matters in tech, in your inbox every morning. Sign up for Source Code.

By 11 a.m., the diagnosis was clear: Ransomware was spreading through the network. The malicious code was encrypting devices and information, rendering them inaccessible unless city officials decided to contact the attackers. Presumably, they would then get instructions about paying a ransom.

City officials had a game plan in place to guide them through some of the toughest decisions ransomware victims face — the biggest being whether or not to pay the demand. The Louisiana state government and three of its school districts were hit by two separate ransomware attacks in 2019, so New Orleans officials had prepared and rehearsed an incident response plan.

The first step LaGrue took — before she even consulted with the mayor — was to disconnect all city systems from the internet. More than 3,000 computers and 400 servers were shut down to prevent the malicious code from spreading, she said.

"We sounded every type of alarm we could," LaGrue said. The city declared a state of emergency within about 30 minutes of identifying the attack, she added. Within an hour, the FBI and other state and federal agencies were on the ground to help respond to the attack. Normal lines of communication — including the city's email system — were inaccessible, and a war room was set up to help relay information.

City officials knew early on that they would not engage with the attackers and that no ransom would be paid. The U.S. Conference of Mayors, of which New Orleans Mayor LaToya Cantrell is a member, last year adopted a resolution opposing such payments after cities including Baltimore and Atlanta were hit by ransomware attacks. Additionally, the FBI discourages organizations from paying ransomware demands, arguing that it incentivizes crime, emboldens attackers and is sometimes futile — attackers on occasion don't decrypt the data after a payment is made.

Because it didn't pay the attackers, New Orleans is still recovering; LaGrue estimates that it will take another four months to make a full recovery. So far, her staff has focused on restoring key priorities. The first step was inspecting and cleaning thousands of government computers, and removing ones that were too old to run up-to-date software, she said. Over the last month, the city has focused on restoring data from backups, as well as services including email and printing. Over the last few weeks, the city's IT staff has been particularly focused on assisting the city's infrastructure and public safety agencies in their preparations for Mardi Gras on Tuesday.

The financial impact of the attack is still uncertain. A city spokesperson said that the current estimated cost of the attack is $7.2 million, adding that he "wouldn't be shocked if it got higher." The city has a $3 million cyber insurance policy that will help cover some of the costs, which include replacing between 500 and 800 computers as a result of the incident, LaGrue said. The city is planning to add three cybersecurity specialists to its IT staff, which currently includes 65 workers, she said.

Ransomware has been a rapidly growing problem for all types of organizations, including Fortune 500 firms and small nonprofits. The antimalware firm Malwarebytes said in August that it observed a roughly 365% spike in ransomware detections between 2018 and 2019. Municipalities, educational institutions and health care organizations are prime targets for ransomware attacks due to their lack of spending on security and outdated hardware and software, according to the report.

Because ransomware affects a range of organizations, the decision to pay can't be one-size-fits-all, according to Chris Hallenbeck, CISO for the Americas at cybersecurity firm Tanium.

"In an ideal world, no one pays the ransom," he said. "It's feeding the beast and creating an ecosystem that encourages criminals. But the reality of many organizations is that every day they're down there's massive impacts to the services they provide."

Overall, about a third of companies pay the ransom, according to security firm Emisisoft.

Which brings us to Maastricht University in the Netherlands, a recent example of an organization that decided to pay a demand after falling victim to ransomware. On Dec. 30, the university decided to pay 30 bitcoin, or about $305,000, to regain access to its systems and data.

Several factors made the attack particularly disruptive, and increased the university's incentives to pay the demand, according to a forensic report from FOX-IT, the security firm that assisted with the incident. A key factor was that attackers were able to encrypt backups for some of the university's critical systems. That meant that, without obtaining a decryption key from the attackers, the university would have to rebuild its infected systems from scratch — a process that would stall the work of students and researchers for many months. Additionally, unlike New Orleans, Maastricht University did not carry cyber insurance to help cover the costs of lengthy disruptions.

The attacker had access to the university's infrastructure for months and studied it to figure out how to maximize damage. They first gained access through a phishing email on Oct. 15 that instructed someone in the university community to sign an attached document and scan it back to the sender. Upon clicking the attachment, malware was installed that gave the attacker administrative access to the device.

On Nov. 21, while scanning the university's network, the attacker found a server with missing security updates and used it to obtain full access to the organization's infrastructure. On Dec. 23, the attacker deployed the Clop ransomware variant on 267 servers. The timing was particularly challenging for the university: Most faculty, students and support staff were away for the holiday break. Employees from IT, facilities, communications and a range of other departments had to skip part of their holiday to respond to the emergency, according to a spokesperson. One week after the ransomware was installed, the university decided to pay the ransom.

Nick Bos, vice president of the university's executive board, said in the report that the decision to pay the demand was a "devil's bargain": The university would have to live with the ethical issues that come from paying criminals in order to resume operations, research and studies.

At a symposium this month attended by students and faculty, Bos said the risks of losing the data were so great that the university did not attempt to negotiate with the attackers; administrators wanted the process to be as calm as possible, he said.

Bos said he was confident that paying the demand was the right choice. About five weeks after paying the demand, students and faculty were able to access almost all of the university's technologies, as well as attend classes and take exams.

"We felt, in consultation with our management and our supervisory bodies, that we could not make any other responsible choice when considering the interests of our students and staff," Bos said in the report.