The story of how the city of Atlanta reacted against a ransomware attack at the end of March 2018 is instructive both in terms of what not to do and how expensive such an attack can become. The city actually experienced two separate attacks, one that began March 22 and another on April 5. This is just part of an overall trend where ransomware is on the rise. The Verizon Data Breach Investigations Report for 2018 says that ransomware has “overtaken all other forms of malware to be the most prevalent variety of malicious code for” 2017, and 2018 doesn’t look very different.

The first attack took down a number of city services, including online bill paying, the water department and court systems. Some law enforcement officers had to write their reports by hand. However, the city was able to make their municipal payroll and their city-owned airport continued with uninterrupted operations. The city was asked to pay $51,000 in ransom. A second attack hit the water department website again, according to Reuters. “I just want to make the point that this is much bigger than a ransomware attack,” said Keisha Lance Bottoms, the mayor of Atlanta. “This is really an attack on our government, which means it’s an attack on all of us. We are dealing with a hostage situation,” she said. But the mayor also admitted that cybersecurity wasn’t initially a high priority for her or the city, although it is now.

The first attack was based on the SamSam malware. CSOonline has details about the ransom notes and how they were tied to this particular malware strain. SamSam ransomware differs from other ransomware because the attackers don’t rely on user-based attack vectors, such as phishing campaigns. Instead, they use compromised hosts to gain a foothold and then move laterally through the network, taking their time to analyze weaknesses and points of leverage. This type of malware also hit the Colorado state Department of Transportation, which was able to restore its systems without paying any ransom. But then it was hit with a second attack a week later.

It seemed the Atlanta city government refused to pay based on subsequently events, when they hired a series of consultants to help fix things. Eventually, they will have spent more than $2 million in contracts with various consultants such as E&Y, Secureworks, Microsoft, CDW and others that the city has listed on its website.

Their first mistake was not heeding any early warnings about how ill-prepared they were, according to this report from a local TV station. Fixes were planned for the spring of 2018 but unfortunately not completed before the attacks happened.

But they compounded this mistake with a lot of sloppy IT work. One of the issues for Atlanta was how exposed it was. The city had open Windows RDP ports with no multi-factor authentication protection and also had open SMB shares and FTP servers too, making them very easy to access and infect. Rendition Infosec documents these issues in a blog post here. These consultants had found the infamous NSA-based DoublePulsar malware on several city computers last year — computers that weren’t patched for several weeks after their owners were notified. These delays in patching were one of the big reasons why the footprint of the ransomware was so large, and so difficult to contain.

Certainly, Atlanta isn’t the only city which has poorly prepared for potential attacks. A recent survey of municipal  IT workers found that most of them don’t know how frequently they are under attack, can’t determine who the attackers are and don’t even keep track of them when these attacks happen. The survey found that almost half of the respondents experience daily cyberattacks, and the researchers think this is even conservative. They conclude, “If local officials are going to do a better job protecting their information assets, they’ll first need to know a lot more about what’s actually happening.”

What can we learn from Atlanta? Lax security, delayed patching, sparse backups, lots of open ports for hackers to access all led to the inevitable. These are some of the reasons why getting its online sites up and running took them weeks, if not months. Ultimately, Atlanta IT needs to change their culture to fix these common mistakes and be more attentive.

But Atlanta was also behind the times when it comes to having top-shelf protection solutions. Reviewing whom they have paid since the breach, the city has purchased multiple protection solutions, including Forescout, temporary staffing, incident response services, and Duo authentication tools. That’s great but they should have been using these tools from the get-go.

Should they have paid the ransom? It is tempting to pay, particularly when you think (mistakenly, in the case of Atlanta) that your backups are fine. Yes, the economics of paying can be a better than the costs and consequences of trying to fix things yourself – if you are confident that the payment will actually result in decrypting your data and returning your systems back to a working state.

But that doesn’t always work, especially when you realize that your backups aren’t adequate. A business can still have disruptions, as we have seen with the aftermath of the Atlanta cleanup stretching well into the summer. And remember that you are dealing with criminals, who don’t necessarily have to give you anything in return for your ransom payment.  There is no guarantee that you will get your files decrypted, either. “Organizations should never have to think if paying the ransom is a better way out than restoring data compromised by ransomware,” says Rick Vanover, the director of product strategy for Veeam Software.

Finally, you need to vet your backup and recovery procedures, to make sure that they actually protect your data. “Organizations must have confidence in their backup architecture. It has to be resilient against threats such as ransomware today,” Vanover says. Atlanta never truly tested their recovery processes until it was too late. “One way to look at this is to pay now or pay later. Pay now to be resilient. Pay later to document that proper preparation was not in place.”