Like many of you in college, I read Elisabeth Kübler-Ross’ Death and Dying about the various stages of grief. I think IT security managers go through similar stages when their networks have been breached by hackers and malware. There is the shock of the breach, then denial that their equipment was at fault, then anger at the hackers for having targeted their company. Eventually, everyone gets down to work trying to fix the problem, and finally accepts that it happened.

We have all seen what happens when IT staffs never make it past the denial stage: their networks remain in ruins for weeks or months. They overspend on consultants, they have to scrap and replace their servers, and they suffer tremendous business losses. Sony Entertainment, the City of Atlanta, Maersk shipping, the list is far too long.

I was thinking about these stages this week when I was at SaltConf18 listening to how users of SaltStack are deploying this tool to help them make their enterprises more secure. Some users have developed their own home-grown solutions, cobbling together various routines that provide for widespread patching across all of their systems. Others were eagerly learning about the news on how to deploy SecOps for SaltStack, which was announced this week and will be delivered next year. When the product was introduced, Mehul Revankar, a senior product manager for the SecOps product, spoke about how it took care of the various different stages of identification, remediation, creation of actionable content and being able to scale up well to protect the largest collections of servers and endpoints.

Just like in coping with the loss of a loved one, we have to figure out how to move through these stages constructively and productively. Getting unstuck is key. When it comes to people, we have psychoanalysis and supportive friends to help us through these dark times. But when it comes to protecting our computing infrastructure, we have to turn to better automation to help us through the response and remediation of our equipment. (Maybe there is a role for therapy, but I’ll put that aside for this blog post.)

Certainly, SecOps isn’t the first security tool to use automation, and it won’t be the last. Many vendors are moving into this territory, frankly because they don’t have a choice. When you have to patch ten thousand Linux or Windows servers because of a vulnerability, you can’t do the job manually. Oftentimes, the window of opportunity for such massive patching is a matter of hours or days before the first exploits start showing up in the wild. By now we all know what happened at Equifax last year when they delayed patching their Apache Struts servers. They were still stuck at the denial stage.

As First Data’s VP Amaya Souarez said in her keynote session at SaltConf18, “You can’t hire yourself out of this problem, we have to automate.”

A recent study of several dozen IT executives supports this need for better security automation. One was quoted in the study saying, “The future of security is [being] as autonomous as possible — where a combination of real-time, intelligent analytics, and integrated automation and remediation cover an ever-increasing part of manual investigative and response runbooks.”

That was the design goal of SecOps for SaltStack. The trick is being able to break down the process — going from recognition to remediation — in such a way that an automated tool can sequentially apply a series of security policies and rules to make the automation work under a wide variety of conditions. To be effective, automation has to deal with circumstances when a rule fails as well as when it succeeds. At this week’s conference, Justin McMillion and David Kleiner of Sunayu showed how they built their automated auditing tool. Their firm does a lot of work for the Department of Defense to help them keep their Linux servers up to date and within compliance of various DoD standards. They created some clever dashboards and routines using SaltCheck to do this, and mentioned during their session how they were envious of what they could have done if SecOps was available.