Like many of you in college, I read Elisabeth Kübler-Ross’ Death and Dying about the various stages of grief. I think IT security managers go through similar stages when their networks have been breached by hackers and malware. There is the shock of the breach, then denial that their equipment was at fault, then anger at the hackers for having targeted their company. Eventually, everyone gets down to work trying to fix the problem, and finally accepts that it happened.

We have all seen what happens when IT staffs never make it past the denial stage: their networks remain in ruins for weeks or months. They overspend on consultants, they have to scrap and replace their servers, and they suffer tremendous business losses. Sony Entertainment, the City of Atlanta, Maersk shipping, the list is far too long.

I was thinking about these stages this week when I was at SaltConf18 listening to how users of SaltStack are deploying this tool to help them make their enterprises more secure. Some users have developed their own home-grown solutions, cobbling together various routines that provide for widespread patching across all of their systems. Others were eagerly learning about the news on how to deploy SecOps for SaltStack, which was announced this week and will be delivered next year. When the product was introduced, Mehul Revankar, a senior product manager for the SecOps product, spoke about how it took care of the various different stages of identification, remediation, creation of actionable content and being able to scale up well to protect the largest collections of servers and endpoints.

Just like in coping with the loss of a loved one, we have to figure out how to move through these stages constructively and productively. Getting unstuck is key. When it comes to people, we have psychoanalysis and supportive friends to help us through these dark times. But when it comes to protecting our computing infrastructure, we have to turn to better automation to help us through the response and remediation of our equipment. (Maybe there is a role for therapy, but I’ll put that aside for this blog post.)

Certainly, SecOps isn’t the first security tool to use automation, and it won’t be the last. Many vendors are moving into this territory, frankly because they don’t have a choice. When you have to patch ten thousand Linux or Windows servers because of a vulnerability, you can’t do the job manually. Oftentimes, the window of opportunity for such massive patching is a matter of hours or days before the first exploits start showing up in the wild. By now we all know what happened at Equifax last year when they delayed patching their Apache Struts servers. They were still stuck at the denial stage.

As First Data’s VP Amaya Souarez said in her keynote session at SaltConf18, “You can’t hire yourself out of this problem, we have to automate.”

A recent study of several dozen IT executives supports this need for better security automation. One was quoted in the study saying, “The future of security is [being] as autonomous as possible — where a combination of real-time, intelligent analytics, and integrated automation and remediation cover an ever-increasing part of manual investigative and response runbooks.”

That was the design goal of SecOps for SaltStack. The trick is being able to break down the process — going from recognition to remediation — in such a way that an automated tool can sequentially apply a series of security policies and rules to make the automation work under a wide variety of conditions. To be effective, automation has to deal with circumstances when a rule fails as well as when it succeeds. At this week’s conference, Justin McMillion and David Kleiner of Sunayu showed how they built their automated auditing tool. Their firm does a lot of work for the Department of Defense to help them keep their Linux servers up to date and within compliance of various DoD standards. They created some clever dashboards and routines using SaltCheck to do this, and mentioned during their session how they were envious of what they could have done if SecOps was available.

When I look at smaller-sized tech companies, I tend to judge them by the company that they keep. By that I mean who they partner with, who are their customers, and where are their products being used. By any of those metrics, SaltStack is in very good quarters indeed.

At the SaltConf18, we heard from several large customers using Salt to run some very sophisticated and complex networks, such as Cloudflare and IBM Cloud. Both companies run their infrastructure with just a few staffers, which is another testimonial to how powerful Salt can be in its automation and orchestration features.

Tom LeFebvre is a network engineer and was the presenter for Cloudflare. Cloudflare runs about a tenth of the total global Internet traffic across its infrastructure, and is used by some of the largest web properties to accelerate the delivery of their content. They manage more than seven thousand servers with Salt, located in more than 150 different data centers running more than 250 Salt Master copies.

They are deep users of Salt, and are constantly trying to improve their deployment to make it operate faster and more reliably. When you are connecting servers between China and the US, you have to keep network latencies and traffic to a minimum, especially as it has to traverse the Great Chinese Firewall.

Some of the things they have learned is to try to use packages rather than scripts to update server operating systems, and use highstate calls whenever possible to reduce the loads being placed on the Minions. They also developed a series of graphical dashboards that keeps track of the highstates and set up special alerts for help troubleshoot failed conditions or when Minions were consuming too much time to complete their tasks. They tied these conditions to notifications that were sent out to the staff via Google chat messages, which shows how easy it is to extend Salt with other services. They also rewrote some of their Pillars into pure Python, again to help increase performance. Finally, they are increasing the number of Masters deployed in each data center to handle their canary deployments, which means providing an early warning when something goes wrong with one of their massive system rollouts or upgrades.

Also presenting at the conference were an unlikely couple: Nathan Newton from IBM Cloud and Mike Wiebe from Cisco. The two have been active in working with SaltStack to modify its minions and other code to work with the giant network gear that IBM Cloud uses to run its global network. Newton spoke on how he has just 12 team members that runs their network and a large part of that efficiency is due to Salt. IBM Cloud has tens of thousands of Cisco NX-OS and Arista EOS network switches that are spread across 80 data centers around the world.

Again, what impressed me was how both men were working with SaltStack to extend the original premise of the product to handle the completely different context of network management, by having the Minions run directly on the Cisco gear. Newton said during one of his presentations, “IBM is good at building data centers, but once they are built the next day we need automation to take care of them.” That’s where they need help. They reached a tipping point last year where they were maintaining 60,000 different devices and “we couldn’t do it manually. We needed to be more proactive and have better automated tools.” That’s where Salt came into play. One of the reasons why they duo went with Salt was because of its event-driven automation, and the ability to cause particular actions and not just notify the team when something went wrong.

What impressed me most about both IBM and Cloudflare’s implementations was how willing they were to keep pushing Salt to do more and do it better. Both of them obviously believe in the product to trust it to be such a critical part of their network infrastructure.

We’ve all heard about how everything is going digital, using on-demand cloud-based services and mobile technology. But at a session at SaltConf18, we heard a very different perspective from Cyndi Tetro, the head of the non-profit Utah Women’s Technology Council and the CEO for ForgeDX.  She spoke at one of the conference sessions and painted a very exciting picture of how the physical world is really driving change and innovation in the digital universe.

Tetro spends time looking at this intersection and tracks products that have made inroads into improved business and customer experiences. For example, we all like to talk about the Internet of Things, but now some of us have dozens of connected devices that we use regularly. What about connected lawnmowers and sprinklers that can, Roomba-style, maintain your yard without your intervention? Or a coffee mug that can keep itself at a constant temperature and detect when it is filled with liquids? Analysts predict that by 2020, there will be 50 billion connected devices, and increasing exponentially from there.

Most of us when we visit one of the theme parks in Orlando or Southern California don’t really think about all the connected devices there, but Disney and other park operators are constantly thinking about how to improve the visitor experience. I got a chance many years ago to tour Disneyland with one of its network engineers: back then they could barely scratch the surface in terms of synchronizing the music with the Main Street Parade and putting in enough fiber to carry all their digital traffic. Since then, Disney engineers created their “World of Color” light show that depends on a variety of digital technologies to coordinate more than 18,000 elements such as lighting, water fountains and music.

World of Color was created more than a decade ago, and since then the entertainment company hasn’t stopped innovating. Disneyworld today offers its Magic Band that can be used to open your hotel room door, charge purchases at the park’s various gift shops, skipping lines for the rides, and personalizing your family’s experience at the park. These are all examples of what Tetro says are “finding the compelling real-world experiences and then using digital technology to make them possible.”

What about the 2016 Super Bowl, when Carolina Panthers All-Pro linebacker Thomas Davis ended up wearing a 3D-printed brace that was custom-made in a few hours? Again, the digital tech — the 3D printed object — literally made this player’s day, said Tetro. She also mentioned fitness clubs and gyms that are using connected technology so that groups of people can do their spin classes in separate cities or even in their own homes, connecting online and being led by a superstar instructor. Equinox is one of the pioneers in this technology.

Tesla is often used as an example of a connected car, but many other automakers are using technology in more mundane ways, for just-in-time assembly line methods. Wireless networks send specifications directly to the manufacturing machinery seconds before it is placed inside the vehicle. Tetro calls these “dynamic manufacturers.”

Finally, there is the smart or connected city. Tetro mentioned the NYC-based Hudson Yards development, which is being constructed literally over the railroad tracks just west of Pennsylvania Station. The sales center provides a complete digital experience so that buyers can look at video walls and see 360-degree views of what their apartment will provide.

Tetro says that “our job in technology is to make it disappear, even though it is a big part of whatever we are doing. Everything that you can touch is heading towards being infused with technology in some fashion, but the interesting points are how the digital and physical worlds interact with each other. We are just on the cusp of many advancements.” It was a fascinating and very fast-paced look at what our world is evolving into.