Bryan Doerr has been involved with tech companies for decades, most recently leaving Savvis/Century Link as their CTO before agreeing help bootstrap Observable Networks. I asked him to reflect back on his career and where the infosec industry is headed in general. “There is a lot of security industry maturation still to come, a lot of wood left to chop,” he told me in a phone interview last week. “While there are still some pockets of maturity here and there, they usually are only found with the largest companies who can afford it.”

Looking back more than a decade, the biggest change has been being able to deliver security as a subscription service, he said. “First we had pre-built security appliances, but lately we have seen managed detection and response services,” such as what his company delivers. “And it isn’t just a change in how protection is delivered, but how the subscription service can be more affordable for mid-market customers.”

Another big change is how end user customers finally are getting some benefit from sharing threat intelligence. “No one wanted to talk about where or how they were attacked and share these specifics with anyone else,” he said. This intelligence sharing has made the subscription service vendors more potent and compelling and has boosted the ability to respond effectively to threats.

“Ten years ago security was built on a simple idea: that we know about our attackers and threats, and through some means we could prevent those bad guys from getting inside our networks. Back then, we had a limited number of threats, so we could more readily recognize and block them. That is so far from where we are today. The fundamental nature of what is a threat and how attacks use technology has changed completely. The idea of tracking attack signatures makes a lot less sense when every attack is unique.”

Doerr agrees that the days of the perimeter being the sole point of defense are also long over. As an example, he points out the recent IoT botnet attacks.

One benefit from the last decade has been the move towards increasing virtualization. “This absolutely was a positive influence, and helped us to better design and operate more secure systems and more complex infrastructure,” he said. Before virtualization, we had too many different fiefdoms dedicated to particular circumstances. Each one had different configurations and staffs who were maintaining them. All of that variation left us vulnerable.”

But with virtual machines, “a lot of automation has been brought to bear to keep a consistent environment running. That means we can provision VMs, kill them off, and recreate them easily. This makes it more efficient to scale up and down and we don’t have to spend our time patching systems.”

Another issue is the nature of modern network traffic. “Our networks are becoming increasingly encrypted, we can’t even see what is going on over the wire and view the payloads, and this adds another layer of difficulty. Right now less than half of all traffic is encrypted, but it won’t be long before it becomes 100%. We won’t be able to readily examine any of this traffic, which will make networks harder to defend and detect exploits.”

When he was at Savvis, one memorable experience was upgrading one of their data centers. Thanks to a routing bug the entire data center couldn’t come back online. “We tripped over it on a Saturday, and didn’t immediately understand what we were doing. It was easy to miss a single use case that caused the problem. That was a humbling experience and gave me an appreciation of the magnitude of the business that we had running. You don’t feel it until something terrible happens and you see how significant these outages are.” The situation drove home the point that he needed to stay in touch with his technology and understand that it is not just an abstraction, but also a very real entity.

I asked him who had the better job, the CTO or the CIO? He was firmly behind the CTO position. “CTOs will have jobs for forever, because they help organizations understand the evolution of technology and anticipate the direction of that evolution. The CIOs still have some soul searching to do.”

David O’Berry was a former CIO at a state agency with 1000 employees, now he works for a security vendor. To give you an idea of his credentials, he has CISSP‑ISSAP, ISSMP, CCSP, CRISC, CSSLP, MCNE, CSPM and a CRMP!

He met his wife in college when a virus erased his senior thesis text and backups: luckily she was both a fast typist and a good sport. “That was by far the most expensive virus of my entire career!” Later on he had to attack another floppy-based virus, which was difficult because he had to run around the office finding infected disks and literally destroying them. He also faced down the Nachia/Welchia worm, which infected a PC that was not patched because the user was out on maternity leave.

“When I was a CIO, imaging software probably saved us the most time and had the strongest impact initially along with mail filtering products and endpoint management tools for remote control. Besides these products, I believe that standardization of what we did and how we did it had the single largest impact on our organization being able to progress as rapidly as we did with as limited resources as we had.”

For fighting insider threats, “you have to have contextually aware DLP and scanning products as well as what I call “Distributed Peer Review” by the nodes that attach to the environment.  Each node has to contribute to the survival of the organism by being a sensor in the larger scheme of things.” He has seen plenty of ransomware, and feels that “first and foremost it is a test of backup and recovery plans. Having a known-state in that area fell out of vogue for a while but now it is more important than ever even if it seems like boring blocking and tackle.”

At his current employer, “we do use MDM and they also allow BYOD. As a former CIO, we had not adopted BYOD when I left but had made the entire workforce mobile and managed it accordingly. We also had implemented Imprivata for its single sign-on package.”

When it comes to securing the cloud and his cloud-based servers, “there are similar challenges to what we have been pursuing since the dawn of time. Visibility is king.  Constructs that give you real-time visibility give you the edge over any other type of product when coupled with real-time mitigation and resilience.”

Now that he is on the vendor side, “I would say that the state of cybersecurity has gotten a lot worse since I made the jump because the pace of innovation and change has hit a vertical level and never stops. Malware creators have become more and more adept at how to attack the exploding number of devices. I believe we have a chance to get out in front of the next phase of this is, but to do so we have to share information in real-time as well as allow companies to participate without artificial barriers to entry. However, our window of opportunity is closing rapidly.”

With all the changes to infosec technology, here is a not-so-outrageous idea: maybe you should take a page from the US Secret Service playbook in how you run your IT security department. Actually, this idea didn’t come from me, but from someone who actually is familiar with both roles. Nathaniel Gleicher is trained as a computer scientist and a lawyer, and currently is the Head of Cybersecurity Strategy at Illumio, a security vendor. Previously, he prosecuted cybercrime at the US DOJ and served as Director for Cybersecurity Policy at the White House National Security Council. While he worked at the White House, he saw multiple data breaches. “Every breach relies on lateral movement, and instead of attackers being at risk once they get inside, they’re able to take all the time that they need to identify high value information and cause damage.”

He thinks organizations need to take a different, simplified approach and go back to the basics: get visibility inside the data center and cloud and then be able to truly lock the doors inside.

In a blog post for his firm, he writes: “Like the Secret Service, cybersecurity defenders face a similar problem: they are defending high-value assets that must be protected, but also have to speak to hundreds or thousands of other servers. You have to have visibility, and reduce your attack surface, and focus on the security consequences for your most valuable assets. Shutting down the attack surface constrains attackers, makes lateral movement harder, forces attackers to risk exposure, and makes other security tools more effective.”

Sadly, most organizations focus their cybersecurity spend today at the perimeter, making no effort to secure or even understand the interior of their data centers. After reading Gleicher’s post, I asked him if there is a difference between interior and exterior networks any longer. He told me in a phone interview, “Everything is a potential threat. One difference is that you can have greater control around an interior network. And your network visibility is much more limited with exterior ones. But that’s missing the point. An intruder can find something once they are inside your network and can look around. Organizations are trying to layer defenses at the fortress wall, while the cyber attackers are parachuting inside and then free to move around as they want inside the data center and cloud.”

He continued, “I still have conversations with CISOs that don’t know how their devices are connected to their networks. And I don’t mean just a list of these devices, but how they are related to each other, both logically and operationally. This is the kind of information that attackers can exploit.”

His work with the Secret Service has him focused on understanding some of these lessons from providing physical security to protect the President. “People don’t see the Secret Service advance work that was done months before any presidential visit. They had to map the location and understand the physical space. The same is true for cybersecurity, because we need to identify the attacker quickly and respond fast too. This means that any cybersecurity effort should start months before any potential attacker actually shows up.” In other words, it isn’t just about stopping someone from getting across the White House fence, but understanding what will happen once then enter the grounds and what they might end up doing.

He agrees that good security isn’t easy. And he started early in his career with his first IT job for the Peace Corps. There he created a created a campus-wide network to connect 85 machines that were located in the different buildings of a college on a Caribbean island. Less than five minutes after it was first connected to the Internet it was breached. It took him several tries to close various ports and other vulnerabilities before he could defend the network properly. “This was an early lesson on how hard it was to do security properly: there are way more people trying to get in than keeping them out. It also showed me that the steps to strengthen data security aren’t rocket science and are very straightforward. It is a lot more how to orchestrate them and use them efficiently across the enterprise.”

Instead of focusing on the lack of response, he says we should be doing a better job of evaluating the highest-value targets, which is another lesson he learned from watching the Secret Service in action. He said, “You shouldn’t be in the business of protecting the app that handles your employee’s lunch request.” And not everything in the data center should be treated equally, too. “There are some things in your data center that are more valuable and you have to focus on what needs the most protection. If a burglar gets into your house and gets into your basement that is different from him getting into your bedroom where you keep your jewelry.”