I spent last week visiting a data center tucked into an anonymous office park in Champaign, Ill. The data center is operated by Amdocs, a company that makes its money doing managed back office applications for telecom companies, such as Sprint, Metro PCS, and others. The visit was part of a general press briefing about what Amdocs is doing, but the term “single point of failure” kept coming up.

If you are going to host apps for telecom vendors, you have to know what you are doing in terms of providing uptime. You need redundant everything, from the plug that a router connects to for power to the backup of the backup diesel generator that has to fire up when you lose main AC power from the utility.

Actually, the most impressive part of the tour was the empty “situation rooms” that Amdocs has built. They are empty because there wasn’t any crisis going on – each room is dedicated to a particular customer and is where the account team gathers when they have a problem to work on. Think “24” but with far nerdier people. And that brings up a good point: what is the rest of CTU doing to protect the other 300 million of us that aren’t directly threatened by the current plot? All the action is happening on the main stage. But I digress.

I started thinking about other IT managers who haven’t completely thought through this issue that I have met down through the years.

There was one manager at a very large financial services firm near Washington DC that I interviewed a few years ago. Gazillions of dollars a day pass through its computer networks, and as you might imagine the firm had three Internet providers – not just two, but three – to provide connectivity. Each provider had a separate path and pole for their line from the firm’s server room. Well, that sounded all well and good until the day that a truck collision happened in the Baltimore Harbor Tunnel – a main north-south artery about 50 miles away. Trouble was all three of the Internet provider’s lines went through that tunnel and the firm was offline from the Internet until they got things re-routed. Now they have four Internet providers, and they got them to share their route maps (try doing this with yours, and good luck) to make sure there was no single point of failure.

Another time I was helping another firm in Florida upgrade one of its high-end network servers back in the late 1990s. This was a Tricord server, which took an ordinary Intel CPU and wrapped it around all sorts of redundant things: two power supplies, RAID hard drives, two physical processors, separate memory, and so forth. We had to pull and replace the network cards from this $40,000 server. This required powering down the beast and opening it up. Sadly, the one thing that wasn’t redundant was the physical power plug that went from the server into the wall – and the $25 part that the ordinary plug fit into went south when we powered the unit down. It took a few white-knuckle hours to locate a new part and get it over to us before we could bring the Tricord up again. I bet no one thought that probably the least sophisticated part in the whole machine was going to fail.

These days, you see lots of gear that have two physical power plugs, and at Amdocs’ data center they have two separate power paths just in case one goes out. That means taking that path back to a generator and line conditioning gear too.

Here is a story from my own mistakes, lest you think I am just harping on my subjects here. Several years ago, I was running this email list server on a friend’s Linux server that was in his California basement. The friend is one of the original Internet heavyweights, and knows his systems and has plenty of backups. However, the day came when a lot of flooding in his area knocked out all of his Internet connections, and I wasn’t able to access my list. Well, I thought I had all sorts of backup procedures in place and had saved copies of the server list configuration, so I could bring it up on someone else’s server. However, I had neglected to do one simple task – make a copy of the names of everyone on my list. Now I do. You would think something this simple would not have eluded me but you would think wrong.

So single point of failure: it is easier to say than to do. And when you see what Amdocs had to do to deliver on this maxim, you would be impressed.