A recent story in the NY Times caught my attention. It is about a block in the Cleveland area where some of their wireless car key fobs and garage door openers stopped working. The block is near a NASA research facility, so that was an obvious first suspect. But it wasn’t. The actual source of the problem turned out to be an inventor that was flooding the radio spectrum at the same frequency as the fobs use: 315 Mhz. Once the radio emitter was turned off, the fobs and garage doors started working again. The issue was the inventor’s radio signal was so strong it was preventing anything else from transmitting on that frequency. He had no idea that he was the source of the radio interference.

This story reminds me of an experience that I had back in 1991 or so. At the time, I was the editor-in-chief of Network Computing magazine for what was then called CMP. It was a fun and challenging job, and one day I got a call from one of my readers who was the IT manager for the American Red Cross headquarters in DC. This was Jon Arnold, who spent a long career in IT, sadly dying of a heart attack several years ago. Turns out they had a chapter in Norfolk Va. that was having networking issues. Their office was a small one, of about 25 or so staffers as I recall. Every day their network would start slowing down and then eventually go kaput for several hours. It was at a different time during the day, so it wasn’t the Cleaning Person Problem (I will get to that in a moment). It would come back online sometimes by itself, sometimes with a server reboot. The IT manager asked if I would be willing to lend and hand, and the first person that I thought of to help me was Bill Alderson.

I first met Bill when he was a young engineer for Network General, which made the fabulous Sniffer protocol analyzer. Many of you who are not from that generation may not realize what this tool was or how much of a big deal it was to have a device that could record packet traffic and examine it bit by bit. Today we have open source tools that do the same thing for free, but back then the Sniffer cost four or five figures and came with a great deal of training. Bill cut his teeth on this product and now has his own company, HopZero that has an interesting way to protect your servers by restricting their hop count.

Bill and I first met back in 1989 when I worked at PC Week and we wanted to test the first local area network topologies. We set up three networks, running Ethernet, Token Ring and Arcnet in a networked classroom at UCLA during spring break. All were connected to the same Novell Netware server. Ethernet won the day (as you can see in this copy of the story), and the other topologies died of natural causes. But I digress.

Jon, Bill and I flew to Norfolk and spent a day with the Red Cross staff to try to figure out what was happening with their Novell network. We did all sorts of packet captures that weren’t conclusive. Our first thought was that it had to be something wrong with the server, but we didn’t see anything wrong. Our second thought was more insidious. Being in Norfolk, we were directly down the road from the naval base (you could say that about much of the town, it is a big base). We actually managed to get through to the base commander to find out if their radar was active when they were coming into port. Imagine making that phone call these days in our post-9/11 world? Anyway, the answer we got was negative. Eventually, after hours of shooting down various theories, we figured out the cause of the problem was a wonky network adapter card in someone’s PC. It usually operated just well enough that it didn’t interfere with the network most of the time. Once we replaced the card, everything went swimmingly, and we could put away our tin-foil hats.

Okay, so what is the Cleaning Person Problem? This sounds like folklore, but another reader told me about a problem they had on their network years ago. The reader was periodically disconnected from his network at the same time every night. He was one of the few people online at the time in their office, so it wasn’t like there was high traffic across the network. Eventually, after several evenings he figured out the problem: The cleaning crew was vacuuming the rug in the server room, and the network cable to the server was being run over by the vacuum. Because the cable wasn’t properly crimped and because it was run under the carpet (who knows why this was done), it was shaken just loose enough to disconnect it from the server. When the crew was finished, the cable would operate just fine. Thankfully, they made a better cable and ran it elsewhere where no one could step on it.

The Cleveland folks that had their car fobs disabled actually had it easy: the fault was in a very deliberate emitter that – while initially difficult to trace – was a very binary cause. Their challenge was that not every car fob and garage door was affected. The two scenarios that I mention here were not so cut-and-dried, which made troubleshooting them more difficult. So keep these stories in mind when you are troubleshooting your next computer or networking problem, and don’t be so quick to blame user error. It could be something not as obvious as the odd radio transmission.

This post is taken from a recent issue of the Internet Protocol Journal and reused with permission. It is written by Leroy Harvey, a data network architect.

The networking world seems to be losing sight that NAT is a crutch of sorts, a way of dealing with the primary problem of a lack of IPv4 addresses. An earlier article in IPJ stated that NAT provides a firewall function. I think NATs and firewalls are mutually exclusive, even if they are found on the same networking device. This is because NATs don’t by themselves provide any natural protection from the host on the other side of a protection point. The two can operate independently.

NAT does present real-world problems with a number of products, such as Microsoft AD Replication and IBM’s Virtual Tape Library. Passing through a NAT breaks the application’s intended communication model and requires compensating mechanisms.

We are asking the wrong question if we say, “should I deploy IPv6 now”. Someone once told me that IPv6 was here to stay. To my way of thinking, it has not arrived after 25 years.

Let’s look at the situation where we want to merge two large company networks together that both make extensive use of NAT. This becomes more complex than if the two networks were originally using a valid replacement for IPv4 and sadly, that protocol doesn’t exist. While I agree with the notion that the Internet can’t be completely stateless, this doesn’t justify using NAT as middleware. Justifying NAT for the sake of IPv4 life-support is nonsensical.

We should appreciate NAT for its role as a tactical compensating mechanism for IPv4 address depletion, not a a strategic future-proofing scalability mechanism for IPv4. Really what many are saying about NAT is just putting lipstick on the IPv4 pig. Unfortunately, in IT there is nothing more permanent than a temporary solution. Let us not fall victim to this easy psychological trap only because we seem to have collectively painted ourselves into a corner of sorts.

Over the weekend, I had an interesting experience. Normally, I don’t go into my office then, which is across the street from my apartment. But yesterday the cable guy was coming to try to fix my Internet connection. During the past week my cable modem would suddenly “forget” its connection. It was odd because all the idiot lights were solidly illuminated. There seemed to be no physical event that was associated with the break. After I power cycled the modem my connection would come back up.

I was lucky: I got a very knowledgeable cable guy, and he worked hard to figure out my issue. I will save you a lot of the description here and just tell you that he ended up replacing a video splitter that was causing my connection to drop. Cable Internet is using a shared connection, and my problem could have multiple causes, such as a chatty neighbor or a misbehaving modem down the block. But once we replaced the splitter, I was good to go.

Now I have been in my office for several years, and indeed built it out it from unfinished space when I first moved in. I designed the cable plant and where I wanted my contractor to pull all the wires and terminate them. But that was years ago. I didn’t document any of this, or if I did have misplaced that information. But the cable tech took the time to make up for my oversight, He tracked down my misbehaving video splitter that was buried inside one of my wall plates. And that is one of the morals of this story: always be documenting your infrastructure. It costs you less to do that contemporaneously, when you are building it, then when you have to come back after the fact and try to remember where your key network parts are located or how they are configured.

Part of this story was that I was using an Evenroute IQrouter, an interesting wireless router that can optimize for latency. I was able to bring up this graph that showed me the last several minutes’ connection details so I knew it wasn’t my imagination.

Now my network is puny compared to most companies’, to be sure. But I have been in some larger businesses that don’t do any better job of keeping track of their gear. Oh the wiring closets that I have been in, let me tell you! They look more like spaghetti. For example, here I am in the offices of Checkpoint in Israel in January 2016. Granted, this was in one of their test labs but still it shouldn’t look like this (I am standing next to Ed Covert, a very smart infosec analyst):

Compare this with how they should look. This was taken in a demonstration area at Panduit’s offices. Granted, it was set up to show how neat and organized their cabling could be.

Documentation isn’t just about making pretty color-coded cables nice and neat, although you can start there. The problem is when you have to change something, and then you need to keep track when you do. This means being diligent when you add a new piece of gear, or change your IP address range, or add a new series of protocols or applications. So many times you hear about network administrators that opened a particular port and didn’t remember to close it once the reason for the request was satisfied. Or a username which was still active months or years after the user had left the company. I had an email address on Infoworld’s server for years after I no longer wrote for them, and I tried to get it turned off to no avail.

So take the time and document everything. Otherwise you will end up like me, with a $5 part inside one of your walls that is causing you downtime and aggravation.