Lessons Learned from Large Network Testing

Last week I was out at my old alma mater, Stanford University. I was working with the central network administration group and having a lot of fun. The Stanford network is immense: there are more than a thousand servers, probably in the high thousands at last count. And more than 75,000 assigned IP addresses, with some of them actually being used too! Ironically, we were doing our testing in the same building where I first set up Network Computing’s Real World test labs about 15 years ago, and where the first router was invented about 30 years ago. With all that history surrounding me, it was great to be down and dirty in the lab.

My purpose was to test a bunch of SSL VPN products for Information Security magazine. While I won’t give away the results of the test — you’ll have to wait until the article is printed this fall — I will provide some of the lessons learned while working through the tests for those of you that are about to evaluate these products for your own use. The lab just had a few machines to do the test, but what was important was being around all these folks that were responsible for such a large network, and understanding the issues through their eyes. Here is my top ten list.

1. Assemble a team before you even think about testing anything. SSL VPNs touch a lot of different places across the enterprise. At Stanford, I was lucky enough to work with six or seven very experienced people who understand their piece of the puzzle intimately. You’ll need folks who understand desktop, server, firewall, authentication, and security issues. And sometimes, you’ll need to gather them all in the same room to resolve some thorny problems.
2. What and how you authenticate your users is critical. We were using existing Radius and LDAP servers coming from Stanford’s Active Directory repository. The biggest problem we had was setting these up correctly to work with the different products. But even once we got things cooking, the issue remains on how you specify your users credentials, and what rights you give them once they get authenticated. This has nothing to do with the eventual VPN product that you pick, but you need to be aware of this before you buy anything and make sure that your directory can contain this information, or you can import it there via some XML trickery.
3. Where your authentication servers are placed in your network is also an issue. Some of the VPN products we tested wanted to see these servers on the protected, interior network. This meant moving the servers from wherever they were on the Stanford infrastructure, which wasn’t gonna happen.
4. Migration issues. Stanford has this crazy quilt of firewall rules and existing VPNs that won’t be easy to transition to a newer SSL situation. Make sure that you haven’t hard-coded something into your own infrastructure that won’t be easy to translate into the various rules and schemas used by your SSL VPN.
5. Applications, browser and client operating system versions, and server configurations matter. Make sure you find all the applications that remote users will be running over the VPN, and test them across the various combinations of browser and OS combinations, both with and without the network extension clients. That is a lot of regression testing.
6. If you have a lot of Macs, Linux users, or machines running Windows prior to 2000, prepare for some grief here. Most of the products prefer you run Win XP/2000 clients only. Stanford’s network is about half Windows and half other things, so this was a big issue for them. But even if you have just a couple of non-Windows users, they could be at the top of the corporate food chain and require access from their machines.
7. If you are going to get involved in using the endpoint checking routines that the VPNs offer, do your testing with appropriate link latencies included for the average DSL and remote user. A lot of these routines send a bunch of bits back and forth across the link pre- and post-login, and these can significantly increase the login times for users while they are inspecting each machine for anti-virus, firewall and other configuration tests. Latency matters. If you can’t simulate it, make sure you go to a few slower-speed networks, such as your local coffee bar, and try the connections out.
8. Think about who is going to administer these boxes once you bring them in. Many of the products we tested didn’t fare well when it came time for multiple people to be admins and having different departments be responsible for different subsets of users or access rules. Few of the products enable this scenario very easily.
9. If you operate a huge network like Stanford, you probably want a box that supports true active/active failovers on a clustered configuration. Most of the vendors require some additional load-balancing product to enable this configuration.
10. Finally, don’t get lost in the admin interface. They are all complex beasts, but remember what you are trying to do is allow remote access. Keep sight of that goal as you plow through the various menus.

Check out my SSLVPN page for links to other product reviews, and stay tuned for the piece in InfoSec this fall.

0 thoughts on “Lessons Learned from Large Network Testing

  1. You are so correct! Large customers always have larger networks than anything a vendor could build in their regression labs – required way to test.

    In my Wellfleet router days we did final beta testing with the largest customers to really shake out a release of code. They knew more than we did about deploying this stuff.

    Charlie Kraus
    Dir HBA Business Unit
    LSI Logic

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.