Corporate IT shops that need to roll out new network, infrastructure and Internet applications or who have to make expensive equipment upgrades are increasingly turning towards independent test labs as part of their decision process. The labs can help identify weak points, application bottlenecks, design flaws, and show-stopping bugs in a non-production environment, and are also useful “bake-offs” to pit potential candidates against each other in controlled scenarios that can mimic their eventual use. And these days there are several outsourced labs that are available from a number of small to large-scale providers (see chart below).

Outsourced Independent Test Labs

* Other offices outside of the US as well.

Why go with an outsourced lab? The two biggest reasons are expertise and resources. “Some IT shops do have the testing expertise, but they don’t always have the time to set up a test given the press of daily operations,” says Joel Snyder, senior partner of Opus One in Tucson, Ariz. “It is often better to bring someone in who does testing as part of their daily job.” And the specialized test gear is more than a bunch of PCs on a gigabit switch. “We have all sorts of test equipment here,” says David Newman, director of Network Test in Thousand Oaks, Calif. “Most enterprises don’t have specialized test instruments such as the Spirent Test Center that can emulate millions of routes and generate billions of packets. They could rent the tools but they probably don’t have the expertise to use them. Network testing is not a core competence for many IT organizations.” And Greg Shipley, the CTO of Neohapsis in Chicago says, “There are still specialized skills, test plans, environment preparation, environment validation, and a slew of other components necessary to do world-class testing that are not usually found in abundance in corporate IT departments.”

Testing can also avoid potential show-stopping issues with a roll out. “We did a test for global logistics company and found incredible poor performance of a IBM ISS security appliance,” says Snyder. “It turns out that they were sending traffic over a well known port that ISS expected to see something else and didn’t know how to decode the traffic. We fixed the configuration of the ISS box and saved our client a lot of grief had they put this into production without doing any testing.”

“The Ohio Secretary of State came to us because they expected a great deal of load on their Web site on election night and wanted to validate that they wouldn’t experience any problems,” says Steve Gruetter, the director of Platform Labs in Columbus, Ohio. “They had made a complete revision of their infrastructure since the 2004 election and had not done any type of load/stress testing. We lined up a consultant who helped them design a test with our hardware, software and bandwidth and found that they could handle nearly three times the load of 2004 with about the same amount of bandwidth, and they did fine on the night of the election.”

“Organizations will often come at product evaluations differently,” says Shipley. “Some want help assessing features and functionality, others are looking for help with performance testing, others are looking at the security of the product itself. Some want little more than helping define their evaluation criteria, where others want help with all of the above.”

Here are a few tips to consider when shopping for a test lab:

— If technical criteria matter for your purchase, make sure you understand what they are before you put together your test plan. “When the time comes to deploy a product, you have to prove that you didn’t make a big mistake. Even so, most people don’t usually buy products solely on technical criteria,” says Snyder.

— Understand what testing for performance really means, and what are the boundaries of acceptable and unacceptable performance before you set up your test plan. “Have clear goals going into tests. Don’t change objectives during the project, says Newman.

— Know what the facility specializes in, and what the actual lab testers’ expertise is before you hire them. This is just common sense, but each test lab has a different specialization as well as size and facilities. They range in size from the one-person Network Test to the multi-office global companies of Applabs and West Coast Labs.

Some, like Network Test, do more performance-related tests, while Opus One specializes in firewall and Virtual Private network devices. ExtremeLabs has deep expertise in network and Internet applications: “One of the most difficult tests we had to do was to simulate 200,000 DNS requests a second, all with different source IP and MAC addresses,” says Tom Henderson the managing director of ExtremeLabs, Inc. based in Indianapolis, Ind. “We found a flaw that the developer fixed within a few days.” They also work in virtualization consolidation scenarios, where a company wants to migrate their physical servers to virtual ones and understand what the resulting configuration, load, and performance will be.

Platform Lab doesn’t actually conduct the testing – it is a state-funded facility and will connect clients with the appropriate consultants to perform the tests. “We just setup the hardware, software, and bandwidth to their specs. While we don’t do the tests ourselves, we have relationships with 125 consulting firms,” says Gruetter. And Applabs does a lot of work in storage area networks testing and stress testing eCommerce Web sites.

— Know what equipment, bandwidth, and test gear you need. If you want a large collection of actual PCs (versus virtual machine instances) to do your tests, the best place to start is with Applabs facility in Utah. They have 1,000 individual PCs that can run automation software for their tests – the other labs use virtual machines or synthetic test tools to simulate client and server loads. “We are the only game in town,” says Doc Parghi, the senior VP for North America for Applabs. “The Utah test lab is a small part of our business and we have many other services, too. We have a significant presence in the UK and the majority of your staff is in India where we do automation and regression testing.” Wherever you go, you should find out what test gear the lab has and whether it will be appropriate or not for your needs.

— What kind of vendor support is expected during the actual test? “I often see that vendors don’t always send their A-list engineers though we are doing tests that may result in a purchase of tens of millions of dollars of their gear,” says Newman. “You just can’t send a junior field sales engineer who may not understand the business need or how to best deal with the flaws that we uncover during the tests, or the complexity of the application that we are testing.”

— What kind of audience will be reading the resulting reports? You also need to define whether a C-level executive will be reading the results, or will you need your own IT engineers to review the reports. “Sometimes our clients want recommendations, other time they are satisfied with just the raw results,” says Newman. A good strategy is to examine some writing samples before hiring, and most of the lab Web sites have links to download these or can send them on request. Applabs also can call on its own analysts that they can bring into a test evaluation too.

— Finally, how much is this going to cost? Each lab prices their services differently, and getting an estimate might take some work. For example, the Columbus Platform Lab charges ala carte for bandwidth, machine usage and other items, such as renting any test equipment. They charge clients based on a formula of $12/MB per day of testing. But since they have access to some very large Internet pipes, up to gigabit Ethernet speeds, that can get expensive if a client wants a lot of bandwidth for their tests. Network Test wouldn’t quote any prices, but they include all test gear, bandwidth, and equipment as part of their fee. ExtremeLabs charges $3,500 a day with a four-day minimum. Opus One charges $2,000 a day.