It isn’t often that you have access to your local neighborhood data center, literally a few steps down the block. But I did. I live in a residential area of St. Louis called the Central West End, and I pass by the offices of the Regional Justice Information Service (REJIS) almost every day. When I learned that they were going to be expanding their data center, I knew that I had to be there for the actual move. That was before I found out about the background check and that I would have to be there to witness the move during the middle of the night. But we are getting ahead of the story.

 

In the process of reporting on their move, I got to see some terrific best practices about how to pick up your servers and minimize downtime, too. Hold these thoughts, I promise I’ll get back to these issues.

 

Moving a data center isn’t easy under the best of circumstances. And no matter how hard you plan, there are still things that you don’t think about, like a brand new elevator that wasn’t working. More on that too.

 

REJIS is an interesting enterprise: they were founded in 1976 to provide IT services to the public sector in the St. Louis area. They now handle more than 200 different government clients for applications development and support more than a dozen different programming languages. They do batch and online processing and hosted facility and server management. With more than $15 million in annual revenues, the majority of their customers are county and local government criminal justice agencies, although they also provide data connections to St. Louis’ most noticeable landmark, the Gateway Arch. They also support more than 1,000 mobile devices that are in local police cars. There are about 150 employees in the building, most as you can imagine involved in IT-related jobs. REJIS has about 100 Intel-based servers, mostly Dells, and an IBM eSeries too.

 

Consider this: given REJIS’ client base, they have all sorts of connectivity to their clients. They make use of a complex network of frame relay, T-1s, ISDN, cable modems, MPLS & Fiber, and even dial-up. All of these links are encrypted, as you might imagine given the sensitivity of the data that traverses these networks. And all of these connections will have to be moved from their old wiring closet to the new one next door.

 

With all this connectivity, REJIS needed to take some extra steps to ensure that all the communications lines would work after the move.

 

BEST PRACTICE TIP: “Put staff at various remote locations during the move,” says Eric Gorham, director of IT for the organization.  That way as services are restored they can be tested remotely without impacting your client’s staff.

 

REJIS also has an automated background investigation system and implemented the first automated fingerprint system in Missouri. I got to experience that first-hand – in order to enter their data center during the move, I had to be checked out. This was one database that I didn’t want any hits and luckily, I passed.

 

“When a cop pulls someone over on the street and runs a check on their plate and their driver’s license, you can get over 20 different responses from various law enforcement data bases,” says Gorham. “We can then organize this information for the officers in their patrol car.”

 

REJIS had outgrown their 30-year old data center that was located in the basement of their office building. [When it came time to expand, they decided to build a new data center next door, in a former parking lot and double their floor space in the process. The new data center would accommodate hot and cold isles for distributed servers, rather than the old big-iron designs of the last century. “We had 12-inch raised floors that were getting crowded and reducing our airflow,” says Gorham. “The new data center has 24-inch floors so we don’t have to worry about hot spots anymore.” The new data center would also give them more floor and rack space to grow their managed hosting service, and be more secure facility in case of fire, earthquakes (we do have them here, hard to believe), potential flooding, severe wind events, and tornados.

 

A basement isn’t a good place for a data center: “We are in an earthquake zone and our old data center was below grade, not to mention being underneath a five-story building. We have had some water there before and it could have been worse,” said Gorham. “Plus, there was a lot of dead wiring too that was blocking airflow. We needed a better disaster recovery plan and having a new data center in a separate building will help.” Plus, the old building didn’t have a loading dock, making deliveries difficult. Finally, they wanted to support a backup location for the main state data center located a few hours away in Jefferson City. These features were incorporated into the new building.

 

The move was scheduled for the beginning of Memorial Day weekend, to give the IT staff an extra day in case they had problems bringing everything back online and because most of their clients who work normal daytime, weekday hours wouldn’t be around. It was scheduled to take place in the middle of the night, at 2 am, during the lightest user load on their system.

 

This meant some extra planning: “At two in the morning, you aren’t going to find an extra five-foot cable, so we bought many extra supplies just in case,” says Gorham. “Happily, we can return the unused ones to our vendors.”

 

BEST PRACTICE TIP: Don’t forget about cables and other supplies, too.

 

Now, I am not a night person: usually, I am in bed by 9:30 pm. But to get this story, I set my alarm and arrived shortly before 1 am on Saturday, May 26.

 

Despite the shortness of my commute, I am probably the last person to get to their office. There are already about 50 people crowded into the common break room. Already something has gone wrong. The new elevator isn’t working, and Frank, the elevator technician on duty that night, has already been called in to debug it.

 

BEST PRACTICE TIP: I notice that Gorham’s assistant is walking around with a sheaf of papers. It is the master contact list of every staffer home, cell, and pager number, as well as contact information for key clients and vendors.  Should REJIS need help during the move, they have all the contacts in one place.

 

When Frank heard where he had to go, he was a bit worried. “I am going to pull up to this building in the middle of the night and every cop in St. Louis is going to be there,” he told me as he was sitting on the floor of the service room, trying to debug the elevator circuitry. “Sure enough, here I am at this hi-tech cop shop.” The reason he is sitting on the floor is because there is this huge pile of manuals next to him “The manuals aren’t well written, and the documentation on the error message that I am getting is very obscure.” What delicious irony, for those of us that have had to do the very same thing with computer manuals.

 

While Frank puzzles over the manuals, Gorham is downstairs in the break room, giving everybody their instructions for the night. We are sitting next to a stack of our own – a large collection of pizzas, cookies (carefully home-baked and labeled by flavor by some volunteer), sodas and plenty of coffee. It is going to be a long night ahead and the movers need their fuel. “If you are going to ask people to come to work in the middle of the night, you want to make sure that they are going to be fed and keep them happy while they are working,” says Gorham.

 

To get people juiced, Gorham draws a few random names for prizes – this being St. Louis, more than two dozen people have donated pairs of Cardinals baseball tickets, and he has also purchased several restaurant gift certificates. Then we all go to work.

 

At 2 am, Frank gets the elevator going and the moving crew begins to take apart the control consoles that are needed in the new building. The crew starts to roll the IBM e-Server and virtual tape system across the floor and upstairs into the new building. Everything will sit on top of WorkSafe Technologies’ ISO-Base seismic isolation platforms www.worksafetech.com) that can dampen vibrations in case of earthquakes. 

 

The total distance for the move from the old to new buildings is less than 100 feet – but it is up two flights of stairs and down a couple of halls. Hence the need for the working elevator.

 

As each machine is taken off the old rack, it goes by Chassidy’s station  She is in charge of blowing out and clearing the accumulated dust with a reverse vacuum.

REJIS isn’t moving everything out of the basement. They are dismantling two robotic tape libraries’ that have out lived their usefulness and its 10,000-plus tapes. “Our new library has the same storage with just 1,500 tapes and is compatible with the state data center formats, making it better for disaster recovery purposes,” says Gorham. And they plan on recycling a huge pile of bus and tag cables, and other old equipment [PHOTO 4630] to EPC Recycling.

 

By 2:30 am, “All of our red servers are in place,” says Gorham. “We thought it was going to take longer.” REJIS tagged each server by color, with red being the most critical (Law Enforcement servers, for example) that needed to have the least downtime.

 

BEST PRACTICE TIP: Use triage to determine the most mission-critical machines to move first. Tag every server with information about its IP address, old and new rack locations, and server name to make the move easier.

By 3:58 am, the huge IBM servers are connected and back online. “I’m happy about the limited downtime,” says Gorham. “We were really only down for 118 minutes for mission critical services.”

 

BEST PRACTICE TIP: Tag each cable of every server, near where the cable is connected on the back. When you cut the cables, you still have the tags to guide where to hook up the new wires. [PHOTO 4636]

 

Seeing how things are going well, I head home for a few hours rest. I am back by 6:30 am and walk into the server room with one cable lying across the floor: “We just had one bad connection that we needed to fix,” says Gorham. That was it, all other systems are up and running.

 

All in all, it was a good night for a move. By 7am, as breakfast is served in the break room, most of the crew is finishing up and getting ready to head home. But not before a few lucky ones are picked at random to get some Cardinals tickets as their reward for a good night’s work. Now if only we could make it into the World Series this year too!