A/B testing is like many things that can be vexing about the Web: a simple concept can turn into a complex programming project. But while the idea is simple — producing two (or more) different webpages for your site and instrument them to see which one drives more traffic or more sales – getting it to work can be fraught with politics and the actual implementation details.

Why bother? Mainly because there is almost nothing else that you can do that can have such a big effect. Just by changing the text size or button color you can generate a 50% increase in click through rates, which is what one of Denmark airports found with their website when they swapped out text that said “Shop Online” to “Buy Tax Free.”

A/B testing isn’t new: magazine publishers have experimented with various cover layouts and images to determine what sells best on newsstands. But it still may be new to your IT department, or your management, and it is high time to get started with the process. Many modern websites still squander their home page real estate, filling up the page with blank spaces or on things users ignore: filler, self-promos and ads. Usability guru Jakob Nielsen recommends to “cut the fluff and spend the pixels on design elements of interest to users — mainly content, but also navigation.”

So we’ve put together some basic tips, and this slideshow showing you what worked in the past for major ecommerce retailers, and some suggestions on potential ways to attack your first testing project.

Your first challenge is all about politics, and who owns which portion of your website that you are trying to improve. “Sometimes a chief marketing officer loses track of what their customers are buying, and having A/B tests reconnects customers with the company,” said Brooks Bell, head of a testing firm that bears her name. (I met her at a trade show earlier in November.) But instead of picking the CMO or another executive, it might be better in the long term to find someone at the bottom of the food chain looking to shine. One of Bell’s earliest projects was such an individual who didn’t have any staff or much budget for testing. As a result of what they found and the bottom-line benefits they produced, “the manager now has a staff of ten and is training other groups within the corporation to do other testing, and was promoted to a director level position that is now very influential and visible within the organization.” So A/B tests can be a way to advance your own career too!

But you also want to be careful that your design is far enough alone where the testing can be useful. User experience expert Danielle Cooley says, “A/B testing is great when you’re trying to determine if a specific detail should be one way or the other, such as should a call to action button be orange or green? But it doesn’t help much early in the design process, or that you still need to have a sensible workflow and solid page design.”

Once you have found your testing champion, your next challenge is purely technical: what kind of testing platform are you going to use and whether you choose someone in-house or a consultant such as Bell, Wingify, Monetate or the dozens of others who ply this trade. You might want to designate a couple of your technical staff and give them a week and the necessary training to get up to speed on the topic, and let them find the right tool for their work. Keep in mind that you don’t just want a coder: you want to build a longer-lasting team to conduct and interpret the tests. “You need lots of talented people to be able to ask the right questions and interpret the insights from the data that you collect, [along with] skilled analysts and marketing people to fully realize your changes,” said Bell. “We typically use a war room where we have all the stakeholders present and brainstorm on the best strategies.”

But if you do look outside, most of the consultants have pretty convincing stories about the amount of dough they have saved their clients over the years or the power of their approaches, so it can be hard to evaluate them.

Finally, you have to keep it going. Testing is a process, not a destination. One part of the challenge with testing is being able to scale your testing effort up as word gets out around the company about your results. Another is that the best IT shops know that they have to frequently make changes to their sites to stay fresh and appeal to new eyeballs. No one is better than this than Intuit, makers of QuickBooks.

Matthew Heusser, Managing Principal for Excelon Development saw when he visited their offices how two interface designs are rolled out for a month and statistics are gathered to decide on which one is ultimately deployed. These tests have helped Intuit weather numerous changes over the years to keep their product fresh. As he writes, “The amazing thing about Intuit is that it has already survived three major market shifts, from DOS to Windows, Windows to the Web, and now the conversion to Social/Mobile products.” That is a lot of history.

Here are some final words of wisdom from Bell: “There is always something that an A/B test misses, even with the best of intentions and careful planning. So if get stuck, think about redoing your tests with a different segmentation of your audience, or with a different focus. There is no single best testing option. Just because you can measure something in a test doesn’t mean you should do it. Think more strategically and long-term, and make sure your executives ultimately support what tests you are doing.”

There are dozens of A/B consultants, tools, and compendiums of tips and tricks.  Here are a few links to check out:

— Robin Johnson, writing earlier this year in Optimizely, gives you 71 different ideas on things to test, including website copy, visual elements, and just plain common sense things such as whether you have a positive or negative spin on what you are trying to sell.

— Uri Bar-Joseph, writing in SearchEngineWatch, has eight rules for A/B testing, including focusing on one variable, choosing your groups randomly and measuring the results carefully.

–And if you want to have some fun with A/B tests of the past, take a look at these 12 results that were somewhat counter-intuitive from WhichTestWon.com, a commercial A/B testing provider.