Like anything in science there are some thing you need to know before running A/B tests, there are some basic things you have to get right or the whole test will be meaningless.
As I said in the previous post A/B testing is done by dividing your visitors into two groups, we’ll call them A and B, and showing a different page to each group and than measure what group performed better, there are quite a few split testing tools out there and they should handle the technicalities I describe here automatically – but it’s important to know this in order to understand what your tool is doing.
First, to get any result at all you need a measurable result, usually we simply count how many time visitors from each group performed some action (typical actions are: buying something, downloading something or filling a form) – those actions are usually called “conversions” (the person converted from a visitor to a customer or a lead).
After we decided what our conversion action is we get to the second point, have to split our visitors between those groups, it’s extremely important to randomly divide visitors between groups because otherwise we are likely to get people how are more likely to buy to begin with in one group – and this will make this group perform better regardless of the actual design we show them.
As a (overly simplified) example, if we show one design to people visiting on workdays and a different design to people visiting on the weekend and we sell business software the workdays design will win simply because people who want to buy business software usually don’t look for products on their day off.
It’s also important to know that all sort of tiny details effects visitors behavior (page load times is a notable example) and that if the visitor knows he/she is participating in a test the test is just about completely ruined.
In the next post I’ll teach you how to read the results of a split test (it’s not difficult but can be somewhat counter-intuitive).
posted @ Monday, March 1, 2010 11:15 AM