What is A/B testing?

An A/B test always produces a winner, even when there is no real difference. Flip a coin a hundred times with two coins and one usually lands on heads slightly more often, without being a better coin. Statistical significance is the line where the difference between A and B is too large to reasonably be chance, and you only reach that line with enough sample:

• Statistical significance means the difference is probably real, not noise. A common threshold is 95 percent confidence, that is at most a five percent risk that you see a difference that is not actually there.
• The sample, the number of visitors and conversions, decides how confidently the test can tell signal from noise. The smaller the difference you want to detect, the more visitors you need.
• As an illustration: if a page has 400 visitors a week and converts around two percent, that is roughly eight conversions a week per variant. One or two sales either way then move the figure so much that it takes weeks before a real difference can be told apart from luck.

Work out the required sample before you start, not after. A sample size calculator↗ takes your current conversion rate and the smallest improvement you want to detect, and gives the number of visitors the test needs per variant. CXL's guide to testing statistics↗ goes deeper on why that figure is not negotiable. In short: the traffic decides whether an A/B test can give you an answer at all, measured against the business and not against a curve that looks promising on a Tuesday.

An A/B test that holds up follows the same order every time. Careless work on one step makes the result unreadable, however nicely the tool visualises it. Six steps separate a test that settles something from one that merely looks scientific:

1. 01Formulate the hypothesis. A test starts from a hypothesis with structure: if we do X, then Y happens, measured via Z. "If we move the price guarantee above the button, then the share who proceed rises, measured via clicks to checkout." What you test and why is decided in the hypothesis work, not here.
2. 02Build the variant. One change at a time, against the control. Keep B as close to A as possible apart from the very thing the hypothesis concerns, otherwise you will not know what caused the outcome.
3. 03Set the sample in advance. Work out how many visitors per variant the test needs, based on your conversion rate and the difference you want to detect. That figure, and therefore roughly how long the test should run, you lock before the start.
4. 04Run to significance. Let the test run until the predetermined sample is reached. Look along the way by all means, but make no decisions yet: a lead on day three is usually noise that evens out.
5. 05Read the result. A test gives three answers: B won, B lost, or no reliable difference. All three are decision support, not status. "No difference" is an answer too, it means the change was not worth the traffic.
6. 06Draw the lesson. Write down the hypothesis, the outcome and what you think lay behind it. The next hypothesis gets sharper from the previous test, and the pipeline improves over time.

The test setup and the data are yours: your accounts, your tools, your history of what has been tested and won. An experienced hand does in an hour what an inexperienced one takes four to do, and the difference shows most clearly in the reading: seeing whether a win is real or whether the test was stopped a day too early.

A/B tests rarely fail because of bad ideas. They fail because of impatience and carelessness with the statistics. Five pitfalls keep coming back:

• Peeking and stopping too early. Ending the test the moment B leads is called peeking, and it is the most common way to fool yourself. Look often enough and you will nearly always find a temporary lead that does not hold. Set the sample in advance and run to the end.
• Too many variants at once. A test with A, B, C and D splits the traffic into more groups, so each variant gets fewer visitors and significance is delayed. With limited traffic, two variants are almost always right.
• Testing trivia. The button's exact shade rarely moves the needle. Tests cost time and traffic, so spend them on changes that can make a noticeable difference if the hypothesis holds, not on what happens to be easy to build.
• Too little traffic. A page with a few hundred visitors a week rarely reaches significance in a reasonable time. Then A/B testing is the wrong tool, not a test to be run patiently forever.
• No predetermined sample size. Without a target for how many visitors the test should collect, "are we done?" becomes a feeling instead of a threshold. Then the most impatient person wins, not the variant that is actually best.

The common thread is statistical discipline, not more tools. Evan Miller's classic on how not to run an A/B test↗ shows mathematically why peeking inflates false wins. The pitfalls are all about the same thing: letting impatience go before the sample. A result read too early is an answer you trust without having earned it.

A/B testing is the right tool when the traffic is enough to give an answer in a reasonable time. When it is not, there are better ways forward. Ask two things before you set up a test:

• Is the traffic enough? If the sample calculation says the test would have to run for months before it reaches significance, A/B is the wrong path. Then directly implementing a high-confidence hypothesis is often wiser: make the change, follow the outcome against the business over time, and learn from the direction rather than from a test that never becomes significant.
• Are you testing one thing or several that interact? An A/B test compares two whole variants. If you want to understand how several elements work together, headline and image in combination for instance, that is a multivariate test, and it needs far more traffic than an ordinary A/B test because the combinations split the visitors into many more groups.
• Is the question worth the traffic? Every test ties up visitors who would otherwise have met your best known variant. Test what can move the business, implement the obvious directly, and save the statistics for questions where the answer really is uncertain.

The choice between testing and implementing directly is not a matter of faith, it is a matter of traffic. With plenty of traffic, an A/B test gives the safest answer. With little traffic, a well-grounded change followed up against the business is often better than a test that never reaches significance. An experienced hand sees the difference quickly, and that often decides whether the month's work moves the needle or just fills a report. To find out what would actually raise your conversion, see how Memorise works with conversion.