We ran eleven landing page tests over about seven months and declared a winner in three of them. Two of those winners subsequently failed to replicate.
The problem was not the page. It was that we were running tests that could not have produced a reliable answer.
The sample size problem
The fundamental issue and the one almost everybody underestimates.
Detecting a small difference in conversion rate requires a large number of conversions, not a large number of visitors. If your baseline conversion rate is two percent and you want to detect a relative improvement of ten percent, the required sample is much larger than intuition suggests — typically tens of thousands of visitors per variant.
We were running tests with a few hundred conversions and reading differences of a few percent as meaningful.
At that sample size, the noise is larger than the effect we were trying to detect. The results were essentially random, which explains why the winners did not replicate.
A sample size calculator, used before starting rather than after, would have told us that most of our tests were incapable of producing a result. That is a five minute check we were not doing.
Stopping early
The second error and it compounds the first.
We watched results daily and stopped when significance appeared.
The problem is that if you check repeatedly and stop when a threshold is crossed, you will cross it eventually by chance. Peeking inflates the false positive rate substantially, and stopping on the first crossing is close to guaranteed to produce spurious winners.
The discipline is to decide the sample size in advance, run to it, and look once. That is much harder than it sounds when there is a dashboard.
We now hide the results until the test completes, which sounds childish and was the only thing that actually stopped us peeking.
Testing things too small to matter
The third error and the one that wasted the most time.
Button colours. Headline wording variations. Image swaps. Small changes to layout.
These produce small effects, and small effects require enormous samples to detect. For a site with modest traffic, they are undetectable in any reasonable time frame.
The tests that produced our only genuinely replicated result were large structural changes — a completely different page structure, a different offer, a different flow with fewer steps.
Big changes produce big effects, which are detectable at achievable sample sizes. This is the practical argument for radical redesign tests over incremental ones on low-traffic sites.
Measuring the wrong outcome
The error that cost us actual money rather than just time.
We tested on immediate conversion rate. One winning variant increased conversions and produced a noticeably higher refund rate afterwards.
The page had oversold. It converted better and produced worse customers, and our metric could not see that.
We now measure downstream where possible — retained customers, refunds, second purchases — even though it lengthens the test considerably.
What we do now
Calculate the required sample before running anything, and if it is not achievable in a reasonable period, do not run the test.
Test large changes rather than small ones.
Fix the sample size in advance and do not look until it completes.
Measure a downstream outcome, not just the immediate one.
And accept a much lower testing volume, which was the hardest cultural change because a testing programme that runs two proper tests a quarter looks less impressive than one running two a week.
The alternative to testing
The thing I would say to anybody at low traffic volumes.
If you cannot achieve the sample sizes, testing is not available to you as a decision method, and pretending otherwise produces confident wrong answers.
What is available is qualitative work — watching people use the page, asking them what they expected, reading the questions they ask support.
That does not give you a percentage and it reliably identifies problems that are obvious once seen. Most of the genuine improvements we have made in the last two years came from watching six people use the site rather than from any test.
Sequential testing, briefly
Worth mentioning because it addresses the peeking problem properly rather than by willpower.
There are statistical methods designed for continuous monitoring, which adjust the threshold to account for repeated looks and allow a test to be stopped early without inflating the error rate.
Several testing platforms implement these. If yours does, use it and the peeking problem largely disappears.
If it does not, the fixed-sample approach with the results hidden is the workable alternative, and it is what we ended up doing.
What we test now instead
Having concluded that most page tests were not viable, the testing effort moved.
Offers, which produce large effects. Pricing structure, similarly. Whole flows rather than pages, such as removing a step entirely.
And audience and creative in paid media, where the traffic volumes are high enough to reach adequate samples in days rather than months.
The general principle is to test where the traffic is and where the effects are large, which is rarely the page everybody wants to argue about.