Why peeking at A/B tests finds false winners

Check a standard A/B test every day and its 5% false-positive rate quietly becomes 21%. We simulated 4,000 tests to show why, and what fixes it.

The Mockly team

October 26, 20268 min read

Your test has been running for five days. The dashboard says the new headline converts 30.9% better, with p = 0.036. Do you ship it?

If the test was planned for two weeks, the honest answer is no. Here it's also the right answer: the two versions in this test were identical. Checking results as they arrive and stopping when they look good is called peeking, and with the standard statistics behind most A/B test calculators, it quietly multiplies false winners.

This post shows how often peeking misleads you, why, and what to do instead, with numbers from 4,000 simulated tests.

What is peeking?#

Peeking is checking an A/B test's results before it reaches the sample size it was planned for, and acting on what you see: stopping early to ship a winner or drop a loser.

The trouble is that a standard significance test, the z-test or t-test behind most A/B test calculators, makes its promise about one look. A p-value under 0.05 means that if there were no real difference, a result this extreme would turn up less than 5% of the time, at a single look planned in advance. Every extra look is another chance for random noise to cross that line. Stop at the first crossing, and the real rate of false positives is far higher than the 5% you think you're running at.

A winner that wasn't#

Here's the test from the top of this post, day by day. Both versions convert at exactly 5%, so B never really beats A. Watch the gap between them:

One A/A test, two weeks: B against A, and where each test calls a winner

Day 5 · 5,000 visitors · p = 0.036

B against A: +30.9%
Standard test: B wins
Anytime-valid: No winner

Show the numbers

End of	B against A	p-value	Standard test	Anytime-valid
Day 1	+27.4%	0.41	No winner	No winner
Day 2	+20.2%	0.37	No winner	No winner
Day 3	+39.0%	0.052	No winner	No winner
Day 4	+26.5%	0.099	No winner	No winner
Day 5	+30.9%	0.036	B wins	No winner
Day 6	+26.5%	0.045	B wins	No winner
Day 7	+14.6%	0.20	No winner	No winner
Day 8	+14.5%	0.18	No winner	No winner
Day 9	+5.7%	0.55	No winner	No winner
Day 10	+8.6%	0.35	No winner	No winner
Day 11	+5.7%	0.52	No winner	No winner
Day 12	+4.2%	0.61	No winner	No winner
Day 13	+2.6%	0.75	No winner	No winner
Day 14	+1.5%	0.84	No winner	No winner

Both versions convert at 5%, so any winner is a false one. The black line is how far apart B and A are, in standard errors: a standard test calls a winner beyond the dashed lines, the anytime-valid intervals beyond the blue ones. Dots mark the end of each day.

In the first days, B pulls ahead by chance, as one version always does while the numbers are small. On day 3, it's 39% ahead, with p = 0.052: nearly significant. On day 5, p = 0.036. Check once a day, stop at the first p < 0.05, and you ship B on day 5.

Keep the test running instead, and the gap fades. By day 14, B is 1.5% ahead, with p = 0.84. Nothing was there. You'd have looked at the moment the noise happened to be loud.

The blue lines are where Mockly's anytime-valid intervals would call a winner. They sit further out, most of all in the first days, when there's little data and noise is loudest, and this test never comes close to them.

How often peeking finds a false winner#

One test is an anecdote, so we ran 4,000. Each simulated experiment is an A/A test: two identical versions converting at 5%, with 1,000 visitors a day for 14 days. Since nothing differs, any winner is a false one.

We read the same 4,000 experiments on schedules from once, at the end, to every 15 minutes, and stopped each one at its first significant result. We read them two ways: with a standard two-proportion z-test at p < 0.05, and with the anytime-valid intervals Mockly's studio uses. Both waited until each version could expect 10 conversions, as the studio does, which spares the standard test its noisiest early looks.

A/A tests that named a false winner, by how often they were checked

Checked once a day · 14 looks

Standard test, p < 0.05: 20.9%
Anytime-valid (Mockly): 1.3%

Show the numbers

Checked	Looks	Standard test	Anytime-valid
Once, at the end	1	4.4%	0.3%
Once a week	2	7.8%	0.6%
Every other day	7	15.5%	1.0%
Once a day	14	20.9%	1.3%
Twice a day	28	26.0%	1.5%
Every 6 hours	56	29.5%	1.6%
Every 3 hours	112	32.3%	1.7%
Every hour	336	35.9%	2.0%
Every 15 minutes	1,344	38.1%	2.1%

4,000 simulated A/A tests: two identical versions converting at 5%, 1,000 visitors a day for 14 days, each stopped at the first look that called a winner. The same tests are read on every schedule.

Checked once, at the end, the standard test keeps its promise: 4.4% false winners, within the 5% it allows. Checked once a day, it names a winner in 20.9% of experiments where there's nothing to find. Every 15 minutes, roughly what a dashboard left open in a tab does, it's 38.1%: more than one in three.

The anytime-valid intervals named a false winner in 0.3% to 2.1% of the same experiments, depending on the schedule, and never more than the 5% they allow.

These numbers come from one setup. A different conversion rate, more traffic, or a longer test moves them a little, but not the pattern. Statisticians have described it since at least 1969, when Armitage, McPherson, and Rowe worked out how repeated significance tests inflate false positives, and Evan Miller's essay How Not To Run an A/B Test made it famous among marketers.

Why looking more often makes it worse#

Think of the gap between two identical versions, measured in standard errors: the black line in the first chart. It wanders up and down at random. A standard test draws a fixed line at 1.96 standard errors and asks whether, at one moment, the gap is past it. At any single moment, it is about 5% of the time.

But a test you check isn't asked about one moment. It's asked whether the gap is past the line at any of the moments you look, and each look is another chance for it to be. Looks close together are nearly the same look, which is why the curve flattens out instead of shooting to 100%. It never stops rising, though. Given enough looks, a line that wanders at random eventually crosses any fixed line: that's a mathematical certainty, the law of the iterated logarithm, not bad luck.

Three ways to stop fooling yourself#

Look once#

Decide the smallest lift worth detecting, work out how many visitors you need with a power calculation, run the test until you have them, and look once. That's what a standard test assumes, and it needs the fewest visitors.

It's also hard to live with. Nobody wants to keep sending half their traffic to a variant that's obviously broken, or to keep waiting when the answer has looked clear for a week. And the dashboard is right there.

Plan a few looks in advance#

Clinical trials faced the same problem decades ago. A group sequential design fixes a small number of interim looks before the test starts, and uses a stricter line at each, so the chance of a false winner across all of them stays at 5%. Pocock's design, from 1977, and O'Brien and Fleming's, from 1979, are the classics.

They work, but only for the looks you planned. An extra look, or a look at the wrong time, breaks the guarantee.

Use statistics that allow any number of looks#

Anytime-valid inference makes the promise hold at every moment, rather than at one planned moment. A confidence interval becomes a confidence sequence: a range that, with 95% probability, contains the true difference at every point in time at once. Look after every visitor, stop the first time it excludes zero, and the chance of a false winner stays at or below 5%.

In the chart of one test, that's the blue curve: further out than 1.96, most of all at the start, when there's the least data. Researchers at Stanford and Optimizely brought always-valid p-values to A/B testing in the mid-2010s, and later work on confidence sequences made the approach practical for everyday tests. Several experimentation platforms now offer it.

What anytime-valid statistics cost#

Nothing is free. To stay valid at every look, the intervals have to be wider than a standard test's at its one planned look, so they need more visitors to detect the same lift. For an 80% chance of detecting a 20% lift on a 5% conversion rate, Mockly's intervals need 33,860 visitors, where a standard test read exactly once needs about 16,300.

The simulation shows the same thing from the other side: checked once, the anytime-valid intervals named a false winner 0.3% of the time, far below the 5% they allow. That unused margin is what pays for looking as often as you like.

What you get for it:

No sample size to fix in advance. Run until the result is clear, or until you've waited as long as the decision is worth.
Big differences are called early, good or bad, so a broken variant doesn't keep getting half your traffic.
Nobody's habits can break the result. Refreshing the dashboard, a weekly report, or an AI agent checking an API every minute: none of it makes the verdict less trustworthy.

For teams that look whether they're supposed to or not, which is nearly every team, that's the right trade.

How to read a test without fooling yourself#

Decide before you look. Write down the smallest lift that would change what you do, and how long you can wait for it. That's your plan, whatever statistics you use, and how long should you run an A/B test? has a calculator for it.
Know which promise your tool makes. If its statistics are fixed-sample, look once, at the planned size. If they're anytime-valid, stop when the interval excludes zero, or when the planned visitors are in.
Run whole weeks. Valid statistics tell you a difference is real for the visitors so far. Weekday visitors behave differently from weekend ones, and a novelty can fade, so a test stopped on day two may not hold for next month's visitors.
Check the split. If visitors don't split the way you set (a sample ratio mismatch), something is treating the variants differently, and no statistics can rescue the result.
Read "no clear winner" as an answer about size. It doesn't mean the versions are the same. It means any difference is smaller than the test can see yet.

How Mockly handles it#

Every result in Mockly's studio uses anytime-valid confidence sequences, so the verdict is safe to act on whenever you read it: every day, every minute, or every time an agent asks. With more than one variant, the intervals are adjusted together, so more variants don't mean more false winners. A check of the traffic split runs alongside, also valid at every look, and How long to run shows how many visitors and days each lift needs at your traffic.

The numbers in this post come from the studio's own statistics code, and a test in our codebase runs the simulation again to check every one of them. Reading results explains the verdicts and the numbers behind them, and the product page shows them in the studio.