How long should you run an A/B test?

How long should you run an A/B test? Until it has enough visitors to detect the smallest lift that would change what you do, rounded up to whole weeks, and never less than one. For a page converting at 5%, detecting a 20% lift takes 16,310 visitors with a standard test read once. At 1,000 visitors a day, that's 17 days, so plan three weeks.

That answer comes from a handful of numbers, most of them yours to choose. This post explains each one, gives you a calculator for your own, and covers what to do when the answer comes out as four months.

Work out how long your test needs#

Change the numbers to yours. The two columns are two ways of reading a test, once at the end or as it goes, and both are explained below.

How long to run your test

Conversion rate now

%

Of the page's visitors.

Smallest lift worth detecting

%

From 5% to 6%.

Visitors a day to this page

Only those who reach what you're testing.

Versions, the original included

234

Traffic split evenly.

Read once, at the endA standard test: decide on that day, not before.

3 weeks16,310 visitors · 17 days

Checked as you goAnytime-valid, as Mockly reads results: stop once it's clear.

5 weeks33,860 visitors · 34 days

The smallest lift your traffic can detect

In	Read once	Checked as you go
1 week	31% or more	44% or more
2 weeks	22% or more	31% or more
4 weeks	15% or more	22% or more
8 weeks	11% or more	16% or more

What decides how long a test takes#

Your conversion rate now#

Rates near zero are harder to tell apart than rates nearer the middle, because a few conversions either way make a big difference. A page converting at 1% needs about five times the visitors of a page converting at 5% to detect the same relative lift.

The smallest lift worth detecting#

This is the number that matters most, and the one most often skipped. It's called the minimum detectable effect, and it isn't the lift you hope for. It's the smallest lift that would change your decision: if a 5% rise in sign-ups wouldn't be worth shipping the change, don't plan a test to find one.

Lifts here are relative. A rate going from 5% to 6% is a 20% lift, or one percentage point.

Your traffic#

Count only the visitors who reach the part you're testing. A test on the pricing page gets the pricing page's visitors, not the whole site's.

How many versions#

Each version beyond the original takes a share of the traffic, and is one more comparison. Each comparison is held to a stricter line, so that more versions don't mean more false winners. With a 5% rate and a 20% lift, two versions need 16,310 visitors, three need 29,628, and four need 43,512.

How sure you want to be#

Calculators, this one included, assume the usual settings: a 5% chance of naming a winner that isn't one, and an 80% chance of detecting the lift if it's real, called 80% power. Ask for more certainty and you pay for it in visitors.

Why small lifts take so long#

The visitors a test needs grow with the square of how fine a difference it has to see. Halve the lift you want to detect, and you need almost four times the visitors: at a 5% conversion rate, a 20% lift takes 16,310, and a 10% lift takes 62,462.

Visitors needed for each lift, both versions together

Rate now	+5% lift	+10% lift	+20% lift	+30% lift
1%	1.27M3.54M	326K833K	85.4K199K	39.6K87.6K
2%	630K1.68M	161K393K	42.2K93.7K	19.6K41.2K
5%	244K612K	62.5K143K	16.3K33.9K	7.56K15K
10%	116K275K	29.5K63.8K	7.68K15.2K	3.54K6.81K

Read along a row and the numbers fall quickly as the lift grows. That's why a site with modest traffic should test bold changes. Small tweaks usually make small differences, and small differences take the most visitors to see.

Run whole weeks, and at least one#

Visitors on a Monday morning aren't the visitors on a Saturday night. On most sites, traffic and conversion rates change through the week, so a test that runs from Tuesday to Friday measures weekday visitors and tells you little about anyone else. Rounding up to whole weeks gives every day of the week the same weight, and the calculator does it for you.

Two more reasons not to stop at the first moment the numbers allow:

• Novelty fades. Returning visitors may click a new design because it's new. After a few days that wears off, and an early lift can shrink.
• Unusual days happen. A sale, a newsletter, or a mention somewhere sends visitors who behave differently from the rest. A longer test dilutes them.

Read once, or check as you go#

A standard test makes its promise about one look, at the size you planned. Check it every day and stop when it looks good, and you'll name false winners far more often than 5% of the time: in our simulation of 4,000 tests, 20.9% when checked daily. Why peeking at A/B tests finds false winners shows why.

That leaves two honest ways to run a test:

• Read once. Fix the number of visitors in advance, don't act on the results until you have them, and then decide. This needs the fewest visitors.
• Check as you go, with statistics built for it: anytime-valid intervals, which hold however often you look. The calculator's second column plans for them, as Mockly's results do. To be as sure by the planned end, they need two to three times the visitors, but you can stop as soon as the result is clear, which for a large difference, good or bad, comes much sooner.

When the answer is too long#

If the calculator says four months, a four-week test won't tell you much. Don't run it anyway and trust whatever comes out. Change the test instead:

• Test a bolder change. A new offer or a new headline can move a rate a long way. A small tweak, if it moves it at all, moves it a little, and a little takes the most visitors to detect.
• Measure an earlier step. Clicks on a "Start trial" button happen far more often than paid subscriptions, so a test judged on them gets an answer sooner. Keep an eye on the later step too: in Mockly, as a secondary goal or a guardrail.
• Test where the traffic is. A home page or a busy landing page gets an answer long before a page deep in the site.
• Test fewer versions. Two versions get there fastest.
• Plan for a bigger lift. The calculator's last table shows the smallest lift your traffic can detect in one, two, four, and eight weeks. If 25% in four weeks is the best it can do, test changes that could plausibly deliver it.

Can you run a test too long?#

Yes. Past the length it needs, a test costs more than it tells you:

• Visitors drift between versions. People clear cookies, switch devices, and browse privately, then come back as new visitors who may see the other version. The longer the test, the more of them there are. Safari, for one, caps cookies that scripts write at seven days, which limits how long a testing tool that runs in the browser can remember a visitor.
• Everything else changes. Seasons, campaigns, and other changes to the site mix into the result.
• Search engines notice. Google asks sites to run experiments only as long as necessary, and warns that it may act on tests that run unnecessarily long. Does A/B testing hurt SEO? covers the rest of what it asks.
• You wait instead of learning. Every week spent on an inconclusive test is a week not spent on the next one.

If the plan says eight weeks or more, rethink the test before you start rather than after.

How Mockly does it#

Every experiment in Mockly shows How long to run with its results: the visitors and days it needs to detect a 5%, 10%, or 20% lift, at its own conversion rate and the traffic it's actually getting. It's the planner behind this post's calculator. The results use anytime-valid intervals, so the verdict is safe to act on the day it appears, and a check of the traffic split runs alongside.

Mockly's visitor cookie comes from your site's server, in the same response as the page, rather than from a script, so Safari's seven-day limit on script cookies doesn't cut it short. Visitors keep their version for as long as the cookie lasts, up to a year. Reading results explains the planner and the verdicts.

Frequently asked questions#

What's the minimum time to run an A/B test?#

One full week, whatever the numbers say, so that every day of the week is in the result. Even a busy page that has all the visitors it needs in two days should run the week out.

How many visitors do I need for an A/B test?#

It depends on your conversion rate and the smallest lift you want to detect: from a few thousand, for a large lift on a common action, to millions, for a small lift on a rare one. The table above covers common cases, and the calculator covers yours.

What is a minimum detectable effect?#

The smallest change in conversion rate a test is planned to detect reliably, usually with an 80% chance. A test can still find a smaller change, but it isn't likely to, and planning for a smaller one takes more visitors.

Can I stop a test early if it's clearly winning?#

With a standard test, no: stopping at the first significant result is peeking, and it multiplies false winners. With anytime-valid statistics, yes, that's what they're for, though running whole weeks is still wise.