Reading results

Each experiment's results come with a verdict in plain words, a row of numbers for each variant, and an estimate of how long it still needs. The intervals stay valid however often you check, so you can look every day, or every minute, and act as soon as the verdict says you can.

This page explains what each part means. The math stays light: enough to read the numbers correctly.

Where to find them#

In the studio, open Analytics, then the Experiments tab, or click Full results in the editor's Experiments tab. The list shows every experiment with its status, verdict, and visitors. Select one to see it in full. The Overview tab has a line for each experiment too.

Results cover the experiment's current revision, from when its results last restarted, whatever period the page is set to. The page refreshes every 30 seconds while it's visible.

Verdicts#

The verdict is the first thing on an experiment's results, and the list repeats it in a few words:

A winner reads like this:

bold converts better than control+12.5% (+3% to +23%) in conversion rate. The interval stays valid however often you check, so you can decide now.

A variant that converts worse adds "Consider pausing the experiment or shipping control." When there's no clear winner yet, the verdict says how big a difference the experiment can already rule out:

No clear winner yetA lift of 39% or more would very likely have shown by now; smaller differences need more visitors. You can keep checking: the intervals stay valid.

No clear winner isn't evidence that the variants convert the same. It means any difference is smaller than the experiment can see so far.

The numbers#

Under the verdict are the totals: visitors, conversions, who's ahead so far (and whether that's significant yet), and how long the experiment has been running. Then there's a row for each variant:

Hover over a lift or a difference to see what it means, how its interval is adjusted, and its p-value.

Rates#

Each rate comes with a 95% interval (a Wilson interval): the range of rates that fit what's been seen so far. It describes one variant on its own. To compare a variant with control, read its lift and difference.

Lift and difference#

Lift is relative: a rate of 6% against control's 5% is a lift of +20%. The difference is the gap in percentage points: +1 pts.

Each has a 95% interval, such as +20% (+4% to +38%). When the interval doesn't include zero, the difference is significant, and that's what the verdict is based on. The lift's interval and the difference's interval always agree on whether it's significant.

Until there's enough data, the interval reads "Needs more data". While control's own rate is too uncertain to bound a relative change, the lift's interval reads "Can't bound yet", but the difference still has one.

Intervals that stay valid however often you check#

Most A/B testing statistics assume you look once, at a number of visitors you picked in advance. If you check such a test every day and stop the first time it says significant, you name false winners far more often than the 5% it promises, because every look is another chance for noise to cross the line. This is called peeking.

Mockly's intervals for lift and difference are confidence sequences: intervals built to hold at every moment, not only at one planned moment. Across all your looks together, the chance of a false winner stays at 5% or less. Mockly checks this with simulations that look after every single visitor. So you can check as often as you like, and stop as soon as the verdict names a winner.

Why peeking at A/B tests finds false winners shows how much this matters, with 4,000 simulated tests.

Several variants#

With more than one variant besides control, each is a separate comparison, and each is another chance for a false winner. Mockly splits the 5% between them, a method called the Bonferroni correction. With control and two other variants, each comparison is held to 2.5%. The intervals shown are already adjusted, so they always agree with the verdict, and their explanation says so: "adjusted for 2 comparisons with control". A variant with a weight of 0 isn't compared.

When results start#

A comparison needs enough data before its interval means anything. The smaller of the two groups, the variant or control, needs enough visitors to expect 10 conversions, and 10 visitors who don't convert, at their combined rate so far. At a 5% conversion rate, that's 200 visitors.

Until then, the verdict is "Not enough data yet", with a bar that shows the visitors in the smallest variant against the number needed:

Not enough data yetResults start once the smallest variant has about 200 visitors, enough to expect 10 conversions at the current rate. So far: 120.

With no conversions at all, it says that results start once visitors convert. Charts of the conversion rate and visitors over time start once the experiment has visitors on two different days.

The traffic split check#

If visitors don't split the way the weights say, something treats the variants differently, and the results can't be trusted. Mockly checks the split with a test that, like the intervals, stays valid however often it runs. It's strict: a correct split raises a false alarm in at most 1 experiment in 1,000.

When the split is off, the verdict reads "The traffic split doesn't match the weights", says which variant is furthest from its target and by how much, and holds the results back. Common causes are a redirect, caching, a variant that fails to load, or bots. Fix the cause, then restart the results from the experiment's card in the editor.

A variant with a weight of 0 that still gets visitors is always flagged.

How long to run#

How long to run shows how many visitors the whole experiment needs for an 80% chance to detect a lift of 5%, 10%, and 20%, if it's real, and how many more days that takes at the pace so far. It starts with the first conversions, uses the experiment's overall conversion rate so far as the baseline, and counts days once the experiment is a day old.

Small lifts take far more visitors than large ones. For example, with a 5% conversion rate and two variants split evenly:

A row reads "Enough now" once the experiment has that many visitors, and "Not practical" when the number is out of reach. While the experiment is paused there are no days, since it gets no visitors, and once it's shipped the estimate is gone.

The estimate is conservative: it ignores the chance that a real lift shows up early, so by the number it gives, the chance of detecting the lift is at least 80%. Decide on the smallest lift that would change what you do, and use that row as your plan. To plan before an experiment has visitors, use the calculator in How long should you run an A/B test?

What checking anytime costs#

Being valid at every look takes more visitors than a test read once at a planned size: about twice as many, at that size. For a 20% lift on a 5% conversion rate, that's 33,860 visitors, against about 16,300 for a test read once. In exchange, you don't have to fix a sample size in advance, a large difference can be called early, and checking never makes the result less trustworthy.

Restarting results#

Results restart, counting only visitors who arrive from then on, when you:

• save a new traffic split while the experiment has results, with Save and restart results
• click Restart results at the bottom of the experiment's card, for example after fixing what skewed the split

Everyone keeps their variant, and nothing is deleted: the visitors from before stop counting. The results then show when they restarted. Saving the same split again, or pausing and resuming, doesn't restart them.

A publish that changes an experiment's variant names or its goal goes further. It starts a new revision, assigns visitors again from scratch, and starts the results fresh. See Experiments.

Shipping a variant#

When a variant wins, ship it from the experiment's card: the Ship menu asks which variant, and from the next page view everyone sees it and nothing more is recorded. The results stay, without the estimate of how long to run. When a variant loses, you can ship control the same way.

To make the change permanent, replace the experiment in your code with the winning markup and publish. The experiment is then archived.

Revenue#

An experiment with metric="revenue" (see Goals) is decided on revenue per visitor: what its visitors sent with the goal, divided by every visitor, buyers or not. Its results change in a few places:

• The summary shows Revenue in place of conversions, with revenue per visitor.
• Each variant shows Per visitor, with the share converting, and its revenue, with revenue per buyer.
• Lift and difference compare revenue per visitor, with the same intervals as rates, which stay valid however often you check. The difference is an amount, such as +0.42 per visitor.
• Results start once the smallest variant can expect 10 buyers, as rates wait for 10 conversions.
• How long to run plans for lifts in revenue per visitor.

A note under the table says how much one visitor's revenue can count for. Once 100 visitors have bought, each one's revenue is capped at the 99th percentile of buyers', so a single huge order can't swing the result. Revenue per visitor varies far more than a conversion rate, so a revenue experiment usually needs more visitors.

Guardrails#

Guardrails (see Goals) get a table like secondary goals: each variant's rate on each guardrail, its lift and difference, and a line saying "mustn't drop" or "mustn't rise", with "no harm detected" or who hurts it.

When a variant significantly hurts a guardrail, the verdict says so, in place of any winner:

bold hurts signup, a guardrail

That comes first because shipping a variant that hurts a guardrail does harm the goal doesn't show. Only a traffic split that doesn't match the weights comes before it, since then nothing can be trusted. To stop such a variant on its own, turn on Pause automatically if a guardrail is hurt on the experiment's card. The studio checks experiments with guardrails every 15 minutes, and its paused note says why it paused.

Segments#

Segments, under the variants, splits an experiment's visitors by Device, New or returning, or Source, and compares the variants in each segment the same way as the whole. A visitor counts in the segment of the visit they first saw the experiment in. Sources show the largest seven by name and the rest as Other sources.

Segments are exploratory. With many segments, some look different by chance, so every interval is widened for all the comparisons on screen: one variant in three segments is three comparisons. Even so, use segments to find what to test next, such as a variant that seems to win on phones, and test it with targeting, rather than to pick a winner.

Visitors who first saw an experiment before Mockly recorded visits aren't in any segment, and the card says how many.

Past runs#

A run ends when you ship a variant, when its results restart, or when a publish starts a new revision or removes the experiment from your code. Each finished run is kept under Past runs, below the experiments on the Analytics page's Experiments tab, with how it ended, when, and its results exactly as they stood then. Later visitors don't change them. Runs that never had a visitor aren't kept.

Notes#

Each experiment has Notes above its verdict: what it tests and why. When a run ends, its notes go with it into Past runs, where you can edit them to record what you learned and what to try next. Notes hold up to 4,000 characters.

Secondary goals#

An experiment that watches secondary goals gets a Secondary goals table, with each variant's rate, lift, and difference on each one. They don't change the verdict, but a winner that converts worse on one says so. See Goals.