How Mockly works

Mockly splits the work in two. Publishing does everything that can be decided ahead of time: it checks your code, renders every variant of every page, and builds the scripts and the stylesheet. That leaves the server very little to do when a visitor arrives: pick the visitor's variants and join pieces of HTML that already exist.

This page follows both steps, then covers what gets recorded and how the preview differs from the live site.

When you publish#

Publishing runs over the whole site, in this order:

1. Check. Every page and component is checked against the language's rules and type-checked as TypeScript, the same checks the editor runs as you type. Any error stops the publish. Warnings don't.
2. Fetch data. Pages that use fetchJson() get their data now, with the secrets saved in the studio. A request that fails stops the publish. See Data from APIs.
3. Render every variant. Each page is rendered to HTML. Where it has an experiment, each variant of that part is rendered on its own. A route with parameters, such as /posts/[slug], becomes one page per entry.
4. Build scripts. Each interactive part, or island, is compiled to JavaScript. A page loads only the code for the islands it renders, and code that pages share downloads once. A page without islands loads less than 1.25 KB of JavaScript, for goal tracking and timings.
5. Build a stylesheet with exactly the utility classes the site uses.
6. Collect files. The version records your files in public/ as they are now, so replacing one in the editor doesn't change the live site. Resized copies of images are made where they're missing, and robots.txt and sitemap.xml are written from your site settings.
7. Save the version. Everything is saved together as a new version, which becomes the live site in one step. Experiments new to the site start running, and experiments whose variants or goal changed start over.

A page with an experiment is stored as fixed pieces of HTML with a slot between them, and a finished fragment for each variant. The page from the Quickstart looks like this:

piece      <!doctype html> … <main class="…">slot       launch-headline             control   <h1 class="…">Plan every launch in one place</h1>             outcome   <h1 class="…">Launch on time, every time</h1>piece      <p class="…">Checklists, owners, and dates your team can see.</p> … </html>

Each experiment adds a slot, not a copy of the page. A page with three experiments of two variants each stores six fragments, rather than eight whole pages, one for every combination.

When a visitor arrives#

For each request for a page, the server:

1. Finds the site's live version, which it keeps in memory.
2. Matches the path to a page. A path with a trailing slash redirects to the same path without it. A path that matches no page gets your pages/404.tsx, or a default not-found page, with status 404.
3. Reads the visitor's cookies, and gives a first-time visitor an ID.
4. Picks a variant for each experiment on the page.
5. Joins the page's pieces with the chosen fragments, and sends the result.
6. Records the page view and the experiments the visitor saw, without making the response wait.

Nothing is rendered or parsed while the visitor waits. The Performance tab in Analytics shows how long this takes, and each response carries the time in a Server-Timing header, which shows in the browser's network panel.

Your page code never sees the request. Pages are rendered when you publish, so a query parameter such as a UTM tag is recorded, but can't change what the page shows.

Pages are sent with Cache-Control: private, no-store, since each visitor may get a different variant. A version's scripts and stylesheet have the version in their address, so browsers cache them for good.

How a variant is chosen#

On a first visit, the server gives the browser a random visitor ID in a cookie. For each experiment, it combines the experiment's name, its revision, and the visitor ID into a number from 0 to 9,999. The weights divide that range among the variants, in the order they're declared: with weights of 70 and 30, the first variant gets 7,000 of the numbers and the second gets 3,000.

The same visitor always gets the same number, so they see the same variant on every page and every visit. An experiment that appears on several pages under one name is one experiment, and each visitor gets the same variant on all of them. Across many visitors the numbers spread evenly, so the split matches the weights.

The server also keeps each visitor's variants in a second cookie. When you change an experiment's weights in the studio, visitors who already have a variant keep it, and new visitors are split at the new weights.

The experiment's status, set in the studio, comes first:

When a publish changes an experiment's variants or its goal, the experiment starts a new revision. The revision is part of the number, so everyone is assigned again and the counts start fresh. Restart results in the studio is different: everyone keeps their variant, and only visitors from then on are counted.

Cookies#

Each cookie belongs to the site's own path. mk_visit lasts 30 minutes from the last page view, so it ends with the visit; the others last a year. mk_vid, mk_a, and mk_visit are HttpOnly, so scripts on the page can't read them. The server checks every value it reads: a tampered or malformed value is ignored, and a bad visitor ID is replaced with a new one.

Why nothing flickers#

Testing tools that run in the browser usually load the page first, then swap in the variant, so visitors can see the original for a moment, or a blank page while the swap waits. Mockly picks the variant before the response leaves the server, so the HTML that arrives already is the visitor's variant. Pages show in full without JavaScript. Only interactive parts and click goals need it. A/B test flicker: what causes it and how to fix it goes further.

What gets recorded#

As it serves a page, the server records:

• a page view: the route, the time, the referring site, UTM tags, and the device, browser, and OS
• an exposure for each running experiment on the page: which variant this visitor saw
• the page's goal, if its <Page> has one

The page's script sends two kinds of report to your site, never to anyone else:

• click goals, from elements with track, as each click happens
• engagement: how long the page was visible, how far it was scrolled, and how fast it loaded, sent when the page is hidden or closed

Form submissions go to your site too, and the server records the form's goal itself, so it counts with or without JavaScript.

A conversion counts for an experiment when a visitor reaches its goal at or after the first time they saw the experiment. Each visitor counts once, in the variant they saw first. See Reading results, and for what's stored and what isn't, Analytics and performance.

Bots#

Crawlers, link previews, uptime monitors, HTTP libraries, headless browsers, and requests without a user agent get the same page as anyone else, but leave no page view and no exposure, so they can't dilute your results. Bots are recognized by their user agent, so a bot that claims to be a browser is counted. What a crawler sees is chosen by the same rules as for anyone else, which is what search engines ask of tests: see Does A/B testing hurt SEO?

The preview and the live site#

The preview renders your pages the way publishing does, with these differences:

Edits in the editor reach visitors only when you publish. Changes to an experiment's status or weights in the Experiments tab are different: they apply to the live site on the next page view, without publishing.