A/B Tests

An A/B test shows different versions of the same page to different visitors and measures which one converts better. In Moonship a test is made of three things: the page it runs on, the goal it should move, and two or more variants — the untouched original plus the changes you want to try.

Everything runs on the tracking snippet you already have. There's no second script to install and no code to write: variants are defined in the visual editor, and the snippet applies them in the browser. Tests live on the A/B Tests page.

Creating a test

Click New test and fill in:

• Test name — how you'll recognize it, e.g. “Shorter hero headline”.
• Hypothesis (optional) — what you expect to happen and why. Worth writing: it's what stops a losing test from being quietly reinterpreted as a win.
• Page to test — a full URL or a bare path. Only the path is ever matched (no scheme, host, query string, or hash), so the same test keeps working on staging and production.
• Match — exactly this path, path starts with, or path contains. Use a prefix match to run one test across a whole section of the site.
• Goal to measure — an existing goal. A session counts as converted if it completes this goal after being shown a variant.
• Variants — up to 6, with integer traffic weights that add up to 100. The first variant is always the untouched original and acts as the control.

A test is created as a draft, so nothing is served to visitors until you start it.

Building variants in the visual editor

Open Visual editor from a test's menu. It loads your real page next to a variant switcher — click any element to change it, and the change is recorded against that variant.

From the click-to-edit panel you can rewrite an element's text, change a link's URL, set text colour, background, and font size, or hide the element entirely. For anything the panel doesn't cover, use Add a change by selector to enter a CSS selector and pick the operation yourself:

• Replace text — swaps the element's text content
• Replace inner HTML — swaps its markup
• Hide element — removes it from the layout
• Set a CSS property — one property and value
• Set an attribute — e.g. href, alt, aria-label, placeholder, class

The control can't be edited — a patched control would make “no change” the wrong baseline. Each variant holds up to 30 changes.

Changes are content changes, not code. Event-handler attributes and the attributes that load code (src, srcdoc, integrity, on*, and friends) are rejected, so a test can never introduce script onto your site.

The editor preview renders your page server-side and serves it script-free, so anything a framework draws at runtime may be missing from the preview. Selectors are always matched against the real page when the test runs, and the editor tells you if a saved change didn't match anything.

Starting, pausing, and completing

A test moves through draft → running → paused → completed. Before it can start, Moonship checks that it will actually measure something:

• a goal is linked
• there are at least 2 variants
• at least one non-control variant defines a change
• the target page is set

If any of those is missing, the Start button explains what's needed. Once started, allow up to a minute for the snippet to pick the test up. Pause stops serving variants but keeps the data; resuming continues the same test.

While a test is running you can still rename variants, change their weights, and edit their changes — but variants can't be added or removed, because their identities are already stamped onto historical exposure data. Pause the test first. A completed test is locked.

Reading the results

Each test shows a row per variant, computed over the test's whole run rather than the dashboard's date range — an A/B test's numbers only mean anything across the full period:

• Sessions — sessions shown this variant
• Conversions — of those, how many completed the goal
• Conversion rate — with a 95% Wilson interval drawn beside it, so you see the range the true rate plausibly sits in rather than a single number
• Lift — relative change against the control
• Confidence — from a two-sided two-proportion z-test against the control
• P(beats control) — the Bayesian probability this variant is genuinely better; hidden below 10 sessions per arm, where the estimate would look more confident than it is

Above the table, a progress bar shows how far the test is toward the per-arm sample size needed to resolve a 5-point change at 95% confidence and 80% power. The target is derived from the control's own observed rate, so it appears once there's enough data to estimate a baseline. Below the table, a one-line read-out says where the test actually stands — still collecting, a variant ahead at your confidence level, or enough traffic gathered with no arm separating (which is itself a result: the change likely doesn't matter much).

Ask why hands the test and its numbers to Ask for a plain-English read and a suggested next step.

How measurement works

The session is the unit of analysis, not the visitor. Visitor identity is derived from a hash that rotates daily for privacy, so counting visitors per arm would double-count anyone who came back across a day boundary. Variant assignment is separately stable — it uses a date-free hash — so a returning visitor keeps seeing the same variant even though they count as a new session.

A session is counted in an arm only once it has been exposed: the snippet confirms the variant's changes actually landed on the page before recording exposure. If a selector has gone stale, the visitor is seeing the original and isn't counted as having seen the variant — otherwise a real change would be reported as “no effect”. A session that fires the goal before its first exposure isn't counted as a conversion, since the change can't have caused it.

Bot filtering follows your site's setting, the same as the rest of your analytics. Nothing about A/B testing is precomputed — results are calculated from your event data each time the page loads.

Shipping a winner

When you've decided, pick Ship <variant> from the test's menu. That records the winner and completes the test.

It does not keep serving the change to 100% of your traffic. Make the winning change in your own codebase — otherwise you'd be left with a “winner” that quietly stops being served the day the test is deleted.

Performance and privacy

• The A/B code is only sent to sites with a running test. Sites that aren't testing anything don't pay for it.
• Each visitor's assignment is resolved on the server and inlined into the script, so the variant is expressed as early as possible. Changes CSS can express are applied at first paint; elements that still need JavaScript are hidden for a moment so the original never flashes, with a hard deadline that reveals them regardless — a broken selector can never leave your content invisible.
• Only the arm a visitor is actually in is sent to the browser. The other variants' changes never leave the server.
• Assignment is computed from the same cookie-free hash the collector already uses. Nothing is stored on the visitor's device, so A/B testing doesn't change your cookie-banner position.
• Single-page apps are handled: on client-side navigation the changes are re-applied for the new path, and the visitor's assignment is unchanged.