Stockertim Dictionary · Marketing · Technology
A/B test
Also called A/B testing, AB test, split test, split testing, A/B/n test, bucket test
In short
An experiment that shows two versions of a page, email or advert to similar people at random at the same time and compares which produces better results.
- Intermediate
- Process
- Checked
In detail
An A/B test is a controlled experiment. Visitors or recipients are split at random into two groups; one sees version A, the current design, and the other sees version B, which differs in one respect, such as the headline, the button, the price display or the length of a form. Because assignment is random and both versions run at once, outside influences such as the weather, the news or the day of the week affect both groups equally, so a difference in results can be attributed to the change.
Good tests change one thing at a time; altering several elements together shows which version won but not why. Each visitor stays on the same version if they return, so the groups remain clean. The test runs for at least one full weekly cycle, so weekday and weekend behaviour are both included, and until enough conversions have been collected for the difference to be trusted.
That last point is statistical. With small numbers, a gap between versions can easily be chance. Testing tools apply a significance test or a similar method, and the sample needed depends on the starting conversion rate and the size of difference worth detecting. Stopping a test the moment one version pulls ahead is a common way to fool oneself.
A/B testing works for any repeated, measurable action: landing pages, email subject lines, advert versions, sign-up flows. Variants with three or more versions are called A/B/n tests, and testing several elements in combination is multivariate testing, which needs much more traffic.
Worked example
A checkout page is tested for two weeks. Version A gets 5,000 visitors and 100 orders; version B, with a shorter form, gets 5,000 visitors and 130 orders.
- Version A conversion rate: 100 / 5,000 = 2%
- Version B conversion rate: 130 / 5,000 = 2.6%
- Relative uplift: (2.6% - 2%) / 2% = 30%
Version B produced 30 extra orders from the same traffic, an uplift of 30%, provided the test confirms the gap is unlikely to be chance.
In practice
Product, e-commerce and email teams keep a backlog of test ideas ranked by expected impact, run tests with dedicated software, and record every result, including failures, so that later tests build on what was learned.
Watch out
- Peeking at results daily and stopping when one version leads inflates false wins; decide the sample size first.
- A difference between 2% and 2.6% is 0.6 percentage points but a 30% relative uplift; say which is meant.
Learn it properly
Official sources
- What is experimental design? · National Institute of Standards and Technology
- Hypothesis tests · National Institute of Standards and Technology
- Sample sizes required · National Institute of Standards and Technology
Facts last checked on 3 November 2026.
Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.