Skip to content

Stockertim Dictionary · Artificial intelligence · Technology

Live sampling

Also called production sampling, live output sampling, output sampling

In short

Live sampling is the regular checking of a small random selection of a running system's real outputs against their source documents, to catch errors the test set did not foresee.

  • Intermediate
  • Process
  • Checked

In detail

A test set shows how a system performs on cases someone thought of in advance. Real work brings cases nobody thought of: a new supplier's layout, an odd customer request, a seasonal product. Live sampling watches for these. On a regular cycle, often weekly, a person takes a random selection of the outputs the system actually produced and checks each against its source, using the same rubric as testing.

Randomness is essential. Picking items that look interesting, or only those flagged as uncertain, gives a distorted picture; a random draw gives every output the same chance of being checked, so the error rate in the sample is a fair estimate for the whole. Many teams add a second, targeted sample of high-value or unusual items on top, reported separately.

The size of the sample is a balance between effort and precision. A few dozen items a week will catch a common problem quickly but can easily miss a rare one, and the error rate from a small sample is only approximate. Results are tracked over time, because a gradual rise often signals a change in the inputs or in the system's behaviour before any complaint arrives.

Each error found is logged with its cause and, where it reveals a new kind of case, added to the test set, so the sampling also keeps testing up to date.

Worked example

Heron Bay Logistics samples the output of an automation that reads delivery notes, which handles 2,000 notes a week.

  1. Sample of 2% chosen at random: 2,000 x 0.02 = 40 notes
  2. Errors found against the source documents: 3
  3. Sample error rate: 3 / 40 = 7.5%
  4. Two of the errors come from a carrier's new note layout, which is added to the test set.

The weekly sample flags a change no test covered, at a cost of 40 checks rather than 2,000.

In practice

Teams draw a random sample of live outputs on a fixed schedule, check them against sources with the testing rubric, chart the error rate week by week, and feed every new type of failure back into the test set.

Watch out

  • Sampling only flagged or interesting items does not estimate the overall error rate.
  • A small sample can miss rare but serious errors; pair it with targeted checks on high-risk items.

Learn it properly

Official sources

Facts last checked on 9 November 2026.

Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.

English