Skip to content

Stockertim Dictionary · Artificial intelligence · Technology

Pairwise comparison

Also called paired comparison, side-by-side comparison, head-to-head evaluation

In short

Pairwise comparison judges two versions of a system by running both on the same cases under the same settings and comparing their results case by case, not only by overall score.

  • Intermediate
  • Process
  • Checked

In detail

When a prompt, model or workflow is changed, the question is whether the new version is better. Comparing two overall scores from different occasions is weak evidence: the cases, settings or markers may have differed. A pairwise comparison removes those differences. Both versions run on the identical set of cases with the same settings, and each case is judged for both, giving a direct pair of results.

Looking case by case reveals what totals conceal. Two versions with the same overall pass rate may fail on entirely different cases, and a new version that fixes ten cases while breaking four is a different proposition from one that changes nothing. The usual summary counts cases where the new version is better, where it is worse and where there is no difference, then examines the worse cases one by one.

For outputs without a single right answer, a reviewer can be shown both outputs side by side and asked which better meets the rubric, ideally without knowing which version produced which. Hiding the labels and randomising which appears first reduce the tendency to favour the familiar or the first-seen.

Pairwise results are still limited by the cases chosen. A comparison on routine cases says little about difficult ones, and when the differences are small, the counts of better and worse cases can be within the range that chance alone would produce.

Worked example

Cobalt Recruitment compares two versions of a prompt that summarises candidate notes, using 50 cases.

  1. Reviewers see both summaries for each case, unlabelled and in random order.
  2. Version B better: 18 cases. Version A better: 7 cases. No difference: 25 cases.
  3. Check the total: 18 + 7 + 25 = 50
  4. The 7 cases where B is worse all involve notes with several job titles.

Version B is better on balance, and the comparison points to a specific weakness to fix before it replaces version A.

In practice

Teams run old and new versions on the same test set before every change, record better, worse and same counts, read every case that got worse, and hide version labels when people judge the outputs.

Watch out

  • Running the two versions at different times or with different settings undermines the comparison.
  • Reviewers who know which version is new tend to prefer it.

Learn it properly

Official sources

Facts last checked on 9 November 2026.

Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.

English