Stockertim Dictionary · Artificial intelligence
Prompt test set
Also called evaluation set, test cases, golden set
In short
A prompt test set is a fixed collection of inputs, each with an expected result, used to compare prompt versions fairly on exactly the same cases.
- Intermediate
- Concept
- Checked
In detail
A prompt test set is to prompts what a set of exam questions is to students: a fixed collection of realistic inputs, each paired with the expected answer or the criteria a good answer must meet. Every prompt version is run on the whole set, the outputs are scored, and the scores are compared. Because the cases stay the same, a difference in score reflects the prompt, not luck in the inputs chosen.
A useful test set mirrors the real work. It includes ordinary cases in the proportions they occur, plus the awkward ones: missing data, ambiguous requests, very long inputs and anything that has caused errors before. Each time a new failure appears in live use, the case is added, so the set grows into a record of known pitfalls.
Test sets need care. If the same cases are used to tune the prompt again and again, the prompt can end up fitted to those cases and do worse on new ones. Keeping a few cases aside for a final check, and refreshing the set as the work changes, limits that risk.
Worked example
Lagoon Logistics compares two versions of a prompt that answers delivery queries, on the same 40 test cases.
- Version A: 29 / 40 = 72.5% correct
- Version B: 35 / 40 = 87.5% correct
- Difference: 35 - 29 = 6 more cases answered correctly
Version B is adopted, and its 5 failures are added to the notes for the next round of work.
In practice
Teams keep a test set for each important prompt, rerun it when the prompt, the model or the documents change, record the scores, and add every new failure seen in live use.
Watch out
- One good answer proves nothing; outputs vary, so judge on the whole set.
- Tuning repeatedly on the same cases can fit the prompt to them; keep some cases aside.
Learn it properly
Official sources
- NIST AI RMF Playbook · National Institute of Standards and Technology
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile · National Institute of Standards and Technology
Facts last checked on 3 November 2026.
Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.