Stockertim Dictionary · Artificial intelligence · Cybersecurity
Red teaming
Also called red-team exercise, adversarial testing
In short
Red teaming is the practice of deliberately testing a system by acting as an attacker or misuser, to find weaknesses before others do.
- Intermediate
- Process
- Checked
In detail
Red teaming is structured, authorised testing in which a group takes the role of an adversary. For AI systems, red teamers try to make a tool reveal data it should protect, take actions it should not, produce harmful or unfair content, or fail under unusual inputs. The aim is to discover problems in a controlled setting, so they can be fixed before real users or attackers find them.
A red-team exercise starts with a scope agreed in writing: which system, which kinds of harm, what is off limits and who must be told. Testers often work from the threat model, trying each listed risk and looking for ones nobody thought of. They record every finding with how they found it, how serious it is and what might fix it, and the system owner tracks each one to closure.
Red teaming matters because ordinary testing checks that a system does what it should, while red teaming checks what else it can be pushed into doing. Teams with varied backgrounds find more, because harms differ for different users. Exercises are repeated after major changes, since a fix in one place can open a gap elsewhere.
At work, the term covers everything from formal exercises by specialists to small structured sessions where a team tries to misuse its own tool, always with permission and within the agreed scope.
Worked example
Northwind Bikes runs a red-team exercise on its customer chat assistant before launch.
- Findings recorded: 14, of which 4 were rated serious
- Findings fixed before launch: 9, so 9 / 14 = 64.3%
- Serious findings fixed: 4 / 4 = 100%
- The remaining 5 minor findings get owners and dates, and a retest is planned after the next update
Every serious weakness is fixed before launch, and the rest are tracked to closure.
In practice
Security and product teams red-team new AI tools before launch and after big changes, work from a written scope, record findings in a tracker and retest fixes.
Watch out
- Red teaming is not unauthorised hacking; it needs written permission and an agreed scope.
- Finding no problems in one exercise does not prove a system is safe; it shows only what was tested.
Learn it properly
Official sources
- AI Risk Management Framework · National Institute of Standards and Technology
- Guidelines for secure AI system development · National Cyber Security Centre
- ATLAS: adversarial threat landscape for AI systems · MITRE
Facts last checked on 3 November 2026.
Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.