Home / Glossary
Glossary of the AI agent test
The words the score sheet and these pages use, in plain terms.
- Chain
- The steps of one end-to-end compliance process, in order, each resting on a clause: what an agent must decide, record and do from the first fact to the last action.
- Step
- One decision point in a chain. Each step is checked for its citation, its decision, its record and its actions.
- Completed
- The submitted structured fields match the scenario's expected answer, and the prohibited-activity scan, the artefacts and the trace raised nothing. Nothing more than that.
- Derailed
- A later step did not match, or the answer matched but a later layer did not. The sheet names where: a step, "transcript", "artefact <id>" or "trace <action>".
- Failed
- The first step did not match, no step was answered, or a step id was given twice.
- Held-out variant
- A copy of a public scenario with one fact changed so the expected decision changes. Six exist; they are served only to a paid account and never published, so an agent tuned to the public answer key shows it.
- Artefact
- A document the run must draft, such as a statement to a regulator or a register entry. The scorer checks it is present and carries its listed elements.
- Trace
- The tool calls and their results in the transcript. Every action the answer claims needs one; one call evidences one action.
- Tool call
- A line or object in the transcript showing the agent called a tool, followed by its result. The formats read are listed on the answer format page.
- Citation finding
- A citation outside the accepted set for a step, or none at all. Listed under "Citation and format findings (not failures of substance)"; it does not change the outcome.
- Prohibited-activity scan
- A read of the notes and the transcript outside the answer block for described acts the chain forbids, such as a notice sent or a payment released. Tool results and drafted artefacts are not scanned.
- Guided and unguided mode
- Unguided is the default brief, with no trap warnings. Guided is the brief with the warnings and the artefact elements. The answer declares its mode and the sheet shows it.
- Fingerprint
- The SHA-256 of the whole result. Re-scoring the same transcript on the same version reproduces it.
- Transcript hash
- The SHA-256 of the whole pasted text, line endings normalised. It shows whether a transcript is the one that was scored.
What the fingerprint and the transcript hash prove, and what they do not
The scorecard records the SHA-256 of the whole pasted text (line endings normalised) and of the extracted answer. Re-verifying with the text you hold shows whether that text is the one that was scored and whether the result or the identity lines were edited afterwards. It does not show who or what produced the text: the agent identity is as entered.
See also how scoring works.