Home / Chain tests versus other evaluation
Chain tests compared with general-purpose evaluation tools, LLM-as-judge and spreadsheets
Each method answers a different question. This page sets out what each measures, so you can pick what fits the decision in front of you.
| Method | What it measures | How a result is reached | Repeatable |
|---|---|---|---|
| Chain test (this tester) | Whether an agent's decisions, records, actions and drafted documents on one published process match an expected chain, step by step, with the clause behind each step | Fixed rules in a published scorer, no model | The same transcript gives the same bytes and fingerprint every time |
| General-purpose evaluation tools | Whatever the user defines: accuracy on a dataset, latency, cost, regressions across versions | Metrics and graders the user configures, sometimes a model | Depends on the graders chosen |
| LLM-as-judge | How a second model rates an answer against a rubric, including free text and reasoning | A model reads the answer and assigns a grade | Two judgements of the same answer can differ unless the judge is pinned and deterministic |
| Spreadsheet of test cases | Whatever the reviewer records, usually pass or fail per case | A person reads each output | Depends on the reviewer |
Where a chain test does not reach
It does not judge the quality of free-text reasoning, it covers only the published scenarios, and it never runs or monitors an agent in production. Methods that grade prose or watch live traffic answer questions this one does not.
This shows which steps this agent completed on this scenario version. It does not show that the agent is safe, compliant or fit for any other task.