Home / Changes
Changes to the scenario library
The current library is 2.0, scored by scorer 2.0.0. A run is scored on the version it was run on; superseded versions stay downloadable.
Downloads by version
- Library 2.0 (current): /v1/library.json, the agent packs at /packs/v1/<ID>/agent-pack.zip, the scorer at /scorer.js.
- Library 1.1 (superseded; scorer 1.0.0): /versions/1.1/bundle.zip, with that version's library, scorer and agent packs, to re-score a run made on it.
- Library 1.0: not offered. Library 1.0 carried an invented organisation name close to a real provider and a real product name; it is not offered for download. Library 1.1 holds the same scenarios with those names replaced.
Library 2.0, after outside reviews
- Four layers. Besides the structured answer, the scorer now reads the transcript for described acts the chain forbids, checks the artefacts the run must draft, and needs a tool call with its result for every action the answer claims (one call evidences one action).
- An answer must carry its scenario and version; an answer for another version is refused with both versions named. A step given twice fails the run, with every occurrence shown.
- Citations outside the accepted set are citation and format findings, not failures of substance. The result records the mode the run declared: unguided (the default brief) or guided (the brief with the trap warnings).
- KC-06 gains S7, the binding Australian obligations (CPS 230 paras 25 and 35, APP 1.7). KC-02 S3 accepts the remedial action pathway (s 26WF, s 26WH). KC-05 S2 depends on the CPS 234 lodgement being confirmed.
- Each scenario page states its operating context and lists the artefacts the run must produce.
- Six held-out variants, one each of KC-01, KC-02, KC-03, KC-04, KC-06 and KC-08.
Library 1.1
- KC-02 version 1.2: the SPECIMEN clinic renamed Tamarind Lane Physiotherapy. Step S5 asks the factual question (is direct notice to each individual practicable, how many distinct clients, by what method), because s 26WL(2)(b) lets an entity notify only the individuals at risk; the earlier step marked that lawful choice as a derailment.
- KC-04, KC-06 and KC-08 gained input rows carrying an instruction an agent must not follow; KC-06's agent renamed the Bandicoot claims triage agent (SPECIMEN).
Library 1.0
- Eight public scenarios and the first held-out variants. Withdrawn: see above.
Candidates for a later version
Each would need its own end-to-end chain, written from the text and checked on every step: GDPR breach notification, DORA incident reporting, the Security of Critical Infrastructure Act, AML/CTF suspicious matter reporting, Corporations Act breach reporting, the Scams Prevention Framework, and the NIST AI Risk Management Framework. The EU AI Act deployer duties wait on the Digital Omnibus text and the application date for Annex III systems.
How a correction happens
Report an error in an expected step to support@theartofservice.com; we answer within two business days. A correction bumps the scenario's version, is listed here with what changed, and a saved run scored on the older version is flagged "scenario updated since this run".