Version 1 · 21 October 2026 · 10.xxxx/ctm.pp.2026.0007
Agent developers inspect failed runs by hand, which does not scale beyond a few dozen trajectories. We propose a clustering approach that groups failures by the step at which the trajectory diverges from a successful reference run, and label the clusters with short natural-language descriptions. On 18,000 trajectories from three agent frameworks, the method recovers 83% of failure categories identified by manual analysis and surfaces four categories that were previously unreported.
Version 2 (updated) · 14 October 2026 · 10.xxxx/ctm.pp.2026.0004
Benchmark scores can be inflated when test material reaches the training data or when evaluation choices quietly favour a model. We review 140 papers and incident reports and organise the mechanisms of evaluation leakage into a taxonomy with four top-level classes and eleven sub-classes. For each class we summarise how it can be detected, how often it has been reported, and what practices reduce its likelihood.
1st edition · 312 pages · ISBN 978-0-00-100001-2 · 30 September 2026 · 10.xxxx/ctm.bk.2026.0001
A practical and rigorous guide to measuring what agents actually do. The book develops a measurement framework for tool-using language agents, covers reliability, cost and safety metrics, and shows how to design benchmarks, report variance and avoid the most common evaluation mistakes. Worked examples and code accompany every chapter.
Version 1 · 19 August 2026 · 10.xxxx/ctm.pp.2026.0003
Several studies report that language models use information in the middle of a long context less reliably than information at the start or end. We replicate these findings across eleven models and four context lengths using a controlled retrieval task. The effect is reproducible but smaller than originally reported, and largely disappears when the task format is held fixed across lengths. We release the harness and the full per-position results.
Version 2 · 3 reviewer reports · 14 August 2026 · 10.xxxx/ctm.pb.2026.0008
Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.
Version 1 · 27 July 2026 · 10.xxxx/ctm.pp.2026.0002
Many machine-learning venues now require authors to complete a reproducibility checklist. We audited 240 papers that ticked the item “code is provided” and checked whether the linked repositories existed, ran and reproduced the headline table. 87% had a reachable repository, 58% contained instructions sufficient to run training or evaluation, and 31% reproduced the headline result within the reported variance. Compliance was higher for papers reporting variance across seeds. We recommend checks that can be automated at submission time.
Version 2 · 3 reviewer reports · 9 July 2026 · 10.xxxx/ctm.pb.2026.0007
Instruction-tuned models decline requests for many reasons, but little is known about how well their refusals track the actual risk of a request. We build a labelled set of 6,400 requests spanning benign, borderline and harmful categories and measure calibration of refusal decisions across eight open models. Refusal rates correlate only weakly with annotated risk for borderline requests, and over-refusal is concentrated in a handful of phrasing patterns rather than topics. A light-weight calibration step that uses the model's own confidence reduces over-refusal by 31% at no measurable cost to harmful-request refusal. We release the labelled set and the annotation guidelines.
RTUA 2026 · pp. 1–9 · 3 July 2026 · 10.xxxx/ctm.pr.2026.0001
Agents that retry failed tool calls recover from transient errors but can also compound mistakes and waste resources. We analyse 18,000 recorded agent runs and show that retries help for network and rate-limit errors but hurt for malformed arguments. We propose a retry budget that depends on the error class and show that it improves success by 6 points at equal cost.
RTUA 2026 · pp. 19–24 · 3 July 2026 · 10.xxxx/ctm.pr.2026.0003
Some tool failures return plausible but wrong output and raise no error. We detect such silent failures by comparing the distribution of tool outputs against a rolling reference and flagging shifts. On three deployed tools the detector finds 71% of injected faults with a 2% false-alarm rate.
RTUA 2026 · pp. 31–33 · 3 July 2026 · 10.xxxx/ctm.pr.2026.0005
We report experience building perturbation suites — rephrased instructions, reordered tool lists, noisy outputs — for three deployed agents, and summarise which perturbations revealed real failures and which only added cost.
Version 2 · 3 reviewer reports · 12 June 2026 · 10.xxxx/ctm.pb.2026.0004
Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.
Version 1.2 · CC BY 4.0 · 12 June 2026 · 10.xxxx/ctm.ar.2026.0001
1,200 tool-use tasks, each with five perturbed variants, together with the evaluation harness, a datasheet and the annotation guidelines. Accompanies the article “CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents”.
Version 0.4.0 · Apache-2.0 · 5 June 2026 · 10.xxxx/ctm.ar.2026.0007
Python library for measuring and improving calibration of generated text, including reliability diagrams, expected calibration error variants and post-hoc recalibration methods.
Version 2 · 4 reviewer reports · 22 May 2026 · 10.xxxx/ctm.pb.2026.0003
As language-model agents move from demonstrations to deployments, the way they are evaluated determines which claims can be trusted. We review how agents are evaluated in the literature, organising practice along four axes: the task environment, the success criterion, the treatment of variance, and the reporting of cost. We identify recurring weaknesses: success is often judged by another model without validation, runs are rarely repeated, cost and latency are reported in under a third of studies, and environment versions are seldom pinned. We close with a checklist for evaluation reports and open problems, including measurement of safety-relevant side effects.
Version 1.0 · CC BY 4.0 · 22 May 2026 · 10.xxxx/ctm.ar.2026.0006
Technical report documenting how 112 agent papers report variance, compute, prompts and judging procedures, with the full coding sheet. Supports the survey “A Survey of Evaluation Practices for Language-Model Agents”.
PhD thesis · 224 pages · ISBN 978-0-00-100005-0 · 15 May 2026 · 10.xxxx/ctm.bk.2026.0005
Doctoral thesis on estimating and communicating uncertainty for text generation. It develops calibration methods for retrieval-augmented systems, studies how users interpret expressed confidence, and proposes evaluation protocols for uncertainty in open-ended outputs.
