Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

PublishingResearch ArticlePeer reviewed

Statistical Power in Agent Benchmarks: How Many Runs Are Enough?

Version 2 · 3 reviewer reports · 14 August 2026 · 10.xxxx/ctm.pb.2026.0008

Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.

Abstract PDF 2,740 1,1033 cited

PublishingResearch ArticlePeer reviewed

Refusal Calibration in Instruction-Tuned Language Models: When should a model decline, and how well do current models know?

Nadia Haddad, Hana Kovács

Version 2 · 3 reviewer reports · 9 July 2026 · 10.xxxx/ctm.pb.2026.0007

Instruction-tuned models decline requests for many reasons, but little is known about how well their refusals track the actual risk of a request. We build a labelled set of 6,400 requests spanning benign, borderline and harmful categories and measure calibration of refusal decisions across eight open models. Refusal rates correlate only weakly with annotated risk for borderline requests, and over-refusal is concentrated in a handful of phrasing patterns rather than topics. A light-weight calibration step that uses the model's own confidence reduces over-refusal by 31% at no measurable cost to harmful-request refusal. We release the labelled set and the annotation guidelines.

Abstract PDF 3,180 1,2604 cited

PublishingDataset & BenchmarkPeer reviewed

CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents

Kofi Mensah, Marcus Tan, Ingrid Novak

Version 2 · 3 reviewer reports · 12 June 2026 · 10.xxxx/ctm.pb.2026.0004

Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.

Abstract PDF 5,330 2,4129 cited

PublishingReview ArticlePeer reviewed

A Survey of Evaluation Practices for Language-Model Agents: What is measured, how it is measured, and what is missing

Kofi Mensah, Ingrid Novak, Lena Hartwell

Version 2 · 4 reviewer reports · 22 May 2026 · 10.xxxx/ctm.pb.2026.0003

As language-model agents move from demonstrations to deployments, the way they are evaluated determines which claims can be trusted. We review how agents are evaluated in the literature, organising practice along four axes: the task environment, the success criterion, the treatment of variance, and the reporting of cost. We identify recurring weaknesses: success is often judged by another model without validation, runs are rarely repeated, cost and latency are reported in under a third of studies, and environment versions are seldom pinned. We close with a checklist for evaluation reports and open problems, including measurement of safety-relevant side effects.

Abstract PDF 6,210 2,87011 cited