Skip to main content
Submit

Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

6 results for “benchmark” in keywordSearch & filters ↓

PreprintsReview ArticleNot peer reviewed

A Taxonomy of Evaluation Leakage in Language-Model Benchmarks

Nadia Haddad, Ingrid Novak, Kofi Mensah

Version 2 (updated) · 14 October 2026 · 10.xxxx/ctm.pp.2026.0004

Benchmark scores can be inflated when test material reaches the training data or when evaluation choices quietly favour a model. We review 140 papers and incident reports and organise the mechanisms of evaluation leakage into a taxonomy with four top-level classes and eleven sub-classes. For each class we summarise how it can be detected, how often it has been reported, and what practices reduce its likelihood.

Abstract PDF 3,350 1,4105 cited
PublishingDataset & BenchmarkPeer reviewed

CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents

Kofi Mensah, Marcus Tan, Ingrid Novak

Version 2 · 3 reviewer reports · 12 June 2026 · 10.xxxx/ctm.pb.2026.0004

Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.

Abstract PDF 5,330 2,4129 cited
BooksMonograph

Evaluating Language Model Agents: Principles, metrics and practice

Lena Hartwell, Kofi Mensah

1st edition · 312 pages · ISBN 978-0-00-100001-2 · 30 September 2026 · 10.xxxx/ctm.bk.2026.0001

A practical and rigorous guide to measuring what agents actually do. The book develops a measurement framework for tool-using language agents, covers reliability, cost and safety metrics, and shows how to design benchmarks, report variance and avoid the most common evaluation mistakes. Worked examples and code accompany every chapter.

Read book PDF 5,240 2,8606 cited
PublishingReview ArticlePeer reviewed

A Survey of Evaluation Practices for Language-Model Agents: What is measured, how it is measured, and what is missing

Kofi Mensah, Ingrid Novak, Lena Hartwell

Version 2 · 4 reviewer reports · 22 May 2026 · 10.xxxx/ctm.pb.2026.0003

As language-model agents move from demonstrations to deployments, the way they are evaluated determines which claims can be trusted. We review how agents are evaluated in the literature, organising practice along four axes: the task environment, the success criterion, the treatment of variance, and the reporting of cost. We identify recurring weaknesses: success is often judged by another model without validation, runs are rarely repeated, cost and latency are reported in under a third of studies, and environment versions are seldom pinned. We close with a checklist for evaluation reports and open problems, including measurement of safety-relevant side effects.

Abstract PDF 6,210 2,87011 cited