Skip to main content
Submit

Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

3 results for “evaluation” in keywordSearch & filters ↓

PreprintsReview ArticleNot peer reviewed

A Taxonomy of Evaluation Leakage in Language-Model Benchmarks

Nadia Haddad, Ingrid Novak, Kofi Mensah

Version 2 (updated) · 14 October 2026 · 10.xxxx/ctm.pp.2026.0004

Benchmark scores can be inflated when test material reaches the training data or when evaluation choices quietly favour a model. We review 140 papers and incident reports and organise the mechanisms of evaluation leakage into a taxonomy with four top-level classes and eleven sub-classes. For each class we summarise how it can be detected, how often it has been reported, and what practices reduce its likelihood.

Abstract PDF 3,350 1,4105 cited
PublishingResearch ArticlePeer reviewed

Statistical Power in Agent Benchmarks: How Many Runs Are Enough?

Hana Kovács, Kofi Mensah, Carlos Medina

Version 2 · 3 reviewer reports · 14 August 2026 · 10.xxxx/ctm.pb.2026.0008

Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.

Abstract PDF 2,740 1,1033 cited