Subject area
11 records across all collections
We audit 2,400 checklist responses submitted over three years and compare them against the papers they accompany. Checks for compute reporting and variance were answered affirmatively in 78% of cases but supported by the paper in only 41%, suggesting that checklists work best when responses must point to a location in the text.
Fatima Zahra El Idrissi, Tomás Ibarra
Datasheets improve transparency but are time-consuming to write. We show that a template-driven workflow with automatic statistics halves the time to a complete datasheet and increases the completeness of answers on a fifteen-item rubric.
We study reports from 300 openly reviewed machine-learning papers. Open reports are longer, more specific and more often cite evidence than comparable anonymous reports from closed venues, and reviewers who sign their reports are no more lenient. We discuss limits of the comparison.
We attempt to build and run 400 research repositories released between 2019 and 2021. Only 38% still run without modification; the most common causes are unpinned dependencies and removed external data. Archived releases with lock files survive at more than twice the rate.
We argue that venues should offer an explicit track for well-designed studies that do not find the expected effect, and outline reviewing criteria that focus on design and power rather than outcome.
2,400 anonymised checklist responses with an independent assessment of whether each claim is supported by the accompanying paper. Basis of the OPENAIR 2026 audit.
Nadia Haddad, Ingrid Novak, Kofi Mensah
Benchmark scores can be inflated when test material reaches the training data or when evaluation choices quietly favour a model. We review 140 papers and incident reports and organise the mechanisms of evaluation leakage into a taxonomy with four top-level classes and eleven sub-classes. For each class we summarise how it can be detected, how often it has been reported, and what practices reduce its likelihood.
Hana Kovács, Kofi Mensah, Carlos Medina
Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.
Many machine-learning venues now require authors to complete a reproducibility checklist. We audited 240 papers that ticked the item “code is provided” and checked whether the linked repositories existed, ran and reproduced the headline table. 87% had a reachable repository, 58% contained instructions sufficient to run training or evaluation, and 31% reproduced the headline result within the reported variance. Compliance was higher for papers reporting variance across seeds. We recommend checks that can be automated at submission time.
Technical report documenting how 112 agent papers report variance, compute, prompts and judging procedures, with the full coding sheet. Supports the survey “A Survey of Evaluation Practices for Language-Model Agents”.
Fatima Zahra El Idrissi, Ravi Menon, Ingrid Novak
A step-by-step handbook for sharing datasets, code and models so that others can find, cite and reuse them. It covers licences, documentation, identifiers, repositories, privacy, and long-term preservation, with checklists that can be adopted by labs and venues.
