Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Abstract
Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.
Keywords statistical power · agent evaluation · variance · sequential testing
Full text
CTM-PB-2026-0008-v2.pdf
Declarations
- Author contributions
- Author contributions are listed in the PDF.
- Funding
- Sample record – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
Peer review
Open reports from independent reviewers. See how outcomes are determined.
- Status
- Peer reviewed
- Requested
- 14 Mar 2026
- Completed
- 12 Aug 2026
- Report identity
- Signed reports
Reports on version 1
- Approved
Dr. Linh Tran
Statistically careful and directly useful to benchmark users. The sequential procedure is well explained.
Strengths
- Clear derivations.
- Open code.
Concerns
- Minor notation inconsistencies.
- Approved with reservations
Dr. Pavel Novotný
I would like to see the procedure applied to a benchmark that was not used to design it.
Strengths
- Practical stopping rule.
Concerns
- No held-out benchmark for validating the procedure.
Authors' response
We added a fifth, held-out benchmark in Section 6.
Reports on version 2Current version
- Approved
Dr. Pavel Novotný
The held-out benchmark confirms the method.
Strengths
- Evidence now sufficient.
Concerns
- None.
Versions
Version 2 · 14 August 2026Latest
Revised after review: held-out benchmark added.
10.xxxx/ctm.pb.2026.0008.v2
Version 1 · 12 March 2026
Posted; peer review requested.
10.xxxx/ctm.pb.2026.0008.v1
How to cite
Kovács, H., Mensah, K., & Medina, C. (2026). Statistical Power in Agent Benchmarks: How Many Runs Are Enough?. CONTINUUM Publishing, CTM-PB-2026-0008. doi:10.xxxx/ctm.pb.2026.0008
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.