2 results for “agent evaluation” in keywordSearch & filters ↓
Technical report documenting how 112 agent papers report variance, compute, prompts and judging procedures, with the full coding sheet. Supports the survey “A Survey of Evaluation Practices for Language-Model Agents”.
Hana Kovács, Kofi Mensah, Carlos Medina
Most agent benchmarks report one run per task, yet agents are stochastic. We analyse run-to-run variance on four public agent benchmarks and derive the number of repeated runs needed to detect a given difference between two systems at conventional power. For differences smaller than five points, single-run comparisons have less than 40% power on all four benchmarks. We provide a sequential testing procedure that stops early when the evidence is clear, and show that it halves the cost of repeated evaluation while keeping the false-positive rate below 5%.
