Research ArchiveSlides
Keynote Slides: Reliability Is a Measurement Problem
Slides of the opening keynote at the 1st Workshop on Reliable Tool-Using Agents, 12 June 2026.
View record 520 2440 cited
Search AEX CONTINUUM
Titles, abstracts, authors, keywords and DOIs across all five collections.
16 resultsSearch & filters ↓
CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents
Kofi Mensah, Marcus Tan, Ingrid Novak
Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.
Research ArchiveDataset
CTM-Bench: Task Set, Perturbations and Evaluation Harness
Marcus Tan, Kofi Mensah, Lena Hartwell
1,200 tool-use tasks, each with five perturbed variants, together with the evaluation harness, a datasheet and the annotation guidelines. Accompanies the article “CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents”.
View record 3,860 2,2409 cited
A Survey of Evaluation Practices for Language-Model Agents: What is measured, how it is measured, and what is missing
Kofi Mensah, Ingrid Novak, Lena Hartwell
As language-model agents move from demonstrations to deployments, the way they are evaluated determines which claims can be trusted. We review how agents are evaluated in the literature, organising practice along four axes: the task environment, the success criterion, the treatment of variance, and the reporting of cost. We identify recurring weaknesses: success is often judged by another model without validation, runs are rarely repeated, cost and latency are reported in under a third of studies, and environment versions are seldom pinned. We close with a checklist for evaluation reports and open problems, including measurement of safety-relevant side effects.
Scaling Behaviour of Parameter-Efficient Fine-Tuning Across Model Sizes
Priya Venkataraman, Daniel Okafor
Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.
Retrieval-Augmented Tool Selection for Long-Horizon Language Agents: Choosing from thousands of tools without placing them all in the prompt
Lena Hartwell, Tomás Ibarra, Marcus Tan
Language agents that act through external tools are limited by the size of their prompt: listing hundreds of tool descriptions crowds out the task and degrades accuracy. We study tool selection as a retrieval problem and propose a two-stage selector that (i) retrieves a short list of candidate tools using an embedding of the evolving task state and (ii) lets the model re-rank candidates with a single constrained call. Across a synthetic suite of 4,000 tools and three long-horizon task families, the selector keeps task success within 2.1 points of an oracle that sees only the correct tools, while reducing prompt length by 87%. Ablations show that conditioning retrieval on the task state, rather than on the initial instruction alone, accounts for most of the gain. We discuss failure cases, including tools with overlapping descriptions, and release the evaluation harness.
Research ArchiveSoftware
Tool-Selection Evaluation Harness
Source code for the two-stage tool selector and the evaluation harness used in “Retrieval-Augmented Tool Selection for Long-Horizon Language Agents”. Includes configuration files to reproduce every table.
View record 1,960 8843 cited
Research ArchiveDataset
Synthetic Tool Registry (4,000 tools)
A registry of 4,000 synthetic tools with descriptions, argument schemas and three task families, used to study tool selection at scale. Also includes a small registry of 600 real API descriptions.
View record 1,420 6122 cited