Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- 1,200 tasks with five perturbation types.
- pass^k metric measures repeated-attempt reliability.
- Best model drops from 71% (pass^1) to 38% (pass^5).
Abstract
Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.
Keywords benchmark · reliability · tool use · agents · datasheet
1.Introduction
A tool-using agent that solves a task on one run may fail on the next. Reliability matters for deployment, but most benchmarks report a single-attempt success rate [1, 2]. We introduce CTM-Bench, which scores consistency under realistic perturbations.
2.Benchmark construction
Tasks
Tasks were written by 14 annotators and cover file handling, calendar management, data lookup, form filling and multi-service workflows. Each task has an executable checker. Following datasheet practice [3], we document motivation, composition, collection, uses and maintenance in an accompanying datasheet.
Perturbations
- Instruction paraphrase.
- Tool-list reordering.
- Injected transient API errors.
- Latency spikes.
- Truncated tool outputs.
Metric
For an agent with per-task success probability pᵢ, pass^k averages pᵢᵏ over tasks, which estimates the chance of succeeding on all k attempts:
3.Results
Table 1 shows that rankings by pass^1 and by pass^5 differ: Model C overtakes Model B once consistency is required. Figure 1 shows how performance decays with k.
| Model | pass^1 | pass^3 | pass^5 |
|---|---|---|---|
| Model A | 52 | 27 | 17 |
| Model B | 66 | 41 | 29 |
| Model C | 71 | 51 | 38 |
- Model A
- Model B
- Model C
4.Availability and intended use
The benchmark is intended for research evaluation, not for training. Tasks are synthetic and contain no personal data. A held-out set of 200 tasks is kept private and scored by an evaluation server to limit contamination, a concern raised for benchmarks generally [4].
Declarations
- Author contributions
- K.M. designed the benchmark. M.T. built the perturbation harness. I.N. wrote the datasheet.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Benchmark, scoring code and datasheet would be deposited with a persistent identifier. Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Liu, X., Yu, H., Zhang, H., Xu, Y., et al. (2024). AgentBench: Evaluating LLMs as agents. International Conference on Learning Representations. arXiv:2308.03688
- [2]Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations. arXiv:2210.03629
- [3]Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. doi:10.1145/3458723
- [4]Raji, I. D., Bender, E. M., Paullada, A., Denton, E., Hanna, A. (2021). AI and the everything in the whole wide world benchmark. NeurIPS Datasets and Benchmarks Track. arXiv:2111.15366
Peer review
Open reports from independent reviewers. See how outcomes are determined.
- Status
- Peer reviewed
- Requested
- 22 Jan 2026
- Completed
- 18 May 2026
- Report identity
- Anonymous reports
Reports on version 1
- Approved with reservations
Anonymous reviewer
The benchmark addresses a real gap: reliability rather than single-attempt success. Documentation of annotation and licensing is thin for a dataset paper.
Strengths
- Perturbed variants give a direct measure of reliability.
- Public hosting with a persistent identifier.
Concerns
- Annotation protocol and agreement statistics are missing.
- The licence of two source tool descriptions is unclear.
Authors' response
We added a datasheet, inter-annotator agreement (κ = 0.81) and replaced the two source descriptions.
Reports on version 2Current version
- Approved
Anonymous reviewer
The datasheet and agreement statistics settle my concerns. The benchmark is well documented and ready for use.
Strengths
- Complete datasheet.
- Clear intended-use statement.
Concerns
- None.
- Approved
Anonymous reviewer
Sound design and honest limitations. A good addition to the evaluation toolbox.
Strengths
- Reliability metric is simple and interpretable.
Concerns
- Future versions should add non-English tasks.
Versions
Version 2 · 12 June 2026Latest
Revised after review: datasheet, annotation agreement, replaced two source descriptions.
10.xxxx/ctm.pb.2026.0004.v2
Version 1 · 20 January 2026
Posted; peer review requested.
10.xxxx/ctm.pb.2026.0004.v1
How to cite
Mensah, K., Tan, M., & Novak, I. (2026). CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents. CONTINUUM Publishing, CTM-PB-2026-0004. doi:10.xxxx/ctm.pb.2026.0004
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.