Skip to main content
Submit

Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

7 results for “tool use”Search & filters ↓

PublishingDataset & BenchmarkPeer reviewed

CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents

Kofi Mensah, Marcus Tan, Ingrid Novak

Version 2 · 3 reviewer reports · 12 June 2026 · 10.xxxx/ctm.pb.2026.0004

Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.

Abstract PDF 5,330 2,4129 cited
ProceedingsFull paperBest paper

Retry Budgets for Tool-Using Agents: When Is Trying Again Worth It?

Aisha Rahman, Lena Hartwell

RTUA 2026 · pp. 1–9 · 3 July 2026 · 10.xxxx/ctm.pr.2026.0001

Agents that retry failed tool calls recover from transient errors but can also compound mistakes and waste resources. We analyse 18,000 recorded agent runs and show that retries help for network and rate-limit errors but hurt for malformed arguments. We propose a retry budget that depends on the error class and show that it improves success by 6 points at equal cost.

Abstract PDF 1,420 5602 cited
PublishingResearch ArticlePeer reviewed

Retrieval-Augmented Tool Selection for Long-Horizon Language Agents: Choosing from thousands of tools without placing them all in the prompt

Lena Hartwell, Tomás Ibarra, Marcus Tan

Version 2 · 4 reviewer reports · 27 February 2026 · 10.xxxx/ctm.pb.2026.0001

Language agents that act through external tools are limited by the size of their prompt: listing hundreds of tool descriptions crowds out the task and degrades accuracy. We study tool selection as a retrieval problem and propose a two-stage selector that (i) retrieves a short list of candidate tools using an embedding of the evolving task state and (ii) lets the model re-rank candidates with a single constrained call. Across a synthetic suite of 4,000 tools and three long-horizon task families, the selector keeps task success within 2.1 points of an oracle that sees only the correct tools, while reducing prompt length by 87%. Ablations show that conditioning retrieval on the task state, rather than on the initial instruction alone, accounts for most of the gain. We discuss failure cases, including tools with overlapping descriptions, and release the evaluation harness.

Abstract PDF 4,821 1,9367 cited
ProceedingsFull paper

Sandboxing Agent Tool Calls with Capability Tokens

Marcus Tan, Jonas Berg

RTUA 2026 · pp. 10–18 · 3 July 2026 · 10.xxxx/ctm.pr.2026.0002

Tool-using agents are typically granted broad credentials, so a single injected instruction can cause large harm. We describe a capability-token scheme in which every tool call carries a narrowly scoped, short-lived token issued by a policy layer outside the model. A prototype blocks all 41 attack scripts in our test suite with under 4 ms of added latency.

Abstract PDF 1,190 4301 cited