49 resultsSearch & filters ↓
Sofia Lindqvist, Julia Fernandes
Auditing an agent requires knowing what it saw, decided and did. We propose a small, open logging schema with eleven required fields, and show that it is sufficient to reconstruct the action sequence for 96% of runs in a sample of public agent traces. The schema is published under CC0.
We report experience building perturbation suites — rephrased instructions, reordered tool lists, noisy outputs — for three deployed agents, and summarise which perturbations revealed real failures and which only added cost.
Priya Venkataraman, Daniel Okafor
An introductory textbook on adapting large pretrained models cheaply. Starting from the linear algebra of low-rank updates, it covers adapters, prompt tuning, quantisation-aware training and deployment, with exercises and solutions for each chapter.
Slides of the opening keynote at the 1st Workshop on Reliable Tool-Using Agents, 12 June 2026.
Kofi Mensah, Marcus Tan, Ingrid Novak
Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.
Marcus Tan, Kofi Mensah, Lena Hartwell
1,200 tool-use tasks, each with five perturbed variants, together with the evaluation harness, a datasheet and the annotation guidelines. Accompanies the article “CTM-Bench: A Benchmark for Tool-Use Reliability in Language Agents”.
Retrieval-augmented generation grounds answers in retrieved documents, yet models remain over-confident when the retrieved evidence is irrelevant or contradictory. We propose a lightweight confidence estimator that combines the generator's token probabilities with a retrieval-agreement score, and calibrate it with temperature scaling on a small validation set. On three question-answering datasets the estimator lowers expected calibration error from 0.19 to 0.06 and enables selective answering: abstaining on the 20% least confident questions raises accuracy from 71% to 84%. The method needs no model retraining.
Python library for measuring and improving calibration of generated text, including reliability diagrams, expected calibration error variants and post-hoc recalibration methods.
