2 results for “datasheet” in keywordSearch & filters ↓
Fatima Zahra El Idrissi, Tomás Ibarra
Datasheets improve transparency but are time-consuming to write. We show that a template-driven workflow with automatic statistics halves the time to a complete datasheet and increases the completeness of answers on a fifteen-item rubric.
Kofi Mensah, Marcus Tan, Ingrid Novak
Existing agent benchmarks measure whether a task is solved once, not whether it is solved reliably. CTM-Bench contains 1,200 tool-use tasks, each with five perturbed variants (rephrased instructions, reordered tool lists, injected API errors, latency spikes and truncated outputs) and reports pass^k, the probability that an agent succeeds on all k independent attempts. We document collection, annotation and licensing following datasheet guidelines, evaluate six open models, and find that pass^1 overstates reliability: the best model falls from 71% pass^1 to 38% pass^5. The benchmark, scoring code and the datasheet are released under CC BY 4.0.
