16 resultsSearch & filters ↓
We audit 2,400 checklist responses submitted over three years and compare them against the papers they accompany. Checks for compute reporting and variance were answered affirmatively in 78% of cases but supported by the paper in only 41%, suggesting that checklists work best when responses must point to a location in the text.
Fatima Zahra El Idrissi, Ravi Menon
Model checkpoints are cited by URL and then disappear. We propose a metadata profile for model checkpoints that supports DOIs, content hashes and links to training data and code, and evaluate it by retro-fitting 120 published checkpoints.
Fatima Zahra El Idrissi, Tomás Ibarra
Datasheets improve transparency but are time-consuming to write. We show that a template-driven workflow with automatic statistics halves the time to a complete datasheet and increases the completeness of answers on a fifteen-item rubric.
We study reports from 300 openly reviewed machine-learning papers. Open reports are longer, more specific and more often cite evidence than comparable anonymous reports from closed venues, and reviewers who sign their reports are no more lenient. We discuss limits of the comparison.
We attempt to build and run 400 research repositories released between 2019 and 2021. Only 38% still run without modification; the most common causes are unpinned dependencies and removed external data. Archived releases with lock files survive at more than twice the rate.
We argue that venues should offer an explicit track for well-designed studies that do not find the expected effect, and outline reviewing criteria that focus on design and power rather than outcome.
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
Long conversations on a phone are limited by the size of the key-value cache. We compare eviction, quantisation and merging strategies under hard memory budgets and find that a hybrid of merging and 8-bit storage keeps perplexity within 3% at one quarter of the memory.
