49 resultsSearch & filters ↓
We audit 2,400 checklist responses submitted over three years and compare them against the papers they accompany. Checks for compute reporting and variance were answered affirmatively in 78% of cases but supported by the paper in only 41%, suggesting that checklists work best when responses must point to a location in the text.
Fatima Zahra El Idrissi, Ravi Menon
Model checkpoints are cited by URL and then disappear. We propose a metadata profile for model checkpoints that supports DOIs, content hashes and links to training data and code, and evaluate it by retro-fitting 120 published checkpoints.
Fatima Zahra El Idrissi, Tomás Ibarra
Datasheets improve transparency but are time-consuming to write. We show that a template-driven workflow with automatic statistics halves the time to a complete datasheet and increases the completeness of answers on a fifteen-item rubric.
We study reports from 300 openly reviewed machine-learning papers. Open reports are longer, more specific and more often cite evidence than comparable anonymous reports from closed venues, and reviewers who sign their reports are no more lenient. We discuss limits of the comparison.
We attempt to build and run 400 research repositories released between 2019 and 2021. Only 38% still run without modification; the most common causes are unpinned dependencies and removed external data. Archived releases with lock files survive at more than twice the rate.
We argue that venues should offer an explicit track for well-designed studies that do not find the expected effect, and outline reviewing criteria that focus on design and power rather than outcome.
Speculative decoding accelerates generation by letting a small draft model propose tokens that a larger model verifies, but the draft model competes with the target model for scarce memory on edge devices. We characterise the trade-off between draft-model size, speed-up and memory footprint, and propose a draft that shares the target model's embedding table and lower layers. The shared draft reduces extra memory from 11% to 3% of the target model and retains 88% of the speed-up of an independent draft.
Preference-based fine-tuning is expensive to study because reward models are large. We show that a reward model distilled to 1% of the size of the original recovers 94% of its ranking accuracy on held-out preference pairs, and that policies trained against the small model transfer to the large one with a modest loss. The result lowers the cost of reward-modelling experiments for groups with limited compute.
