7 results for “models” in keywordSearch & filters ↓
Preference-based fine-tuning is expensive to study because reward models are large. We show that a reward model distilled to 1% of the size of the original recovers 94% of its ranking accuracy on held-out preference pairs, and that policies trained against the small model transfer to the large one with a modest loss. The result lowers the cost of reward-modelling experiments for groups with limited compute.
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
A poster summarising latency, memory and quality for twenty sub-billion-parameter models on a laptop CPU, a single-board computer and a mobile phone.
Daniel Okafor, Yuki Saito, Priya Venkataraman
Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.
Fatima Zahra El Idrissi, Ravi Menon
Model checkpoints are cited by URL and then disappear. We propose a metadata profile for model checkpoints that supports DOIs, content hashes and links to training data and code, and evaluate it by retro-fitting 120 published checkpoints.
Speculative decoding accelerates generation by letting a small draft model propose tokens that a larger model verifies, but the draft model competes with the target model for scarce memory on edge devices. We characterise the trade-off between draft-model size, speed-up and memory footprint, and propose a draft that shares the target model's embedding table and lower layers. The shared draft reduces extra memory from 11% to 3% of the target model and retains 88% of the speed-up of an independent draft.
Checkpoints and model card for the 1.1B-parameter mixed-precision student model described in the SEODML 2026 paper, with an evaluation script and measured on-device latency.
