3 results for “language models” in keywordSearch & filters ↓
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
Daniel Okafor, Yuki Saito, Priya Venkataraman
Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.
Speculative decoding accelerates generation by letting a small draft model propose tokens that a larger model verifies, but the draft model competes with the target model for scarce memory on edge devices. We characterise the trade-off between draft-model size, speed-up and memory footprint, and propose a draft that shares the target model's embedding table and lower layers. The shared draft reduces extra memory from 11% to 3% of the target model and retains 88% of the speed-up of an independent draft.
