5 resultsSearch & filters ↓
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
Long conversations on a phone are limited by the size of the key-value cache. We compare eviction, quantisation and merging strategies under hard memory budgets and find that a hybrid of merging and 8-bit storage keeps perplexity within 3% at one quarter of the memory.
We measure energy per generated token for eleven small language models on three mobile systems-on-chip. Memory traffic, not arithmetic, explains most of the variation, and the best configuration per device differs by up to 2.4×.
Priya Venkataraman, Anders Holm
Personalising a model on a user's device requires small, fast-to-train updates. We study rank, placement and quantisation of low-rank adapters on a phone-class model and find that rank 4 on attention projections gives most of the gain at under 2 MB per user.
A poster summarising latency, memory and quality for twenty sub-billion-parameter models on a laptop CPU, a single-board computer and a mobile phone.
