Reviewed by the event organisers
Permanent, citable homes for conference papers.
Open-access proceedings for conferences, workshops and symposia. Organisers deposit their accepted papers; every paper and every volume receives a DOI.
2 resultsSearch & filters ↓
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
Long conversations on a phone are limited by the size of the key-value cache. We compare eviction, quantisation and merging strategies under hard memory budgets and find that a hybrid of merging and 8-bit storage keeps perplexity within 3% at one quarter of the memory.
