Version 1 · 5 November 2026 · 10.xxxx/ctm.pp.2026.0009
Speculative decoding accelerates generation by letting a small draft model propose tokens that a larger model verifies, but the draft model competes with the target model for scarce memory on edge devices. We characterise the trade-off between draft-model size, speed-up and memory footprint, and propose a draft that shares the target model's embedding table and lower layers. The shared draft reduces extra memory from 11% to 3% of the target model and retains 88% of the speed-up of an independent draft.
Version 1 · 29 October 2026 · 10.xxxx/ctm.pp.2026.0008
Preference-based fine-tuning is expensive to study because reward models are large. We show that a reward model distilled to 1% of the size of the original recovers 94% of its ranking accuracy on held-out preference pairs, and that policies trained against the small model transfer to the large one with a modest loss. The result lowers the cost of reward-modelling experiments for groups with limited compute.
SEODML 2026 · pp. 1–10 · 1 October 2026 · 10.xxxx/ctm.pr.2026.0006
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
SEODML 2026 · pp. 11–20 · 1 October 2026 · 10.xxxx/ctm.pr.2026.0007
Long conversations on a phone are limited by the size of the key-value cache. We compare eviction, quantisation and merging strategies under hard memory budgets and find that a hybrid of merging and 8-bit storage keeps perplexity within 3% at one quarter of the memory.
SEODML 2026 · pp. 21–26 · 1 October 2026 · 10.xxxx/ctm.pr.2026.0008
We measure energy per generated token for eleven small language models on three mobile systems-on-chip. Memory traffic, not arithmetic, explains most of the variation, and the best configuration per device differs by up to 2.4×.
SEODML 2026 · pp. 27–32 · 1 October 2026 · 10.xxxx/ctm.pr.2026.0009
Personalising a model on a user's device requires small, fast-to-train updates. We study rank, placement and quantisation of low-rank adapters on a phone-class model and find that rank 4 on attention projections gives most of the gain at under 2 MB per user.
Version 1 · 12 September 2026 · 10.xxxx/ctm.pp.2026.0005
Adapting a pretrained model to a low-resource language often begins with extending the tokenizer, yet the choice of extension method is rarely compared. We evaluate five vocabulary-extension strategies on six languages and find that fertility reduction alone does not predict downstream accuracy. Strategies that initialise new embeddings from semantically related tokens consistently outperform random initialisation, at the same training budget.
Version 1.0 · CC BY 4.0 · 12 September 2026 · 10.xxxx/ctm.ar.2026.0008
Checkpoints and model card for the 1.1B-parameter mixed-precision student model described in the SEODML 2026 paper, with an evaluation script and measured on-device latency.
Version 1 · 1 reviewer report · 28 August 2026 · 10.xxxx/ctm.pb.2026.0006
Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.
Version 1.1 · 142 pages · ISBN 978-0-00-100004-3 · 12 August 2026 · 10.xxxx/ctm.bk.2026.0004
Lecture notes from a graduate course on making transformer inference fast and small: attention variants, caching, speculative decoding, quantisation, batching and scheduling. Includes problem sets and a reading list.
1st edition · 248 pages · ISBN 978-0-00-100002-9 · 18 June 2026 · 10.xxxx/ctm.bk.2026.0002
An introductory textbook on adapting large pretrained models cheaply. Starting from the linear algebra of low-rank updates, it covers adapters, prompt tuning, quantisation-aware training and deployment, with exercises and solutions for each chapter.
Version 2 · 3 reviewer reports · 6 March 2026 · 10.xxxx/ctm.pb.2026.0002
Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.
Version 1.0 · CC BY 4.0 · 6 March 2026 · 10.xxxx/ctm.ar.2026.0004
Configurations, training curves and evaluation metrics for 1,160 fine-tuning runs across six model sizes and four adaptation methods. Model weights are not included.
