14 resultsSearch & filters ↓
Checkpoints and model card for the 1.1B-parameter mixed-precision student model described in the SEODML 2026 paper, with an evaluation script and measured on-device latency.
Daniel Okafor, Yuki Saito, Priya Venkataraman
Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.
Lecture notes from a graduate course on making transformer inference fast and small: attention variants, caching, speculative decoding, quantisation, batching and scheduling. Includes problem sets and a reading list.
Priya Venkataraman, Daniel Okafor
An introductory textbook on adapting large pretrained models cheaply. Starting from the linear algebra of low-rank updates, it covers adapters, prompt tuning, quantisation-aware training and deployment, with exercises and solutions for each chapter.
Priya Venkataraman, Daniel Okafor
Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.
Priya Venkataraman, Daniel Okafor
Configurations, training curves and evaluation metrics for 1,160 fine-tuning runs across six model sizes and four adaptation methods. Model weights are not included.
