Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- Distillation loss is computed through simulated 4-bit quantisation.
- Recovers 41% of the accuracy lost to post-hoc quantisation.
- 3.1× faster than the full-precision student on a mobile processor.
Abstract
Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.
Keywords distillation · quantisation · on-device inference · language models
1.Introduction
Assistants that run on-device avoid network latency and keep data local, but phones offer a fraction of the memory and compute of a server. Two techniques dominate compression: knowledge distillation into a smaller student [1] and quantisation to low-bit integers [2, 3, 4]. Applying them sequentially ignores that the student could adapt to quantisation noise during training.
2.Method
Let Q(·) denote symmetric per-channel 4-bit quantisation with a straight-through gradient estimator. The student's logits are computed with quantised weights and activations, and trained against the teacher's softened distribution:
Outlier activation channels are handled with the scale migration of [3]. Optimisation uses Adam [5]; the quantiser step sizes are learned jointly.
3.Results
Table 1 compares the joint procedure with the sequential baseline. Figure 1 shows accuracy as the bit-width is reduced: the benefit of the joint training grows as the precision falls.
| Student | Bits | Accuracy | Tokens/s |
|---|---|---|---|
| Full precision | 16 | 69.0 | 10 |
| Distil, then quantise | 4 | 63.0 | 31 |
| Joint (ours) | 4 | 65.7 | 31 |
- Distil, then quantise
- Joint (ours)
Latency
On a mid-range mobile processor, the 4-bit student generates 31 tokens/s, against 10 tokens/s for the 16-bit student, a 3.1× speed-up.
4.Limitations
- Evaluated on one student architecture.
- Latency measured on a single device class.
- Long-context behaviour was not evaluated.
Declarations
- Author contributions
- D.O. designed the method and wrote the manuscript. Y.S. implemented the mobile runtime. P.V. advised on the training schedule.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Hinton, G., Vinyals, O., Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint. arXiv:1503.02531
- [2]Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2704–2713. arXiv:1712.05877
- [3]Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. (2023). SmoothQuant: Accurate and efficient post-training quantization for large language models. Proceedings of the 40th International Conference on Machine Learning, 202, 38087–38099. arXiv:2211.10438
- [4]Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. (2023). GPTQ: Accurate post-training quantization for generative pre-trained transformers. International Conference on Learning Representations. arXiv:2210.17323
- [5]Kingma, D. P., Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations. arXiv:1412.6980
Peer review
Open reports from independent reviewers. See how outcomes are determined.
- Status
- Under peer review
- Requested
- 2 Sep 2026
- Reviewers invited
- 3
- Report identity
- Signed reports
Reports on version 1Current version
- Approved with reservations
Dr. Henrik Solberg
A practical method with convincing on-device latency numbers. I would like to see results on one more device class and a fuller ablation of the distillation temperature.
Strengths
- Latency measured on real hardware.
- Straightforward to reproduce.
Concerns
- Only one phone chipset is evaluated.
- Temperature ablation covers only three values.
Versions
Version 1 · 28 August 2026Latest
Posted; peer review requested 2 September 2026.
10.xxxx/ctm.pb.2026.0006.v1
How to cite
Okafor, D., Saito, Y., & Venkataraman, P. (2026). Quantisation-Aware Distillation for On-Device Assistants. CONTINUUM Publishing, CTM-PB-2026-0006. doi:10.xxxx/ctm.pb.2026.0006
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.