Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- A draft that shares embeddings and lower layers with the target model.
- Extra memory falls from 11% to 3%; 88% of the speed-up is retained.
Abstract
Speculative decoding accelerates generation by letting a small draft model propose tokens that a larger model verifies, but the draft model competes with the target model for scarce memory on edge devices. We characterise the trade-off between draft-model size, speed-up and memory footprint, and propose a draft that shares the target model's embedding table and lower layers. The shared draft reduces extra memory from 11% to 3% of the target model and retains 88% of the speed-up of an independent draft.
Keywords speculative decoding · edge inference · memory · language models
1.Introduction
Autoregressive decoding is bound by memory bandwidth. Speculative decoding [1] uses a draft model to propose several tokens at once, so that the target model can verify them in one pass. On servers the draft model's memory is negligible; on a phone it is not.
2.Results
Figure 1 shows the speed-up against additional memory for independent drafts of increasing size and for the shared-layer draft. The shared draft lies on a better frontier: at 3% additional memory it achieves a 1.9× speed-up, against 1.4× for an independent draft of equal memory.
- Independent draft
- Shared-layer draft (ours)
Declarations
- Author contributions
- R.D. designed the method; Y.S. carried out the device measurements.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Leviathan, Y., Kalman, M., Matias, Y. (2023). Fast inference from transformers via speculative decoding. Proceedings of the 40th International Conference on Machine Learning, 202, 19274–19286. arXiv:2211.17192
- [2]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. arXiv:1706.03762
Versions
Version 1 · 5 November 2026Latest
First version.
10.xxxx/ctm.pp.2026.0009.v1
How to cite
Desai, R. & Saito, Y. (2026). Speculative Decoding Under Memory Constraints [Preprint, version 1]. CONTINUUM Preprints. doi:10.xxxx/ctm.pp.2026.0009
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.