Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Abstract
Adapting a pretrained model to a low-resource language often begins with extending the tokenizer, yet the choice of extension method is rarely compared. We evaluate five vocabulary-extension strategies on six languages and find that fertility reduction alone does not predict downstream accuracy. Strategies that initialise new embeddings from semantically related tokens consistently outperform random initialisation, at the same training budget.
Keywords tokenisation · low-resource languages · vocabulary extension · adaptation
Full text
CTM-PP-2026-0005-v1.pdf
Declarations
- Author contributions
- Author contributions are listed in the PDF.
- Funding
- Sample record – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
Versions
Version 1 · 12 September 2026Latest
First version.
10.xxxx/ctm.pp.2026.0005.v1
How to cite
Adebayo, O. & Tanaka, M. (2026). Tokenizer Effects on Low-Resource Language Adaptation [Preprint, version 1]. CONTINUUM Preprints. doi:10.xxxx/ctm.pp.2026.0005
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.