Research ArticleNot peer reviewed
Tokenizer Effects on Low-Resource Language Adaptation
Adapting a pretrained model to a low-resource language often begins with extending the tokenizer, yet the choice of extension method is rarely compared. We evaluate five vocabulary-extension strategies on six languages and find that fertility reduction alone does not predict downstream accuracy. Strategies that initialise new embeddings from semantically related tokens consistently outperform random initialisation, at the same training budget.