Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

2 results for “tokenisation” in keywordSearch & filters ↓

PublishingResearch ArticleUnder peer review

Dense Retrieval for Code-Mixed Queries: A study of Hindi–English and Yoruba–English search

Oluwaseun Adebayo, Elias Wagner, Ravi Menon

Version 1 · 0 reviewer reports · 5 October 2026 · 10.xxxx/ctm.pb.2026.0009

Users frequently mix languages within a single query, but retrieval benchmarks rarely reflect this. We construct code-mixed query sets for two language pairs and evaluate dense retrievers under different tokenisation and fine-tuning strategies. Adapting the query encoder with synthetic code-mixed data improves recall@20 by 9 points over a multilingual baseline, while transliteration normalisation helps for romanised queries but hurts for native-script queries.

Abstract PDF 640 1880 cited

PreprintsResearch ArticleNot peer reviewed

Tokenizer Effects on Low-Resource Language Adaptation

Oluwaseun Adebayo, Mei Tanaka

Version 1 · 12 September 2026 · 10.xxxx/ctm.pp.2026.0005

Adapting a pretrained model to a low-resource language often begins with extending the tokenizer, yet the choice of extension method is rarely compared. We evaluate five vocabulary-extension strategies on six languages and find that fertility reduction alone does not predict downstream accuracy. Strategies that initialise new embeddings from semantically related tokens consistently outperform random initialisation, at the same training budget.

Abstract PDF 1,890 6032 cited