Search AEX CONTINUUM

Titles, abstracts, authors, keywords and DOIs across all five collections.

PreprintsResearch ArticleNot peer reviewed

Tokenizer Effects on Low-Resource Language Adaptation

Version 1 · 12 September 2026 · 10.xxxx/ctm.pp.2026.0005

Adapting a pretrained model to a low-resource language often begins with extending the tokenizer, yet the choice of extension method is rarely compared. We evaluate five vocabulary-extension strategies on six languages and find that fertility reduction alone does not predict downstream accuracy. Strategies that initialise new embeddings from semantically related tokens consistently outperform random initialisation, at the same training budget.

Abstract PDF 1,890 6032 cited

PublishingResearch ArticleUnder peer review

Quantisation-Aware Distillation for On-Device Assistants

Daniel Okafor, Yuki Saito, Priya Venkataraman

Version 1 · 1 reviewer report · 28 August 2026 · 10.xxxx/ctm.pb.2026.0006

Running assistant-grade language models on phones requires both small models and low-precision arithmetic, yet distillation and quantisation are usually applied one after the other. We train a 1.3B-parameter student with distillation losses computed through simulated 4-bit quantisation, so that the student learns to be robust to its own rounding error. Compared with distilling first and quantising afterwards, the joint procedure recovers 41% of the accuracy lost to 4-bit quantisation, reaches 94% of the full-precision student on a suite of assistant tasks, and runs 3.1× faster on a mid-range mobile processor.

Abstract PDF 2,144 7862 cited

PublishingResearch ArticlePeer reviewed

Scaling Behaviour of Parameter-Efficient Fine-Tuning Across Model Sizes

Priya Venkataraman, Daniel Okafor

Version 2 · 3 reviewer reports · 6 March 2026 · 10.xxxx/ctm.pb.2026.0002

Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.

Abstract PDF 3,902 1,5445 cited