2 results for “LoRA” in keywordSearch & filters ↓
Priya Venkataraman, Daniel Okafor
An introductory textbook on adapting large pretrained models cheaply. Starting from the linear algebra of low-rank updates, it covers adapters, prompt tuning, quantisation-aware training and deployment, with exercises and solutions for each chapter.
Priya Venkataraman, Daniel Okafor
Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.
