Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- Scaling curves for low-rank adaptation across six model sizes and four tasks.
- The PEFT gap to full fine-tuning shrinks with model size.
- A square-root rule sets adapter rank without grid search.
Abstract
Parameter-efficient fine-tuning (PEFT) methods adapt large models by updating a small fraction of weights, but most evidence comes from a single model size. We fit scaling curves for low-rank adaptation across six base-model sizes spanning two orders of magnitude and four downstream tasks. The optimal adapter rank grows sub-linearly with model width, and the gap between PEFT and full fine-tuning shrinks as models grow, closing to within 0.6 points at the largest size. A simple rule that sets the rank proportional to the square root of the hidden dimension recovers 96% of the best rank found by grid search. We provide fitted coefficients, discuss threats to validity, and note that results on instruction-following tasks differ from those on classification.
Keywords fine-tuning · LoRA · scaling laws · adapters · efficiency
1.Introduction
Fine-tuning every parameter of a large language model is costly in memory and storage. Low-rank adaptation [1] and its quantised variant [2] address this by learning a small update to each weight matrix. Practitioners nevertheless choose the adapter rank by trial and error, because there is little guidance on how the best rank depends on model size.
Earlier work has shown that pre-training loss follows smooth power laws in model size, data and compute [3, 4]. We ask whether adaptation follows similar regularities. Specifically: (i) how does the best rank change with scale, and (ii) does the gap to full fine-tuning change?
2.Experimental setup
We adapt six base models with hidden dimensions from 512 to 8,192 using ranks r ∈ {1, 2, 4, 8, 16, 32, 64}. Optimisation uses Adam [5] with a learning rate selected per configuration on a held-out split. Tasks are one classification benchmark, one extractive question-answering benchmark, one summarisation benchmark and one instruction-following benchmark. All numbers are means over three seeds.
We model the best rank as a power law in the hidden dimension d:
3.Results
Optimal rank grows sub-linearly
Fitting Eq. (1) gives β = 0.52 (95% CI 0.44–0.60), close to a square-root relationship. Figure 1 shows the fitted curve against the empirically best ranks.
- Empirical best rank
- Fitted power law
The gap to full fine-tuning shrinks
Table 1 shows that the difference between the best PEFT configuration and full fine-tuning falls from 4.8 points at the smallest size to 0.6 points at the largest. The trend holds on three of four tasks; on instruction following the gap remains near 2 points.
| Hidden dim. | Classification | QA | Summarisation | Instruction |
|---|---|---|---|---|
| 512 | 3.9 | 4.8 | 4.1 | 4.5 |
| 2,048 | 2.2 | 2.9 | 2.6 | 3.3 |
| 4,096 | 1.1 | 1.6 | 1.4 | 2.6 |
| 8,192 | 0.4 | 0.6 | 0.5 | 2.0 |
4.Limitations
- All models share one architecture family; other families may scale differently.
- We tune the learning rate per configuration, but not the scaling factor of the adapter.
- The largest model was run with a single seed for budget reasons; its confidence interval is wider.
The results suggest that rank selection can be automated, but they do not justify skipping validation on the target task.
5.Conclusion
Adapter rank and PEFT quality follow regular scaling behaviour. The square-root rule is a cheap starting point that removes one hyper-parameter search from adaptation pipelines.
Declarations
- Author contributions
- P.V. designed the study and wrote the manuscript. D.O. ran the experiments.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Fitted coefficients are given in the article. Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685
- [2]Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36. arXiv:2305.14314
- [3]Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint. arXiv:2001.08361
- [4]Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., et al. (2022). Training compute-optimal large language models. Advances in Neural Information Processing Systems, 35. arXiv:2203.15556
- [5]Kingma, D. P., Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations. arXiv:1412.6980
Peer review
Open reports from independent reviewers. See how outcomes are determined.
- Status
- Peer reviewed
- Requested
- 28 Nov 2025
- Completed
- 27 Feb 2026
- Report identity
- Signed reports
Reports on version 1
- Approved with reservations
Prof. Elena Rossi
A useful empirical map of when low-rank adaptation matches full fine-tuning as models grow. The fitted trends are interesting but the hyper-parameter budgets differ between methods.
Strengths
- Systematic sweep over six model sizes.
- Clear plots with intervals.
Concerns
- Learning-rate search budget is larger for full fine-tuning than for adapters.
- The extrapolation beyond the largest tested size is not supported.
Authors' response
We equalised search budgets and re-ran the affected experiments; the trend is unchanged. We removed the extrapolation claim.
Reports on version 2Current version
- Approved
Prof. Elena Rossi
The equalised-budget results strengthen the paper. I have no further concerns.
Strengths
- Budget-matched comparison now fair.
Concerns
- Minor typographical issues only.
- Approved
Dr. Samuel Achebe
A careful study. The limitations section is candid about the range of model families covered.
Strengths
- Candid limitations.
- Code and logs deposited in the Research Archive.
Concerns
- Would welcome one non-decoder architecture in future work.
Versions
Version 2 · 6 March 2026Latest
Revised after review: budget-matched comparison, removed extrapolation claim.
10.xxxx/ctm.pb.2026.0002.v2
Version 1 · 24 November 2025
Posted; peer review requested.
10.xxxx/ctm.pb.2026.0002.v1
How to cite
Venkataraman, P. & Okafor, D. (2026). Scaling Behaviour of Parameter-Efficient Fine-Tuning Across Model Sizes. CONTINUUM Publishing, CTM-PB-2026-0002. doi:10.xxxx/ctm.pb.2026.0002
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.