Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Abstract
Preference-based fine-tuning is expensive to study because reward models are large. We show that a reward model distilled to 1% of the size of the original recovers 94% of its ranking accuracy on held-out preference pairs, and that policies trained against the small model transfer to the large one with a modest loss. The result lowers the cost of reward-modelling experiments for groups with limited compute.
Keywords reward models · RLHF · distillation · compute-efficient research
Full text
CTM-PP-2026-0008-v1.pdf
Declarations
- Author contributions
- Author contributions are listed in the PDF.
- Funding
- Sample record – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
Versions
Version 1 · 29 October 2026Latest
First version.
10.xxxx/ctm.pp.2026.0008.v1
How to cite
Berg, J. & Holm, A. (2026). Cheap Reward Models for Small-Scale RLHF Experiments [Preprint, version 1]. CONTINUUM Preprints. doi:10.xxxx/ctm.pp.2026.0008
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.