Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- Of 240 papers claiming code, 87% had a reachable repository but only 31% reproduced the headline result.
- Papers that reported seed variance were more likely to reproduce.
Abstract
Many machine-learning venues now require authors to complete a reproducibility checklist. We audited 240 papers that ticked the item “code is provided” and checked whether the linked repositories existed, ran and reproduced the headline table. 87% had a reachable repository, 58% contained instructions sufficient to run training or evaluation, and 31% reproduced the headline result within the reported variance. Compliance was higher for papers reporting variance across seeds. We recommend checks that can be automated at submission time.
Keywords reproducibility · checklists · research software · audit
1.Introduction
Reproducibility checklists [1] were introduced to improve reporting, alongside recommendations for reporting experimental results [2]. Whether ticking a box corresponds to a usable artefact is an empirical question. Errors in experimental pipelines, such as data leakage [3], can also invalidate results even when code is shared.
2.Audit protocol
We sampled 240 papers that declared code availability. For each we recorded whether the repository was reachable, whether it contained instructions, and whether one of two independent auditors could reproduce the main results table using a budget of 24 GPU-hours.
3.Results
Figure 1 shows the attrition from claim to reproduced result. Papers that reported variance across seeds were reproduced in 44% of cases against 22% for those that did not.
| Reporting practice | Papers | Reproduced |
|---|---|---|
| Variance across seeds reported | 96 | 44% |
| No variance reported | 144 | 22% |
4.Recommendations
- Verify at submission that repository links resolve.
- Require a one-command evaluation entry point.
- Request pinned dependency versions and random seeds.
- Offer artefact badges after independent reproduction.
Declarations
- Author contributions
- I.N. designed the audit; K.M. and I.N. performed reproductions.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Larochelle, H. (2021). Improving reproducibility in machine learning research. Journal of Machine Learning Research, 22(164), 1–20.
- [2]Dodge, J., Gururangan, S., Card, D., Schwartz, R., Smith, N. A. (2019). Show your work: Improved reporting of experimental results. Proceedings of EMNLP-IJCNLP, 2185–2194. arXiv:1909.03004
- [3]Kapoor, S., Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. doi:10.1016/j.patter.2023.100804
Versions
Version 1 · 27 July 2026Latest
First version.
10.xxxx/ctm.pp.2026.0002.v1
How to cite
Novak, I. & Mensah, K. (2026). Reproducibility-Checklist Compliance in Machine-Learning Papers [Preprint, version 1]. CONTINUUM Preprints. doi:10.xxxx/ctm.pp.2026.0002
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.