2 results for “software” in keywordSearch & filters ↓
We attempt to build and run 400 research repositories released between 2019 and 2021. Only 38% still run without modification; the most common causes are unpinned dependencies and removed external data. Archived releases with lock files survive at more than twice the rate.
Many machine-learning venues now require authors to complete a reproducibility checklist. We audited 240 papers that ticked the item “code is provided” and checked whether the linked repositories existed, ran and reproduced the headline table. 87% had a reachable repository, 58% contained instructions sufficient to run training or evaluation, and 31% reproduced the headline result within the reported variance. Compliance was higher for papers reporting variance across seeds. We recommend checks that can be automated at submission time.
