3 results for “calibration” in keywordSearch & filters ↓
Python library for measuring and improving calibration of generated text, including reliability diagrams, expected calibration error variants and post-hoc recalibration methods.
Doctoral thesis on estimating and communicating uncertainty for text generation. It develops calibration methods for retrieval-augmented systems, studies how users interpret expressed confidence, and proposes evaluation protocols for uncertainty in open-ended outputs.
Retrieval-augmented generation grounds answers in retrieved documents, yet models remain over-confident when the retrieved evidence is irrelevant or contradictory. We propose a lightweight confidence estimator that combines the generator's token probabilities with a retrieval-agreement score, and calibrate it with temperature scaling on a small validation set. On three question-answering datasets the estimator lowers expected calibration error from 0.19 to 0.06 and enables selective answering: abstaining on the 20% least confident questions raises accuracy from 71% to 84%. The method needs no model retraining.
