Sample record. This work is fictional demonstration content. Results, people and institutions are placeholders; references in full-text articles are real publications.
Highlights
- A confidence estimator that mixes generator probabilities with retrieval agreement.
- Expected calibration error falls from 0.19 to 0.06.
- Abstaining on 20% of questions lifts accuracy from 71% to 84%.
Abstract
Retrieval-augmented generation grounds answers in retrieved documents, yet models remain over-confident when the retrieved evidence is irrelevant or contradictory. We propose a lightweight confidence estimator that combines the generator's token probabilities with a retrieval-agreement score, and calibrate it with temperature scaling on a small validation set. On three question-answering datasets the estimator lowers expected calibration error from 0.19 to 0.06 and enables selective answering: abstaining on the 20% least confident questions raises accuracy from 71% to 84%. The method needs no model retraining.
Keywords calibration · retrieval-augmented generation · uncertainty · selective prediction
1.Introduction
Retrieval-augmented generation (RAG) [1] is widely used to reduce unsupported statements, but retrieval can fail silently: a passage may be on-topic yet not contain the answer. Modern networks are known to be over-confident [2], and the problem is sharper when users assume that a cited answer is a reliable one. We study whether a model's confidence can be made to reflect the quality of its evidence.
2.Method
For a question q with retrieved passages D, let s_gen be the mean token log-probability of the generated answer and s_ret the fraction of passages whose own answer, generated independently, agrees with the final answer. The confidence is a calibrated mixture:
where the weights and the temperature T are fitted on 500 validation questions by minimising negative log-likelihood [2].
3.Results
Table 1 compares calibration. The mixture is better calibrated than either component alone. Figure 1 shows risk–coverage curves: answering only the more confident questions improves accuracy for every method, most strongly for the mixture.
| Confidence signal | ECE | Brier score |
|---|---|---|
| Token probability | 0.19 | 0.21 |
| Retrieval agreement | 0.11 | 0.17 |
| Mixture (ours) | 0.06 | 0.14 |
- Token probability
- Retrieval agreement
- Mixture (ours)
Limitations
- Agreement requires extra generation calls, increasing cost by a factor of about k for k passages.
- Evaluated on factoid questions only.
Declarations
- Author contributions
- A.R. conceived the method and wrote the manuscript; T.I. ran the retrieval experiments.
- Funding
- Sample article – no funding to declare.
- Competing interests
- The authors declare no competing interests.
- Data and code availability
- Sample article. All results, names and affiliations in this demonstration article are synthetic placeholders.
References
- [1]Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33. arXiv:2005.11401
- [2]Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1321–1330. arXiv:1706.04599
Versions
Version 2 · 5 June 2026Latest
Corrected Table 1; added a calibration plot per retriever.
10.xxxx/ctm.pp.2026.0001.v2
Version 1 · 4 May 2026
First version.
10.xxxx/ctm.pp.2026.0001.v1
How to cite
Roussel, A. & Ibarra, T. (2026). Calibrated Uncertainty in Retrieval-Augmented Generation [Preprint, version 2]. CONTINUUM Preprints. doi:10.xxxx/ctm.pp.2026.0001
© 2026 The authors. Published by AEX CONTINUUM under the CC BY 4.0 licence.