Calibrating Language Models: Challenges and Innovative Solutions

Understanding Language Model Calibration

When a language model claims to have 90% confidence in a prediction, it should be correct 90% of the time. However, this ideal relationship is often compromised, leading to a problem of poor calibration. The confidence scores of models then cease to reflect their actual reliability. This issue is particularly prevalent in large language models (LLMs). A survey conducted at NAACL 2024 revealed that the confidence scores of LLMs often diverge from actual correctness rates in tasks such as factual question answering, code generation, and reasoning.

In the biomedical field, a study showed that the average calibration scores varied from 23.9% to 46.6% across all tested models, highlighting a constant and concerning gap. To address these issues, post-hoc recalibration is a standard solution in classical machine learning. It involves fitting a simple function on a held-out validation set to map raw confidence scores to better-calibrated probabilities.

Dominant Recalibration Methods

Three main methods dominate this process: temperature scaling, Platt scaling, and isotonic regression. These methods, although initially designed for discriminative classifiers, require specific adjustments to be applied to LLMs. Temperature scaling, for example, divides the logit vector by a scalar T before applying the softmax function. When T is greater than 1, the distribution flattens and confidence decreases, while when T is less than 1, the distribution sharpens and confidence increases.

Platt scaling fits a logistic function to uncalibrated scores, providing a parametric mapping with two free parameters. This process is particularly data-efficient, as it can produce usable estimates from a smaller calibration set. Isotonic regression, on the other hand, takes a non-parametric approach, learning a piecewise constant, monotonically non-decreasing mapping from uncalibrated scores to calibrated probabilities.

Measuring Calibration

To evaluate calibration, Expected Calibration Error (ECE) is the dominant metric. It bins predictions into confidence intervals, calculates the gap between the average confidence and the observed accuracy in each interval, and averages across the intervals weighted by their size. A perfectly calibrated model would have an ECE of 0. However, ECE alone is increasingly considered insufficient. A research paper recommends pairing ECE with the Brier score, overconfidence rates, and reliability diagrams, as a single number can obscure significant variation in how a model performs poorly.

A reliability diagram plots confidence against accuracy. A perfectly calibrated model lies on the diagonal. An overconfident model falls below: the curve shows high confidence, but accuracy does not follow. An evaluation in 2025 of the GPT-4o-mini model as a text classifier revealed that 66.7% of its errors occurred with over 80% confidence — the classic overconfidence pattern.

Challenges of LLMs

The three recalibration methods assume a fixed output space. A classifier produces a probability per class, and calibration maps them to better estimates. However, LLMs do not operate this way. Four major complications arise: the output space is exponentially large, semantically equivalent outputs can have very different probabilities, confidence varies across granularities, and many LLMs only expose the probabilities of the top k tokens via their API.

Application of Temperature Scaling

Temperature scaling divides the logit vector by a scalar T before applying the softmax function. When T > 1, the distribution flattens and confidence decreases. When T < 1, the distribution sharpens and confidence increases. T is adjusted on a held-out validation set by minimizing the negative log-likelihood. The method adds a parameter, preserves prediction rankings, and is computationally inexpensive.

The original formulation targeted image classifiers like DenseNet. For LLMs, temperature controls the probability distribution over the vocabulary at each decoding step, so the same logic applies. The problem lies in Reinforcement Learning from Human Feedback (RLHF). Post-RLHF models develop input-dependent overconfidence: the degree of poor calibration varies with inputs, and a single T cannot account for this variation.

Average ECE scores above 0.377 have been documented for models like GPT-3 in verbalized confidence tasks, and a 2025 survey confirms that RLHF-tuned models systematically overestimate confidence. Adaptive Temperature Scaling (ATS) directly addresses this. ATS predicts a temperature per token from hidden features at the token level, adjusted on a supervised fine-tuning dataset, instead of using a fixed T. Researchers have confirmed that ATS improves calibration by 10 to 50% without harming task performance. For any RLHF-tuned model, ATS is a more robust foundation than standard temperature scaling.

Standard temperature scaling still works well for base models before RLHF. When poor calibration is roughly uniform across inputs, a single T is often sufficient to correct systematic overconfidence or underconfidence. The problem is specific to post-RLHF models, where input-dependent overconfidence means a single T cannot correct all inputs.

Application of Platt Scaling

Platt scaling fits a logistic function to uncalibrated scores: p = σ(A·s + B), where A and B are learned from a held-out validation set with binary correction labels. The sigmoid form provides a parametric mapping with two free parameters. Platt scaling was initially developed for SVMs but generalizes to any system producing a scalar confidence score.

The two-parameter fit is also data-efficient compared to isotonic regression: it can produce usable estimates from a smaller calibration set, which is important in deployment contexts where labeled correction data is limited. In the context of LLMs, Platt scaling operates on confidence scores at the sequence or token level.

A paper on the confidence of LLM-generated code revealed that Platt scaling produced better-calibrated outputs than uncalibrated scores. Another study on LLMs for text-to-SQL introduced Multivariate Platt Scaling (MPS), extending Platt scaling from a single variable to combine sub-clause frequency scores across multiple generated samples — consistently outperforming single-score baselines.

Two limitations are documented. First, global Platt scaling at the sequence level is too coarse for tasks where correction depends on local editing decisions: a single sigmoid mapping cannot capture sample-dependent patterns of poor calibration. Additionally, Platt scaling may degrade appropriate scoring performance for powerful models.

Application of Isotonic Regression

Isotonic regression takes the non-parametric route. It learns a piecewise constant, monotonically non-decreasing mapping from uncalibrated scores to calibrated probabilities using the Pool Adjacent Violators Algorithm (PAVA). There is no assumed form for the calibration function, making it more flexible than Platt scaling when the confidence-accuracy relationship is not sigmoid-shaped.

The piecewise constant output adapts to any monotonic shape: linear, stepwise, or concave. This adaptability is the main reason why isotonic regression tends to outperform Platt scaling in empirical comparisons. The cost is the risk of overfitting on small calibration sets. The mapping only generalizes well when there is enough data to constrain it.

Empirically, isotonic regression outperforms Platt scaling. A rigorous comparison across multiple datasets and architectures revealed that isotonic regression beat Platt scaling on ECE and Brier score with statistical significance, using paired t-tests with Bonferroni correction at α = 0.003.

In this study, a Random Forest model went from an uncalibrated reliability score of 0.8268, to 0.9551 with Platt scaling, and then to 0.9660 with isotonic regression. Both methods could degrade appropriate scoring performance for powerful models, but the isotonic advantage held consistently.

For multiclass parameters of LLMs, it has been shown that standard isotonic regression can be further improved with normalization-aware extensions, consistently outperforming both OvR isotonic regression and standard parametric methods on NLL and ECE. The data requirement is the constraining factor. The isotonic advantage is real, but it does not transfer to deployment scenarios with limited data.

What the Literature Leaves Open

Three gaps deserve attention before deploying any of these methods. The interaction with RLHF has only been studied for temperature scaling. The performance of Platt scaling and isotonic regression on post-RLHF models has not been systematically tested. ATS exists because standard temperature scaling needed an explicit correction for this case. The need for similar extensions for the other two methods remains an open question.

Most direct comparisons of the three methods come from the general literature on calibration in machine learning. LLM-specific benchmarks that directly test all three are rare. The ICSE 2025 code calibration paper is one of the few, and its scope is limited to code generation.

The size of the calibration set is a real constraint in deployment. The results of isotonic regression in papers assume sufficiently large datasets to constrain the mapping. In production with limited labeled examples, the gap between isotonic regression and Platt scaling may narrow or reverse.

Temperature scaling is the right starting point for most teams. For base models without RLHF, a single T is often sufficient. For RLHF-tuned models, switch to ATS: the temperature per token manages input-dependent overconfidence that a global scalar cannot handle.

Platt scaling is the practical choice when the calibration set is small or when calibration needs to fit into a larger pipeline. It is data-efficient and easy to implement. The limitation is its scope: it cannot capture poor calibration that varies from sample to sample and tends to degrade performance for powerful models.

Isotonic regression has the best empirical track record of the three. Use it when the calibration set is large enough to constrain the mapping without overfitting, and pair it with normalization-aware extensions in multiclass contexts.

The decision that precedes all these methods is what "confidence" means for the task. Token probability, sequence probability, verbalized confidence, and consistency across samples can yield different values for the same output. A calibration method applied to the wrong signal does not improve reliability. Getting this definition correct is the prerequisite for all the above methods to work.