XConf Enhances LLM Reliability Through Analysis of Past Results

A team from Cambridge and Google DeepMind has proposed XConf, a confidence estimation that relies on past performance rather than the present moment. The authors report net gains in calibration and decision-making assistance for automation, without access to logits or training, with a strong requirement: reliable external labels.

Quantifiable Gains in Calibration and Automation Decisions

In the face of self-consistency across ten samples, XConf accumulates 21 wins, 2 ties, and 1 loss at one-tenth the generation cost. The authors report lower calibration error in all cases, three to eight times lower on MMLU-Pro. The gains are particularly pronounced for agents, where failed trajectories may appear clean: on AppWorld, the verbalized confidence averages 0.72 AUROC, while the retrieved experience reaches 0.86. On SWE-bench Verified, XConf surpasses a trained verifier. To drive automation, abstaining from the 10% of cases estimated to be the least confident increases the delivered accuracy by an average of 4.8 points, up to 8.7 points on AppWorld. The most confident decile is correct 98% of the time, compared to 90% for verbalized confidence and 87% for self-consistency, while the fifth least confident concentrates half of the errors. Under controlled question difficulty, the AUROC remains at 0.79. With Claude on AppWorld, the AUROC improves from 0.63 to 0.81 as the bank grows, and for reasoning tasks, a few hundred episodes are sufficient to match self-consistency where verbalized confidence is low. In 23 out of 24 comparisons, a single generation with XConf matches or exceeds ten samples of self-consistency, with much better calibration.

A Determining Constraint: External Labels and an Episode Bank

The system relies on truth labels provided outside the model. The authors note that with self-labels produced by the LLM, the method degrades. When the bank is sparse, the empirical component relaxes towards the base rate corresponding to the declared confidence level, avoiding sharp breaks. Conversely, as the bank grows, calibration improves, as illustrated by the AUROC progression from 0.63 to 0.81 on AppWorld with Claude. For reasoning tasks, a few hundred episodes are already sufficient to reach self-consistency where verbalized confidence remains low.

How XConf Produces Its Success Probability

Each successful or unsuccessful interaction is archived as an episode containing the task, a prior reflection from the model, the declared confidence, the correct or incorrect verdict, and a lesson learned afterward. The lesson is quarantined to avoid knowledge bias of the outcome during subsequent evaluations. For a new task, XConf forms a query combining the task embedding and the declared confidence level, then retrieves the 50 nearest neighbors. The fraction of successful neighbors constitutes a preliminary reading. Removing confidence from this key drops the AUROC by 0.08 on reasoning and 0.12 on agents. Similarity is learned in a resized space from rated results to bring closer episodes that fail for similar reasons. The neighbors are presented to the model in the form of cards containing task, confidence, result, and lesson; the model identifies a failure pattern and reformulates its confidence. The final probability is the simple average between this reflection and the empirical reading. The process is agnostic to the output format, whether it be multiple-choice questions, extended code, or agent trajectories.

XConf Leverages History to Refine Email Routing

In an insurance email routing system, a request containing "stop payment" is initially classified as Payments with high confidence by the LLM. XConf consults the history of similar cases with the same confidence level and finds that only 4 out of 10 actually remained in Payments, with the others falling under Cancellations; the recall is 0.40, and reflection further reduces confidence below the automation threshold. The email then goes for human review. Conversely, a bank transfer question, also evaluated as Payments with high confidence, comes with a history showing 48 correct out of 50 and can be processed automatically. No rule addition is required: the system relies on the actual observed outcome of the precedents.

Scope of Tests and Covered Models

The analysis concerns nine benchmarks that encompass multiple-choice, complex reasoning, math olympiads, code generation, multimodal questions, as well as three types of agent environments. Four models from three different families are evaluated: Gemini 2.5 Flash, Gemini 3.5 Flash, Claude Sonnet 4.6, and Qwen3.5–397B. The protocol employs a five-fold rotation to ensure that no task appears both in the bank and in the estimation, and a chronological growth scenario of the bank to reflect production usage. The cited results include notably AppWorld and SWE-bench Verified.

Positioning Against Traditional Calibration Approaches

Traditional approaches, whether verbalized confidence, token probabilities, or self-consistency, operate in the moment and can be misled by systematic model errors. Classical calibrations, such as Platt scaling, histogram binning, or isotonic regression, also estimate historical frequencies, but at the score level. XConf distinguishes itself by joint conditioning on task similarity and declared confidence level, and by an implementation that requires neither access to logits nor training. The method applies uniformly to heterogeneous productions, from multiple-choice questions to agents.