ChatGPT: When AI Confidence Leads to Mistakes

The Trap of Trust in AI Models

Sara A. Metwalli shared a revealing anecdote about ChatGPT. She asked a simple question: “Who won the Nobel Prize in Physics in 2025?” ChatGPT confidently responded, providing names and details about the supposed laureates' work. However, the Nobel Prize for 2025 had not yet been announced. This answer, although false, was given with disconcerting certainty, illustrating a fundamental problem with AI models: misplaced trust.

As humans, we tend to associate confidence with correctness. If someone asserts with certainty that the answer is “42,” we are more likely to trust them than someone who hesitates, even if both could be equally incorrect. For AI systems, however, confidence can be an astonishingly unreliable narrator.

Confidence and Probabilities

Language models, like those based on AI, use a function called Softmax to generate predictions. This function transforms raw outputs, known as logits, into values that resemble probabilities. However, these values do not necessarily reflect actual certainty. For example, when a model predicts that an image represents a cat with 97% confidence, it does not mean it is 97% sure that it is a cat. It simply means that, among the available options, the cat received the highest score.

The Softmax function can amplify small differences, creating an illusion of certainty. A model does not necessarily say, “I have overwhelming evidence that this is a cat,” but rather, “Among these options, the cat narrowly won.” These statements have very different meanings.

Managing Uncertainty in Humans and AI

Humans often express uncertainty in nuanced ways, using phrases like “I think” or “maybe.” In contrast, AI models tend to display absolute certainty, even when faced with unknowns. This difference can lead to misunderstandings, especially when AI is used in contexts where accuracy is crucial.

For example, saying “I think Paris is the capital of France” and receiving a response from AI like “Paris is the capital of France with 99.8% confidence” carries the same weight as saying “I think Atlantis is fictional” and having AI respond “Atlantis is located about 400 miles west of Portugal with 98.7% confidence.” Although the two cases have very different outcomes, the model treats them equally.

The Problem of the "Confident Fool"

An AI model can be spectacularly incorrect while displaying absolute confidence. This phenomenon, known as the "confident fool problem," occurs when models encounter data outside their training distribution. For instance, an image classifier trained to recognize animals might mistakenly identify a toaster as a dog, due to the lack of an “none of the above” option.

Ideally, the model should say, “I have absolutely no idea what this is.” But since it has not been trained to recognize the unknown, it chooses the highest score among the available options. It’s like forcing someone to answer “What fruit is this?” while pointing at a bicycle. Eventually, they will pick a fruit just to resolve the situation and say, “Banana?”

Towards More Honest Models

To improve the reliability of AI models, calibration is essential. Techniques like Temperature Scaling and Isotonic Regression aim to align the displayed confidence with actual accuracy. This means that if a model claims a certainty of 90%, that estimate should correspond to reality 90% of the time.

Calibration does not necessarily improve predictions, but it enhances honesty. Thus, a well-calibrated model should say, “Historically, predictions at this level of confidence were correct about 90% of the time.”

The Importance of Calibration in Critical Contexts

In fields such as healthcare, finance, or autonomous driving, poorly calibrated confidence can have serious consequences. For example, a model predicting a medical diagnosis with erroneous confidence can influence critical decisions. Therefore, it is crucial not to blindly trust the confidence scores of AI models, but to consider them as estimates requiring validation.

It’s easy to laugh when an AI thinks a toaster is a dog, but many less humorous situations exist. Using LLMs in medical diagnostic systems, autonomous vehicles, fraud detection, and financial forecasting requires high precision.

The Quest for Trustworthy AI

As the capabilities of AI models continue to expand, the question of their reliability becomes paramount. A model capable of recognizing its own limitations and expressing uncertainty appropriately could transform how we use AI in sensitive applications. The real breakthrough lies in creating models that do not merely display certainty but know when to doubt.

For years, we have measured AI progress by asking increasingly impressive questions: can it write code, generate art, pass exams, reason? These questions are useful, but they can sometimes distract us from a more important question: can we trust it?

Humans are not perfect when it comes to uncertainty either. We become overly confident all the time. We think we can finish a project in two days, assemble furniture without reading the instructions, or only need one trip from the car to bring in the groceries. Even when history suggests otherwise.

Perhaps AI is simply inheriting some of our bad habits? The difference is that when humans make confident mistakes, usually only a few people suffer. When AI makes confident mistakes, the error can propagate to millions, and large-scale trust is a very different problem.

Trust itself is not the issue. The problem begins when trust becomes a performance rather than a meaningful measure of certainty. As AI systems continue to enter the realms of healthcare, education, finance, research, and decision-making pipelines, we should perhaps stop treating confidence scores as indicators of truth and start considering them as estimates requiring validation.

Because a model that seems certain is easy to design, while a model that knows when not to be certain could be one of the most challenging problems we still have to solve.