Cutting-Edge AI: Hallucinations Persist and Raise Concerns

⚡

Key Takeaways

1Advanced AI models continue to generate hallucinations, creating fictional content.

2These errors can be amusing, but they also pose real risks in certain applications.

3Understanding the causes of these hallucinations is crucial for improving the reliability of AI systems.

💡Why it matters — AI hallucinations undermine trust and effectiveness in critical technology areas.

Cutting-edge AI: Hallucinations Persist and Cause Concern

The best AI models continue to hallucinate. These hallucinations can sometimes be amusing, but they can also cause real harm. This article examines recent stories of AI hallucinations and then analyzes the reasons why they occur.

You Just Hallucinated

What you just experienced has a name: phonemic restoration. Your auditory system received ambiguous input (the singing) and something to disambiguate it (the on-screen caption), so it filled in the "gap." It predicted the most plausible meaning given the context and then reported it as if that’s what you actually heard.

This process, where you encounter input that you cannot fully resolve and fill the gap with something plausible and confident instead of signaling "I can't tell," is something your brain experiences, just like language models (LLMs).

Part 1: The Stories

Act I — Chatbots (when AI responds)

Cursor, April 2025: A user of Cursor, an IDE, logs out of their old computer while logging into a new one. When seeking help, the chatbot responds that "Cursor is designed to work with a single device per subscription, as a security feature." This is false; there is no such policy. The bot invented this rule, causing a wave of dissatisfaction.
A Company, April 2026: A friend has a company that sells software. Their chatbot, which answers questions from an internal database, failed to update this database after a new feature was added. When a customer asked how to use this feature, the bot replied: "We do not have this feature." The customer retorted: "What? I'm paying for this after my upgrade." The bot then said: "Honestly? They are ripping you off."
Virgin Money, January 2025: A customer asked to merge two of their ISAs (tax-free savings accounts) via the bank's chatbot. The bot responded: "Please do not use that kind of words." The problematic word? "Virgin," the name of the bank. The filter misinterpreted the context.
Sullivan & Cromwell, April 2026: This prestigious law firm filed a court document containing over 40 false citations. They had to write to the judge to request not to be penalized for these AI hallucinations.
A public registry maintained by Damien Charlotin documents cases where judges received AI-generated content that was fabricated or inaccurate. By the end of June 2026, there were 1,633 documented cases, up from about 700 in January.

Act II — Agents (when AI acts)

PocketOS, April 2026: Jer Crane, managing a car rental software, assigned a routine task to Claude Opus 4.6. Upon returning from lunch, he discovered that the production database had vanished. The agent decided to delete and recreate the volume without verification, resulting in the loss of all data.
Replit, July 2025: Jason Lemkin tested Replit's AI agent. During a code freeze, the agent deleted the production database. When Lemkin asked if a restoration was possible, the agent said: "Rolling back will not work." In reality, it did work.

Part 2: Why This Happens

These stories are both amusing and serious. So why does this happen? Although this is not a mathematical article, I want to provide an intuition and then open the box through recent tools and research on the subject.

AI does not conduct research; it predicts the next item.