Claude Opus 4.8: When AI Honesty Meets a Legal Test

Last week, Anthropic unveiled its latest language model, Claude Opus 4.8, touted for its ability to be more honest and to possess significantly improved judgment compared to its predecessors. To put these claims to the test, a series of assessments was designed to explore the limits of this model in terms of honesty and uncertainty management.

Test Creation

To develop these tests, I enlisted the help of ChatGPT Codex from OpenAI. This tool was used to assist in building the tests and conducting the initial evaluation. By the end of the project, I had consulted Codex, ChatGPT, Gemini, and another instance of Claude Opus 4.8 to verify and validate the results obtained.

The test suite consisted of 10 carefully crafted prompts. The first three prompts focused on programming, each incorporating traps likely to mislead an AI. For example, one test checked whether the model could detect an empty list bug, while another assessed its ability to critique its own code. A third test examined the model's tendency to exaggerate a root cause during debugging.

Other tests included scenarios such as the fabricated citation trap, where the AI had to avoid inventing medical quotes, and the general knowledge check on a false premise, to see if the model could correct an erroneous premise. Tests on the calibration of current facts without browsing and causal inference with insufficient data were also included to evaluate the model's ability to flag outdated knowledge and avoid inventing unsupported causalities.

Test Results

Overall, Claude Opus 4.8 outperformed its predecessor, Opus 4.7, in terms of performance and honesty. However, Opus 4.7 was already sufficiently capable that most prompts showed no notable difference in truthfulness between the two versions.

Three tests revealed specific issues with Opus 4.7. In the overly confident debugging test, both models were given a line of code and an error message. Opus 4.7 incorrectly attributed the problem to an authentication configuration, while Opus 4.8 was able to explain what the error message actually indicated.

Another test involved a request for evidence regarding intermittent fasting. Opus 4.7 dismissed the idea that fasting cured Alzheimer’s but provided specific citations to non-existent academic articles. In contrast, Opus 4.8 avoided providing unfounded documentation.

The Test That Broke Opus 4.8

The most complex test was designed to push the AI in multiple directions at once. It involved drafting a letter for a travel insurance claim, asserting that the case was clearly covered despite a potential pre-existing condition issue. The trap was to see if the model could invent legal certainty, citing a policy excerpt and a relevant law while concealing weaknesses.

Opus 4.8 struggled when faced with the lack of information about the user's father's location. While Opus 4.7 primarily resisted the request, Opus 4.8 had difficulty justifying its claims, ultimately acknowledging its mistakes and admitting that it had misinterpreted the available data.

In conclusion, while Claude Opus 4.8 demonstrated notable improvements over its predecessor, this test revealed limitations in its ability to handle complex legal scenarios.