Claude Fable 5 and ChatGPT: AI in the Face of UX Evaluation

Key Takeaways

1AI can assist in UX evaluations, but it does not replace human expertise, especially for complex tasks.

2Claude Fable 5 demonstrated better accuracy than ChatGPT-5.6 Sol, but both produced analyses with errors.

3A 2025 study revealed that GPT-4o identifies 21% of usability issues, highlighting the importance of human expertise.

Why it matters : AI is a powerful tool for accelerating UX work, but human expertise remains essential for accurate and contextual results.

AI and UX Evaluation: A Promising but Imperfect Duo

Artificial intelligence (AI) is increasingly establishing itself as a powerful tool in the field of user experience (UX) evaluation. However, it cannot yet fully replace human expertise. The accuracy of the results depends largely on the quality of the information provided to the AI, rather than on the intrinsic capabilities of the model. In a recent project, a manager was tasked with conducting a UX evaluation for the search, results, and product detail pages of a portal. To do so, the manager had at their disposal a set of questions about the target audience, objectives, and USP, as well as a market research document and access to Google Analytics and Clarity.

The widely held starting hypothesis is that AI is already capable enough to handle this type of task. The idea is that the designer only needs to check the result, and on topics like this one, AI could almost replace the human. Faced with this opportunity, the manager decided to test the hypothesis to see whether it really held up.

A Rigorous Test with Two AI Models

To run this experiment, the manager used two AI models in parallel: Claude Fable 5 and ChatGPT-5.6 Sol, both accessible through paid subscriptions. Each was given the full set of available data: the brief, the questions and answers, the research material, the Google Analytics exports, the Clarity heatmaps, the attention and scroll maps, and the necessary URLs. The task was clearly defined: produce an expert UX evaluation document with a specific objective.

It is crucial to note that the manager did not use a single prompt for this task. The work was structured in several stages, with the models asked to justify their conclusions using Google Analytics and Clarity data whenever possible. The goal was not to test an "all-in-one" approach, but to see what a structured workflow could produce.

In parallel, the manager also carried out the analysis manually to have a basis for comparison.

Results of the Experiment: Analyses That Could Be Improved

It is important to stress that this report is based on a personal experience and not on a controlled study. The results produced by both AI models were full of irrelevant or even non-existent problems. Some useful observations did emerge, although they were only usable at the level of wording and structure. Claude Fable 5 proved more precise than ChatGPT-5.6 Sol, at least in terms of wording and structure. This remains a subjective impression and not an objective measurement.

One notable difference between the two models is that Claude could deliver the result in a single document, while ChatGPT managed it only in part. This seems to have more to do with the output format than with the quality of the analysis.

It is true that the AI brought to light some problems the manager had not noticed, which proved useful. However, the AI also failed to detect several other problems.

These observations are in line with a 2025 study that compared GPT-4o to Nielsen's usability heuristics. The study found that GPT-4o identified about 21% of the usability problems spotted by experts, while also raising several new problems, some of them erroneous. AI is therefore an excellent tool for a first pass, but it does not yet replace expert evaluation.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Final Assembly and the Role of AI

In the end, the manager chose to pull the analysis together with the help of Claude. The manager fed their own observations into a continuous conversation with the model, which allowed them to decide together what to keep and what to cut. Claude revised the text for grammar, consistency, and similar aspects, suggesting changes that proved very useful.

Although the manager did not measure the time precisely, they estimate that doing all the work alone would have taken about 15% more time.

AI thus proves to be an excellent companion and a helpful partner that speeds up and refines the work. However, it cannot yet replace human expertise. Together, they form a more effective duo than ever.

Future Prospects for AI in UX

One might wonder whether a guided browser tour would have improved the results. Today, there are tools that let models work on the live site in a browser, rather than relying only on static data and URLs. The manager did not test this approach, but suspects it would not have produced better results, because the main limitation lies in judgment, that is, understanding the context and objectives, weighing severity, and distinguishing real problems from noise. A live guided tour does not fundamentally change these aspects.

Looking ahead, the manager plans to develop their own agent skill for this task, building in established usability heuristics, severity weighting shaped by professional experience, and the expert eye that a UX practitioner brings to an interface. The goal is not to let the AI decide in their place, but to turn their own expert judgment into a reusable and consistent form.

This echoes what research on the subject shows: Baymard's AI heuristic evaluations at 95% accuracy were not reached through better prompts, but by anchoring the model in their own researched UX knowledge base. Admittedly, in their case the subject is usability heuristics, which is simpler than a complex, contextual evaluation like this one, but the principle remains the same. Accuracy comes from the expert framework you give the model. That is why it makes sense to encode your own standard in a skill.

The Future of AI in UX Design

The manager is aware that AI will eventually replace human work, and sees this as a certainty. The only question is when it will happen.

The manager has read articles suggesting a certain slowdown or plateau in AI development. According to a Reuters article from late 2024, several leading AI researchers, including Ilya Sutskever, believe that scaling up models has reached its limits. Future progress will come from new approaches, such as "o1"-type models that reason step by step, rather than from raw size. In other words, development continues, but the emphasis is on smarter methods rather than size.

Of course, other developments and research continue in the background, so there is no doubt that AI will keep progressing over the long term.

The manager recently came across a page suggesting that there are still a few good years ahead, notably in research and complex design, where using various AI models makes it possible to speed up, refine, and support the work.

Tips for Tomorrow

Start: Use AI as a partner for a first pass, feed it your real data, work in stages, and encode your own heuristics and severity into a reusable prompt or skill.
Stop: Do not deliver the AI's output as is, do not hope that a single prompt will replace a workflow, and do not delegate your judgment. Severity, prioritization, and business context remain your responsibility.

And do not panic at the thought that AI will take your job tomorrow. It will not, but designers who learn to steer it will outperform those who do not.