Blind Test: Astra Impresses, Opus 5.5 Excels in Certain Tasks

Claire Vo confronted several models live, including GPT-6 Astra, GPT-6 Sol, and Claude Opus 5.5, on a panel of tasks ranging from email to video editing. Her personal ranking diverges from that of an LLM judge and highlights different strengths depending on the use cases, with clear limitations in 3D and creativity.

An LLM Judge Challenges the Human Verdict and 3D Reveals Its Limits

An automated judge did not validate the rankings established by Claire Vo, emphasizing criteria that she claims she did not notice. She reused Barbie Bench, a 3D fashion game, to test the visual capabilities and spatial coherence of the models: the generated hands remain problematic. For Claire Vo, these results confirm that AGI has not yet been achieved. The program also details the reasons for the disagreement between the human evaluation and that of the LLM judge.

Astra Preferred, Opus 5.5 Strong on Agents and B2B Frontend, Sol Clear and Affordable

Claire Vo indicates that she had a preference for GPT-6 Astra, while considering that Opus 5.5 dominated her week of testing. She remains ambivalent about GPT-6 Sol. According to her, Opus 5.5 could be the most performant overall, particularly on long-duration agents and B2B frontend. She highlights Sol's clarity of writing, the readability of its PRDs, and its pricing. The SVG illustration tests reversed her initial prediction regarding Anthropic. For video editing, she expressed disappointment, attributing part of this result to the skills required. The final results were detailed for Astra, Sol, and Opus 5.5, with remarks on writing clarity and creative surprises, as well as a breakdown of the areas where each model excels according to her.

A Blind, Multi-Task Protocol Conducted Live

Claire Vo's How I AI bench test was conducted blind: each output was rated without revealing its author. She carried it out live, on the very morning that Anthropic and OpenAI launched new models. The protocol covered a variety of tasks: emails, PRDs, frontend prototypes, backend, long-duration agents, SVGs, and video editing. Aspects such as barriers, personality, and the speed of the models were addressed. Results were drawn for personal productivity, frontend prototypes, backend tasks, and long-duration agents. An SVG illustration test was added, and predictions were made before revealing the models behind each production. GPT-6 Luna was mentioned among the new releases discussed.