Anthropic's Sonnet 5: A Revealing Benchmark for Developers

Sonnet 5: A Detailed Benchmark for Developers

Claire, an artificial intelligence expert, set out to test the latest model from Anthropic, Sonnet 5, by comparing it to several other leading models. She used a benchmark she developed herself, the How I AI Bench, to objectively assess the capabilities of Sonnet 5. This test was conducted blind, pitting Sonnet 5 against models like Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro. The evaluation criteria included performance on specific tasks, prototype creation, and personality analysis of the agents.

A Strategic Pricing Position

Sonnet 5 stands out for its pricing, which is closer to previous models in the Sonnet range than to competitors like Opus. With a cost of $2 per million input tokens and $10 per million output tokens until the end of summer, Sonnet 5 positions itself in a mid-range bracket. However, during Claire's tests, Sonnet 5 did not excel, ranking relatively low in her personal assessment. This suggests that Sonnet 5's attractive pricing does not compensate for a performance that must be judged based on each user's specific needs.

The Importance of Repeatability in Testing

One-off tests, while initially useful, often lack repeatability, which is crucial for truly evaluating a model's performance. Claire found that models like GPT-5.5 and GLM-5.2, while effective at the moment, did not allow for reliable comparison of results over time. Claire's How I AI Bench overcomes this issue by using standardized inputs and a fixed framework, ensuring that each new model is evaluated consistently. Claire utilized Claude Code to build this live benchmark, enabling rigorous and repeatable assessment.

Leveraging Context with Claude Code

One of the interesting features of Claude Code is its ability to use session history to generate benchmark ideas tailored to the user's real needs. Claire was able to ask Claude Code to suggest evaluation tasks based on their previous work, leveraging data recorded on her desktop. This demonstrates how developers can take advantage of this often underutilized history to enhance their own benchmarks.

Evaluating Outputs with Intuition

Building an HTML scoring page to evaluate model outputs based on intuition may seem tedious, but Claire asserts that it is worth it. She manually evaluated 64 generations across five models, assigning intuition scores from 1 to 5 and exporting the results in JSON. This process revealed that human intuition remains a crucial element of the benchmark, often more revealing than automated assessments.

The Limitations of Automated Evaluations

Evaluations by language models, while useful, tend to be overly generous and cluster around the average. Claire asked GPT-5.5 and Opus 4.8 to judge the outputs, but these models missed details that Claire immediately noted, such as flawed prototypes. This observation underscores that current models cannot yet compete with the human eye in detecting certain types of errors.

Divergence Between Human and Automated Judgment

Claire noticed a significant divergence between her own judgment and that of automated evaluations. The models ranked Gemini 3 Pro at the top and Sonnet 4.6 at the bottom, while Claire had the opposite opinion. By using a weighted index of 70/30 in favor of her judgment over that of the LLMs, Sonnet 4.6 moved to the top, indicating that automated evaluations need to better reflect the criteria that truly matter to users.

Personal Preferences and Performance

Despite lower benchmark scores, Sonnet 4.6 remains Claire's preferred model for her daily tasks, primarily due to its personality. She uses Sonnet 4.6 to run her OpenClaw, appreciating how the model interacts with her. Claire specifically pays for API credits to maintain this interaction, as no other model has been able to match Sonnet 4.6 in voice evaluation.

Recommendations for Developers

For developers, Claire recommends GPT-5.5 for PRDs, Sonnet 4.6 for prototypes and discussions, and Opus 4.8 or Sonnet 5 for code navigation. These recommendations are based on a weighted index that reflects Claire's preferences. Opus 4.8 remains relevant for complex UI work, while Sonnet 4.6 is sufficient for simpler tasks.

Future Improvements for the How I AI Bench

The How I AI Bench is still in development, currently in version one, and Claire plans to implement several improvements. The agent bug-hunting task, for example, proved too easy, with every model succeeding. Claire is considering removing this task and integrating more of her personal preferences into the evaluation framework to make the benchmark more relevant. She also plans to continue conducting blind benchmarks whenever a new model is released to ensure ongoing and objective assessment.

Managing Autonomous Coding Agents with OpenAI Symphony

Alessio Fanelli, founder of Kernel Labs, shared with Claire his methods for managing autonomous coding agents from his phone. By using OpenAI Symphony, Linear, and a cloud VPS, he transformed his approach to agent management, shifting from "agent prompter" to "agent manager." This transition allowed him to overcome the limitations of local execution environments and cluttered interfaces.

The Key to Asynchronous Management

Alessio explained that moving to a cloud VPS with multiple communication channels made asynchronous management of agents possible. This enabled projects to maintain momentum, unlike earlier workflows that would stall after a few interventions.

Symphony: A Highly Opinionated Markdown Specification

Symphony is not a magic solution but a highly opinionated Markdown specification that guides model behavior. Claire pointed out that this simplicity is often misunderstood, as the framework is simply a Markdown file that models adhere to.

Tracking Token Costs

Tracking token costs per task is essential for optimizing agent configurations. Alessio showed examples of tasks with costs ranging from 15 million to 221 million tokens, highlighting the importance of this tracking for improving specifications and tools.

Managing Skill Files

Alessio recommends regularly purging skill files to avoid contradictions. Models tend to accumulate instructions, which can become problematic. Keeping files short and precise is crucial for effective management.

Untapped Opportunities in AI

AI offers opportunities for businesses based on heterogeneous data, such as collectible cards or vintage clothing. Language models are capable of handling this complex data without requiring extensive preprocessing, opening new business possibilities.

Enhancing Agent Perceptions

To prolong autonomous executions, it is crucial to provide agents with better perceptions, such as screenshots or videos. Kernel Labs developed Glimpse, a Playwright extension, to enhance agents' perception in the face of user interface ambiguities.

Offloading Context

Context offloading is an underestimated application of AI. Alessio created an email monitoring system that ensures he does not miss any important information, thereby reducing his anxiety.

Compressing the Information Advantage

AI reduces the information advantage that scale once offered. For example, finding undervalued Pokémon cards, a task previously reserved for experts, is now accessible thanks to tools like Codex.

The Impact on Small Businesses

Small businesses greatly benefit from AI, often underestimated in media narratives. Alessio observed in Japan that small operations function efficiently thanks to AI, providing an advantage that larger organizations cannot replicate at the same cost.