AI Product: Prioritizing Error Hunting

A three-step workflow, achievable in about 30 minutes with a coding agent, has been implemented in over 50 companies to uncover major defects in AI products. This approach prioritizes error discovery before writing metrics and relies on comprehensive session traces to transform product intuition into repeatable tests.

A 3-Step Protocol Applied in Over 50 Companies

A three-step process enables error discovery using a coding agent like Codex or Claude. Used with more than 50 companies, it has revealed significant defects affecting customer experience with each deployment. Once the basics are established, the entire workflow can be completed in about 30 minutes.

Starting with Traces and Documenting Them Properly

The first step involves working from traces, which are essential for error discovery. Each trace includes user input, the system prompt, all intermediate actions (retrievals, tool calls, model calls), and the final output of the product. These elements must be sufficient for an examiner to reconstruct the sequence and assess its quality.

Instrumentation, Local Storage, and Possible Providers

Traces can be stored in a database, via an evaluation provider, or simply in a local folder. For simplicity, a local folder may be chosen. If the application does not yet produce traces, a coding agent can instrument it to record each user session as a complete trace, including input, prompt, tool calls and their results, retrieved context, intermediate model calls, and final output. If providers like LangSmith, Arize, Phoenix, or Langfuse are already in place, it is recommended to continue with them while writing a local copy in JSON Lines format (one object per session) in traces/traces.jsonl. In the absence of a provider, this JSONL file is sufficient. Once instrumentation is complete, user activity should be allowed to accumulate to collect traces.

Measured Gains: Lower Costs and Increased Accuracy

Companies report tangible effects after structuring their products around evaluations. Shopify has designed an AI flow builder that is 2.2 times faster and 68% cheaper than the previous system. Cursor indicates a significant increase in user satisfaction and a 41% reduction in costs due to improved Auto Balance routing. Ramp reports an increase in accuracy from 35% to 83% in automatic receipt collection. Harvey has nearly doubled its internal quality score by rebuilding its contract reviewer with evaluations. Rippling, Glean, Abridge, ElevenLabs, and Robinhood also describe systematic use of evaluations to advance their products.

Why Prioritize Error Discovery Before Metrics

Many teams skip error discovery to write metrics because sifting through sessions seems slow and metrics appear easier to automate. This rush leads to formulating inappropriate hypotheses and measuring incorrectly. Error discovery is to evaluation what product discovery is to the product: it identifies which failures are worth measuring and avoids generic dashboards that waste time and lead to poor outcomes. In AI systems that are easy to modify but hard to predict, where one change can deteriorate another, evaluations transform judgment into repeatable tests, convert production errors into test cases, and enable faster delivery by automatically verifying proper functioning. The recommended process includes error discovery and analysis, creation of custom metrics, and a continuous improvement loop; if only one step should be addressed, error discovery should be prioritized.

A Skill Adopted by PMs and Supported by Leaders

The writing of evaluations is gaining traction among product managers, with nearly half of 25 recent job postings requiring this skill. Mike Krieger emphasizes that the ability to write evaluations is likely the most important skill to impart to product leaders. Garry Tan describes it as a strategic bulwark for AI startups. Several stakeholders equate evaluations to new PRDs, while companies claim to be reaping the benefits of their investments in this area.