Claude Code: Measuring AI by Human Impact

⚡

Key Takeaways

1Boris Cherny, creator of Claude Code, proposes measuring the success of AI by the human hours saved rather than by the tokens used.

2He suggests a four-step framework for integrating AI in business, emphasizing the real impact on team work.

3The trend of "tokenmaxxing" is declining, as companies seek ways to optimize their return on investment in AI.

💡Why it matters — This approach could transform how companies evaluate and justify their AI spending, focusing on efficiency and innovation.

A New Approach to Measuring AI Success

Boris Cherny, the mastermind behind Claude Code and designer of Anthropic's most important coding tool, proposes a new method for evaluating the effectiveness of artificial intelligence technologies. Instead of focusing on the number of tokens consumed, Cherny suggests that companies should measure the hours of human work saved through AI. This proposal comes as many businesses seek to optimize their rising expenditures on artificial intelligence. The return on investment (ROI) of AI has become a central topic in the business world.

A Four-Step Framework for AI Adoption

In a series of posts on X Thursday, Boris Cherny outlined a four-step framework for integrating AI into businesses. During the third step, he emphasized the importance of measuring returns once AI is integrated into employees' workflows. He explained that simply monitoring usage via a dashboard is insufficient, as it only reflects activity and not the actual return on investment.

Evaluating Saved Engineering Efforts

Cherny proposes asking the following question: if AI had not been used, how much engineering effort would have been required to accomplish the same task? By calculating the cost in hours of manual engineering, companies can better assess the return on investment of AI. He insists that this approach allows for a clearer understanding of AI's real impact on team efficiency.

The Impact of AI on Innovation

In a subsequent post, Cherny stated that the true advantage of AI manifests when corrections and maintenance occur in the background, freeing teams to focus on creation. This paves the way for innovations that were previously unimaginable. Companies can thus explore new opportunities and push the boundaries of what is possible.

Thoughts on AI's Return on Investment

This is not the first time Cherny has addressed the topic of AI's return on investment. During a discussion with Scale AI, he emphasized that companies need to consider the added value provided by AI, which can manifest in various ways, such as accelerating code development by engineers.

The End of "Tokenmaxxing"

Cherny's comments come at a time when the trend of "tokenmaxxing" is fading. Companies are now looking to maximize the value obtained for every euro spent on AI. Firms like Coinbase and Vercel have shared strategies to reduce costs while maintaining AI usage, such as opting for more affordable Chinese models.

The Rising Cost of AI

The debate over AI's return on investment is intensifying, with influential figures in the business world taking a stand. Jamie Dimon, CEO of JPMorgan, recently stated to CNBC that AI-related costs are rising rapidly, prompting companies to adopt a rational approach. Sam Altman, CEO of OpenAI, also highlighted the importance of this issue during the Allen & Co. conference in Sun Valley, mentioning in an interview with CNBC at the Idaho event that reducing expenses and increasing value are major concerns for businesses.