Cognition and Token Usage: Scott Wu Critiques the Abuses

Scott Wu Questions Token-Based Evaluation

Scott Wu, who co-founded Cognition in 2023, recently shared his views on how employees should be evaluated. According to him, ranking based on token usage is not the best method. Cognition, a company valued at $26 billion, is best known for Devin, an AI-powered coding tool.

In the tech sector, excessive token usage is often criticized for its potentially wasteful nature. Scott Wu, CEO of a rapidly growing AI coding startup, addressed this issue during an episode of the "Founders" podcast that aired last Sunday. He emphasized that there are more effective ways to encourage employees to use AI productively.

Wu stated, "It's directionally correct, but I think there are places where people go too far." He added that rather than ranking engineers based on the number of tokens spent, it would be wiser to evaluate them based on their actual output.

A Growing Company

Cognition was founded in 2023 by Scott Wu, Steven Hao, and Walden Yan. Based in San Francisco, the company has attracted attention from prestigious investors such as Founders Fund, Khosla Ventures, Elad Gil, and Pear VC. In May, Cognition raised over $1 billion, reaching a post-money valuation of $26 billion, making it one of the most valuable AI coding startups in the world.

Wu acknowledged that using AI can be costly. However, he pointed out that if engineers can triple their output through AI, the investment is worth it. He nevertheless warned against tying rewards to overly ambitious metrics, suggesting instead to focus on concrete results such as the number of tickets resolved, as well as the speed and cost of projects.

Industry Criticism of Token Usage

Scott Wu is not the only tech leader to criticize excessive token usage. Many AI tools, such as Claude, Codex, and Cursor, are used to enhance productivity and influence internal evaluations. However, this approach is being questioned.

Jacob Lauritzen, CTO of the legal AI startup Legora, recently stated during a podcast that employees should not be rewarded solely for their use of AI. He explained that this leads to excessive token consumption, where employees use tokens merely to improve their standing during performance evaluations.

"It's a really stupid way to do anything," he added, highlighting the inefficiencies of this practice.

During a Bloomberg conference, Andrew Feldman, CEO of Cerebras Systems, also criticized the idea of giving employees unlimited tokens, calling it "absurd from the start." He compared this approach to buying a Ferrari for grocery shopping, suggesting instead the use of more economical open-source models.

"What we're learning is how to shop at Costco," he concluded, illustrating the importance of efficiency in the use of AI resources.