BNP Paribas denounces tokenmaxxing and bets on AI ROI

Tokenmaxxing Questioned by BNP Paribas

Charles Holive, head of artificial intelligence at BNP Paribas CIB, recently shared his views on tokenmaxxing, a popular practice in Silicon Valley. According to him, this method, which involves maximizing the use of AI and thus the consumption of tokens, is a "vanity metric." Holive emphasized that his team prefers to evaluate the success of AI in terms of revenue generated and productivity gains, rather than based on the number of tokens used.

At the Mistral AI summit in Paris, Holive explained that while his team tracks token consumption, it is not the priority. "We are trying to move away from vanity metrics — billions of tokens per day," he stated. He added that the goal is to focus on concrete outcomes rather than superficial metrics.

Instead of concentrating on the number of tokens, Holive asks different questions of his employees: "What have you done that you weren't doing before?" and "How much faster did you do it?" These questions aim to measure the real impact of AI on productivity and innovation within the company.

American Companies Reassess Their AI Spending

Holive's comments come at a time when several American companies are beginning to question the profitability of massive investments in AI. Amazon, for example, recently shut down an internal leaderboard on AI usage after employees began performing tasks solely to improve their ranking. Meanwhile, Andrew Macdonald, COO of Uber, has publicly expressed doubts about whether rising AI costs translate into more useful products.

GitHub has also adjusted its pricing model for Copilot, moving to a usage-based pricing in response to rising AI bills. These moves reflect a growing trend among companies to reassess the effectiveness of AI spending and seek ways to maximize return on investment.

A Focused Approach on Return on Investment

For the AI projects he oversees at BNP Paribas CIB, Holive specified that his team starts by formulating explicit hypotheses about the amount of revenue or productivity they could generate. Then, key performance indicators (KPIs) are created to track progress monthly or quarterly against these goals. While token consumption is monitored to control costs, it is not the priority.

Holive noted that token usage can still be useful as a measure of adoption, particularly in software engineering. However, he clarified that his approach differs from that of companies that heavily focus on AI usage metrics. "We haven't said, 'use the tool, do your best,'" he stated. "We did the opposite of that."

Reactions from Other Industry Leaders

Holive's approach resonated with other leaders present at the summit. Amit Kapur, head of AI and transformation at Tata Consultancy Services, stated that companies should focus on "business outcomes" and "business impacts" rather than solely on tokens as a line item. Antoine Pichot, director of innovation, digital, and data at La Banque Postale, indicated that his bank evaluates AI projects based on factors such as efficiency gains, customer satisfaction, employee satisfaction, and financial impact.

Holive asserted that while token usage may be an indicator of adoption, it is crucial not to get trapped by metrics that do not reflect the true impact of AI on the business. This perspective could encourage other companies to rethink their success criteria for AI, thereby influencing technology investment strategies in the future.