AI Under Scrutiny: Measuring Its Real Commercial Impact

The Rise of AI Pilot Projects

Last year was marked by a proliferation of artificial intelligence (AI) pilot projects within companies. These initiatives were often accompanied by subscriptions to large language models (LLMs) and increased oversight by managers regarding their use by employees. Discussions around AI were ubiquitous, illustrated by phrases like "AI wrote my memo."

However, despite the initial enthusiasm, widespread disappointment has set in regarding the actual impact of these projects on business performance. With the recent drop in the stock prices of SaaS companies, the crucial question now is whether these AI projects are truly delivering tangible results. AI, while a promising invention, still needs to prove its value as a viable economic innovation.

Towards Agentic AI

The future of AI seems to be heading towards systems capable of making autonomous decisions, also known as agentic AI. These systems must demonstrate their ability to generate measurable business outcomes to be considered true innovations. An invention becomes an innovation when it is associated with a solid business model.

To assess the effectiveness of these systems, it is essential to focus on outcomes rather than activities or anecdotes. Andy Grove, former CEO of Intel, proposed a pragmatic approach to measuring organizational performance based on concrete results. His book, "High Output Management," provides a classic framework for evaluating the results of middle managers, emphasizing the importance of measurement.

Grove argues that organizations should focus on outcomes rather than inputs. Following this logic, it is crucial to first define the business outcome one wishes to achieve, and then measure agentic AI solely based on the improvement of that performance criterion.

The Example of Trax Technologies

Trax Technologies, a company in the Strattam portfolio, exemplifies this approach well. Specializing in global shipment management for large multinationals, Trax identified a key opportunity in using AI to resolve certain exceptions in freight invoices. The company developed the AI Audit Optimizer with the goal of reducing the number of exceptions requiring human intervention.

In the first quarter following the launch of this tool, Trax successfully resolved approximately 826,000 exceptions. Although this figure was promising, it was not yet sufficient to make a significant impact. In the second quarter, the system showed no improvement, remaining stuck at that level. Trax then experimented with different approaches to enhance results.

In the third quarter, the company discovered that a human prompt engineer interacting with the system could make a significant difference. Thanks to this intervention, the resolved exceptions tripled in the fourth quarter, reaching 2.5 million. Trax used data from both successful and unsuccessful resolutions to retrain the system and set ambitious quarterly goals to continue improving performance.

With this output goal in mind, Trax is moving forward by adjusting the interaction points between the prompt engineer and the system. The company has utilized data from both successful and unsuccessful resolutions to retrain the system. It has also set quarterly targets; next quarter, it aims for Trax's AI Audit Optimizer to resolve more than in any previous quarter.

Measuring What Matters

In a context where AI is often surrounded by hype, it is crucial for companies to focus on the concrete results it can deliver. Rather than being seduced by tools or anecdotes, it is essential to determine the outcome measures that truly matter. Drawing inspiration from the teachings of Drucker and Grove, companies can ensure that AI justifies its place by generating tangible business results.

To succeed in this adaptation, it is important to resist the urge to purchase tools or launch pilots without a clear strategy. Companies must focus on measuring results and adjusting their AI tools accordingly. By adopting this approach, they can ensure that AI provides real added value and contributes to their long-term success.