Brief IA
›

AI in Business: Verification Also Comes at a Cost

Business & Startups·Tom Levy·

AI in Business: Verification Also Comes at a Cost

AI in Business: Verification Also Comes at a Cost
Key Takeaways
1In Berkeley, researchers followed a technology company with 200 employees for eight months.
2In an experiment with GPT-4, consultants lost 19 points on a problem that exceeded the model's capabilities.
3SocIAty begins its missions with an audit of actual activities.
Why it matters : the proposed evaluation includes control time, not just generation time.

At Berkeley, researchers have observed an expansion of tasks in a technology company using AI. The results of an experiment with BCG consultants show that assistance can degrade certain responses. At Ceetadel as well as at SocIAty, the review of work precedes the design of agents.

Berkeley Observes Work Encroaching on Breaks

In 2026, researchers from Berkeley studied a technology company with 200 employees for eight months. They found that tasks were expanding and that professional activity often spilled over into breaks, frequently at the initiative of the employees. Maggie Ye, the head of this research, describes the shift from extra effort to a performance that is now expected.

At SocIAty, the support for companies is based on a finding: generating ten scenarios requires little effort, but reviewing them takes time. According to the speaker, the burden thus shifts towards verification, without appearing in the usual indicators. He recommends that managers make this work visible and be able to refuse certain requests, including those coming from their hierarchy.

GPT-4 Does Not Improve All Consultant Tasks

In 2023, an experiment conducted by researchers from Harvard, MIT, and Wharton involved 758 BCG consultants using GPT-4. The researchers observed a net gain in creative tasks. However, on an analysis problem outside the model's scope, assisted consultants scored 19 points lower than others, having placed their trust in an incorrect response.

The speaker from SocIAty emphasizes that models have evolved since then, but their limitations remain. He recommends clearly defining the tasks assigned to the tool and the acceptance criteria for the results. For a business recommendation, it is important to verify margin assumptions rather than just the presentation, and to revisit a first draft delivered in ten seconds if necessary.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

The approach also includes evaluating the employee's understanding and their ability to defend a recommendation. It encourages sharing mistakes to turn them into test cases or rules, and values objections that reveal a concrete limitation. The maintenance of skills falls to the teams, while the company must allow for the exercise of critical thinking.

Ceetadel Analyzes Agencies Before Designing Agents

At Ceetadel, several weeks were dedicated to understanding the actual activities of each agency before any agent design. The architectures are then submitted to the teams for discussion and potential contestation, rather than simply validation. The stated principle is to choose tools based on observed work.

SocIAty has supported the deployment of several hundred agents and begins each mission with an audit of the actual work performed. According to the speaker, the main obstacle is almost never technical, but rather the decision of what should be delegated, controlled, or abandoned. He suggests evaluating the deployment of AI by identifying the activities that the company has been able to stop thanks to it.

MIT Estimates Financial Effects Still Limited

According to MIT, by the summer of 2025, 95% of experiments in generative AI had not produced a measurable impact on the income statement. The speaker from SocIAty contests this percentage but notes that the lack of time to train in AI often comes up as an argument, both from teams and from managers who initiated these projects.

The recommended changes focus on work methods, priorities, and the organization of controls, rather than just the installation of software. They include learning time, unsuccessful trials, the possibility of revising objectives, or eliminating meetings. To measure the benefit, the duration considered must extend to a result that someone is willing to sign off on, including controls.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.