Brief IA
›

Gleb Tsipursky: Measuring AI by Accepted Outcomes

Business & Startups·Tom Levy·

Gleb Tsipursky: Measuring AI by Accepted Outcomes

Gleb Tsipursky: Measuring AI by Accepted Outcomes
Key Takeaways
1Gleb Tsipursky recommends calculating the cost per accepted business outcome.
2At an insurance company, summary assistants and exact quote assistants have been separated.
3He advocates for randomly comparing similar cases with and without AI assistance.
Why it matters : The proposed evaluation includes data preparation, checks, and corrections after delivery.

Gleb Tsipursky, a behavioral scientist and CEO of Disaster Avoidance Experts, recommends that CIOs measure the cost of an accepted business outcome rather than the cost of a response generated by AI. At an insurance company, he reorganized the assistants and reserved certain tasks for simple rules.

The cost also includes subsequent corrections

The calculation proposed by Gleb Tsipursky covers three stages. Upstream, it includes the time needed to gather context and clean the data provided to the system. During production, it incorporates the model's billing as well as controls and corrections. After delivery, it also takes into account exceptional cases, customer claims, and subsequent adjustments.

The approach also stipulates designating a business owner and a technical owner for any substantial use. It requires specifying the conditions for suspending or abandoning the application. In the event of failure, the business owner must be able to halt or reorganize the process.

An insurer separates the tasks of assistants

The assistant tested by the company exhibited a display failure: some lengthy content containing insurance clauses did not appear correctly. However, claims managers needed the exact wording of contracts, both to review claims and to draft letters to policyholders. Without reliable output, a response could therefore be unusable.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Gleb Tsipursky transferred the contract-related advice to Microsoft Teams, which handled these lengthy contents better. He also distinguished between assistants responsible for summaries and those tasked with reproducing exact quotes for letters. This separation better met the presentation and drafting constraints of the correspondence.

For another project with the same insurer, focused on identifying potentially problematic comments in forms, he opted for a progressive treatment. Basic rules and keyword searches could first identify obvious difficulties; only more nuanced comments required interpretation by AI. The handling of common cases thus became less complex and more transparent.

Tsipursky recommends a comparison with existing methods

To decide on the use of AI, Gleb Tsipursky advocates for a controlled commercial experiment. His protocol begins with a precise definition of the work to be evaluated, then compares AI assistance with the best process already available. Comparable cases should be randomly assigned between conventional methods and those utilizing AI.

He also recommends assessing progress made on identified issues, rather than the amount of AI used, to avoid rewarding usage over results. Teams should present their successes, but also the evidence that led them to abandon AI or simplify its use.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.