Vistaly: AI Redefines Opportunity Solution Trees
The Impact of AI on Product Development
Recently, an executive completed a particularly intense engineering project. Although he has been familiar with coding for several years, he does not consider himself formally trained as an engineer. His professional background is more oriented towards product management and design. However, the introduction of artificial intelligence into everyday professional life is radically changing the game. It makes it possible to accomplish things one would never have thought possible. This experience is a perfect illustration of that.
Today, this executive wants to share an engineering adventure. While some technical aspects may seem complex, it is important to emphasize that he had no prior knowledge of these concepts a year ago. Thanks to AI, he was able to learn and progress. His goal in sharing this story is to demonstrate what is now achievable thanks to these new technologies.
Launch of Opportunity Solution Trees by Vistaly
In February, the executive announced the development of two new services for Vistaly: AI-generated interview snapshots and, also AI-generated, opportunity solution trees. A call for partners was launched to test these innovations, and over 100 companies expressed interest. After a rigorous selection process, eight partners were chosen for the initial design phase.
Each team uploaded three customer interviews, from which key moments and opportunities were extracted. This information was then used to generate an opportunity solution tree. The executive's role was to provide the AI services, while Vistaly focused on creating the user interface and associated workflows.
The initial feedback was extremely positive. Teams quickly expressed a desire to integrate more interviews, which required adjustments to meet this growing demand.
The Challenges of Integrating New Interviews
Updating an opportunity solution tree with new interviews turned out to be a much more complex challenge than anticipated. Initially, the difficulty of this task had been underestimated.
The shared goal with Vistaly is not just to produce AI-generated opportunity solution trees, but to enable users to interact with them, adjust them, and collaborate with the AI. The aim is to make the synthesis of interviews easier without doing it entirely for them.
A screenshot from Vistaly shows how two opportunities can be merged, with visible and interactive modifications. The aim is to clearly communicate each change, allowing the team to accept, modify, or reject the proposed changes. To achieve this, it was decided to draw inspiration from the git diff tool, which is commonly used to track changes in code. An equivalent for opportunity solution trees was therefore sought.
This decision proved to be wise, but it also generated a considerable workload. It was not enough for the service to generate an updated tree; it also had to provide a detailed guide of the changes made.
The Prototype: A Step Among Others
A recurring theme in the Just Now Possible podcast is that it is relatively easy to create an impressive prototype with AI, but much harder to transform that prototype into a production-ready product. This is exactly what was experienced.
The opportunity solution tree service consists of two sub-services: the first generates a new tree from scratch, and the second updates an existing tree with new interviews.
The first service was part of the initial alpha phase and worked correctly. However, the second service had to be developed from the ground up before partners could add a fourth interview.
Each opportunity in the tree has sources and children that must not be lost during modifications such as splitting, merging, or deletion. While these elements may seem similar on the surface, they are actually very different. When updating a tree, it is crucial to preserve the existing structure as much as possible, unless the new content justifies a change.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
Despite these challenges, he was pleasantly surprised to find that a functional result could be achieved in just a few days. The design partners began using the service, and one of them quickly reached the beta limits. They then asked to switch to a paid subscription for full access, which was a source of satisfaction.
However, as they uploaded more interviews, the initial design began to show its limitations. Unexpected behaviors were observed.
The AI service generated an updated tree and provided a set of changes with detailed instructions. The Vistaly team noticed that these instructions did not always produce the expected final tree. The intention was to guide the client through each change, but sometimes the tree generated by the instructions differed from the one produced by the service.
Vistaly reported this issue while the executive was heading to New Orleans for Jazz Fest. In hindsight, he is glad he did not know the extent of the work that awaited him upon his return. About 80% of the work remained to turn the tree update into a truly robust product.
A Deep Dive into the Problems
Upon his return, an analysis was undertaken to diagnose the problem, and the magnitude of the task quickly became clear. The service functioned as a pipeline combining LLM steps with deterministic code.
In examining the bugs, it became apparent that the approach was not optimal. The differences between trees, unlike linear documents, are ambiguous.
If a sentence is added to a document, the difference is clear. But if an opportunity is split into two and then merged with another, the initial movement is obscured.
When A is split into A and B, the sources and children of A must be distributed between A and B. If the model decides to split an opportunity, it must determine how to distribute the sources and children between the new opportunities.
For example, if A has sources 1, 2, 3 and children x, y, z, after the split, A might have sources 1 and 2 and child x, while B would have source 3 and children y and z.
When B is merged with C, C inherits the sources and children of B. If C has sources 4 and 5 and children t, u, v, after the merge, C would have sources 3, 4, 5 and children t, u, v, y, z.
When comparing the input tree to the output tree, it seems that some sources and children of A move to C, but without a visible explanation. The split and merge are conceptually clear, but invisible in the difference.
It became clear that tree differences were far more complex than those of linear texts. It was necessary not only to show what had changed but also to explain why, in order to involve the user. Each movement had to be reconstructed step by step.
This meant that the model had to show its work, which opened a new series of challenges.
Refactoring Prompts for Greater Transparency
It was decided to refactor the prompts so that the model would generate both the final output and the intermediate steps.
The prompts already indicated possible actions: add, delete, rephrase, merge, split, etc. But these directives were not sufficient. The model now needed to show its work in the language of change sets, specific movements understandable to the end user.
By rephrasing the prompts, the model was asked to detail each step of the process of creating the final tree. This involved breaking down each action into distinct movements that the user could follow and understand. For example, if an opportunity needed to be split, the model had to explain how it distributed the sources and children between the new opportunities. Similarly, for a merge, the model had to detail how the elements of the merged opportunities were combined.
This approach made the process of updating trees more transparent and understandable for users. By showing each step, it became possible not only to clarify what had changed but also why those changes occurred, thereby strengthening users' trust in the system.
However, this refactoring also revealed new challenges. The model had to be capable of handling complex cases where multiple actions were necessary to achieve the final result. This required close collaboration between the development team and the AI model to refine the prompts and ensure that each step was properly documented.
In the end, this refactoring significantly improved the quality of the service, providing users with a clearer and more detailed view of the changes made to their opportunity solution trees. It also reinforced the commitment to transparency and collaboration, demonstrating how AI can be used to facilitate complex processes while remaining accessible and understandable for end users.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.