Vistaly: AI Redefines Opportunity Solution Trees

The Impact of AI on Product Development

Recently, I completed a particularly intense engineering project. Although I have been familiar with coding for several years, I do not consider myself formally trained as an engineer. My professional background is more oriented towards product management and design. However, the introduction of artificial intelligence into our professional lives is radically changing the game. It allows us to accomplish things we never thought possible. This experience is a perfect illustration of that.

Today, I want to share with you an engineering adventure. While some technical aspects may seem complex, it is important to emphasize that I had no prior knowledge of these concepts a year ago. Thanks to AI, I was able to learn and progress. My goal in sharing this story is to demonstrate what is now achievable thanks to these new technologies.

Launch of Opportunity Solution Trees by Vistaly

In February, I announced the development of two new services for Vistaly: AI-generated interview snapshots and AI-generated opportunity solution trees. We launched a call for partners to test these innovations, and over 100 companies expressed interest. After a rigorous selection process, we chose eight partners for the initial design phase.

Each team uploaded three customer interviews, from which we extracted key moments and opportunities. This information was then used to generate an opportunity solution tree. My role was to provide the AI services, while Vistaly focused on creating the user interface and associated workflows.

The initial feedback was extremely positive. Teams quickly expressed a desire to integrate more interviews, which required adjustments on our part to meet this growing demand.

The Challenges of Integrating New Interviews

Updating an opportunity solution tree with new interviews turned out to be a much more complex challenge than anticipated. Initially, I had underestimated the difficulty of this task.

Our shared goal with Vistaly is not just to produce AI-generated opportunity solution trees, but to enable users to interact with them, adjust them, and collaborate with the AI. We want to facilitate the synthesis of interviews without doing it entirely for them.

A screenshot from Vistaly shows how two opportunities can be merged, with visible and interactive modifications. Our aim is to clearly communicate each change, allowing the team to accept, modify, or reject the proposed changes. To achieve this, we decided to draw inspiration from the git diff tool, which is commonly used to track changes in code. We sought to create an equivalent for our opportunity solution trees.

This decision proved to be wise, but it also generated a considerable workload. It was not enough for our service to generate an updated tree; it also had to provide a detailed guide of the changes made.

The Prototype: A Step Among Others

A recurring theme in my podcast Just Now Possible is that it is relatively easy to create an impressive prototype with AI, but much harder to transform that prototype into a production-ready product. This has been exactly my experience.

My opportunity solution tree service consists of two sub-services: the first generates a new tree from scratch, and the second updates an existing tree with new interviews.

The first service was part of our initial alpha phase and worked correctly. However, the second service had to be developed from the ground up before our partners could add a fourth interview.

Each opportunity in the tree has sources and children that must not be lost during modifications such as splitting, merging, or deletion. While these elements may seem similar on the surface, they are actually very different. When updating a tree, it is crucial to preserve the existing structure as much as possible, unless the new content justifies a change.

Despite these challenges, I was pleasantly surprised to find that I was able to achieve a functional result in just a few days. Our design partners began using the service, and one of them quickly reached our beta limits. They then requested to switch to a paid subscription for full access, which delighted us.

However, as they uploaded more interviews, my initial design began to show its limitations. We observed unexpected behaviors.

The AI service generated an updated tree and provided a set of changes with detailed instructions. The Vistaly team noticed that these instructions did not always produce the expected final tree. We wanted to guide the client through each change, but sometimes, the tree generated by the instructions differed from the one produced by the service.

Vistaly reported this issue while I was heading to New Orleans for Jazz Fest. In hindsight, I am glad I did not know the extent of the work that awaited me upon my return. About 80% of the work remained to transform the tree update into a truly robust product.

A Deep Dive into the Problems

Upon my return, I set out to diagnose the problem and quickly realized the magnitude of the task. My service functioned as a pipeline combining LLM steps with deterministic code.

In examining the bugs, I understood that our approach was not optimal. The differences between trees, unlike linear documents, are ambiguous.

If I add a sentence to a document, the difference is clear. But if I split an opportunity into two and then merge one of them with another, the initial movement is obscured.

When A is split into A and B, the sources and children of A must be distributed between A and B. If the model decides to split an opportunity, it must determine how to distribute the sources and children between the new opportunities.

For example, if A has sources 1, 2, 3 and children x, y, z, after the split, A might have sources 1 and 2 and child x, while B would have source 3 and children y and z.

When B is merged with C, C inherits the sources and children of B. If C has sources 4 and 5 and children t, u, v, after the merge, C would have sources 3, 4, 5 and children t, u, v, y, z.

When comparing the input tree to the output tree, it seems that some sources and children of A move to C, but without a visible explanation. The split and merge are conceptually clear, but invisible in the difference.

I realized that tree differences were far more complex than those of linear texts. We needed not only to show what had changed but also to explain why, in order to engage the user. I needed to reconstruct each movement step by step.

This meant that the model had to show its work, which opened a new series of challenges.

Refactoring Prompts for Greater Transparency

I decided to refactor my prompts so that the model would generate both the final output and the intermediate steps.

My prompts already indicated possible actions: add, delete, rephrase, merge, split, etc. But these directives were not sufficient. The model now needed to show its work in the language of our change sets, specific movements understandable to the end user.

By rephrasing the prompts, I asked the model to detail each step of the process of creating the final tree. This involved breaking down each action into distinct movements that the user could follow and understand. For example, if an opportunity needed to be split, the model had to explain how it distributed the sources and children between the new opportunities. Similarly, for a merge, the model had to detail how the elements of the merged opportunities were combined.

This approach made the process of updating trees more transparent and understandable for users. By showing each step, we were able not only to clarify what had changed but also why those changes occurred, thereby strengthening users' trust in the system.

However, this refactoring also revealed new challenges. The model had to be capable of handling complex cases where multiple actions were necessary to achieve the final result. This required close collaboration between the development team and the AI model to refine the prompts and ensure that each step was properly documented.

In the end, this refactoring significantly improved the quality of the service, providing users with a clearer and more detailed view of the changes made to their opportunity solution trees. It also reinforced our commitment to transparency and collaboration, demonstrating how AI can be used to facilitate complex processes while remaining accessible and understandable for end users.