Vistaly Revamps Its Tool with AI Opportunity Trees

The opportunity tree specialist has rewritten its application to allow an AI to synthesize interviews and keep the tree updated. Beyond prompts, the team describes an orchestration with repair loops, evaluation safeguards, and a transition to Bedrock dictated by data residency. Making the generated changes understandable remains the most challenging aspect.

Compliance and Infrastructure Mandate Bedrock in Europe

Vistaly cites data residency, SOC 2, and GDPR as reasons for the shift to European Bedrock inference. This choice aligns with a stated residency constraint that has shaped the architecture. The company reports that Bedrock constrains tokens regardless of the model used. It adds that model upgrades are not direct replacements, as prompts are specific to each model and its version.

Orchestration and Evaluations Replace Prompt Iteration

The team explains that a change in orchestration resolved a flaw that prompt improvement had not corrected. It describes the exhaustion of prompt-related gains and the need for an agentic repair loop to balance two opposing error modes. Four new evaluations and 16 experimental variants have been introduced to manage a trade-off between missing subgroups and poorly formulated parents. Before engaging an LLM as a judge, a code assertion, such as detecting an excessive number of children on a node, served as a pre-filter. This evaluation was then established as a production barrier. Teresa Torres further reports that she has not been able to calibrate an LLM as a judge.

Understanding Changes, Declared UX Priority

According to Vistaly, the main challenge is not the raw quality of the outputs but helping users understand what has changed. The company observes that users prefer to receive an answer and then be able to correct it, rather than engage in a step-by-step collaboration with the AI. It emphasizes that layered analysis takes precedence over the choice between pipeline or agent, as a misstep in interview preview propagates to higher levels. Change sets cannot be evaluated at the end of the chain, as multiple valid sequences exist for the same input, with only one semantically coherent sequence remaining intelligible. The team indicates that it has taught the agent rules of the game to produce these sets and record semantic movements, such as merging, moving, or reformulating.

Complete Product Redesign and New Workflow

Vistaly decided to start anew rather than integrate AI features into V1, and halted sign-ups for this version during the rewriting phase. According to the team, it took them two and a half months to restore nearly all the functionalities of V1, after three years dedicated to developing the first version. V2 offers the ability to import interviews, generate an overview for each, and then produce a first version of the tree, with an agentic workflow dedicated to drafting and updating. The founders claim that the agent component was the easiest part of the project.

Data Quality and Testing: From Field to Synthetic Transcriptions

Data from the beta highlighted unsuitable "interviews," including sales demos, meetings with stakeholders, and transcriptions produced by LLMs and imported as interviews. To conduct tests in a controlled environment, Vistaly established a system with 40 points to create synthetic transcriptions close to reality. Tree modifications are made via MCP, which includes matching an entire backlog of Jira epics to the opportunity space, and integrating a feedback agent revealed issues related to versioning and restoration. Teresa Torres indicates that essential elements, such as stable identifiers and traceability, were overlooked at the prototype stage. On the usage side, the team describes a chat synthesis perceived as too slow, and a house of cards effect where a poorly formulated opportunity contaminates higher levels, to the point where a client anticipates needing to clean up a branch. The guided interface of V1, faster than before, was not enough to meet expectations, which Vistaly attributes to a competitive context dominated by superficial syntheses rather than good syntheses that are faster.