Brief IA
›

AI: User Feedback Synthesis Enhances Design

AI Tools·Tom Levy·

AI: User Feedback Synthesis Enhances Design

AI: User Feedback Synthesis Enhances Design
Key Takeaways
1The evaluated interfaces meet 77.9% of the criteria with a summary and recommendations, compared to 69.5% with raw comments.
2An AI-assisted researcher retained the final choice of findings and recommendations submitted to the model.
3Reproducing the results with other models remains to be explored.
Why it matters : In this experiment, preparing the research context improves the scores of the generated interfaces compared to merely providing raw feedback.

A research team measures an advantage of 8.4 percentage points for interfaces generated by an AI model from a synthesis accompanied by recommendations, rather than from raw user comments. The comparison focuses on an airline form and a payment process.

Researchers are still exploring several limitations

The specific effect of each component of the context remains to be clarified: the team seeks to distinguish what comes from the aggregation of observations and what stems from explicit design recommendations. They are also examining whether the improvement extends beyond the issues directly targeted by the recommendations, affecting other aspects of the interfaces.

The scope of the results beyond short customer comments also remains to be explored, whether in terms of the nature of the data or its volume. Reproducing these results with other models is part of the open questions. Research is also ongoing regarding the influence of the transmitted format, including Markdown, PNG, or PDF, in a program dedicated to the user experience context provided to the AI.

A researcher selects the recommendations to transmit

In the experiment, each task has about 100 comments. The syntheses are prepared from these same feedbacks: a researcher uses an AI tool to analyze them but retains the final decision on the findings and recommendations transmitted to the model.

The team offers an explanation for the observed advantage: an observation does not necessarily prescribe a response. Noting the disappearance of items from a cart describes a difficulty without indicating which interface to build. A synthesis provides an interpretation; with only raw data, the model must prioritize the information and choose the modifications itself. The researchers believe these trade-offs are difficult without human guidance, as customer feedback is just one piece of data among others.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

As AI-generated prototypes enter daily practices, tools can already chain together feedback analysis, recommendations, and interface generation. However, their ability to absorb large amounts of data does not guarantee relevant selection or correct application in design, the team warns. They therefore recommend creating resources for the tools, in addition to those prepared for colleagues.

The design of UX context specifically refers to the selection, organization, and maintenance of user experience knowledge that guides these generations. For the researchers, identifying the important problems for a product and then translating them into recommendations remains an essential contribution from the relevant professionals, whether the execution is human or automated.

Scores improve on evaluated tasks

Interfaces fed by syntheses accompanied by recommendations meet an average of 77.9% of evaluation criteria, compared to 69.5% with raw feedback. The gap is statistically significant, with p < 0.001. The improvement concerns both exercises but reaches 11.4 percentage points for the flight task, compared to 5.5 for the payment task.

In the first exercise, a one-page form must be created to reschedule a canceled flight; in the second, a multi-step payment process for an online store. Only the research context changes for each exercise, while the instructions and elements of the design system remain unchanged. The complete protocol is more complex than this comparison between two contexts.

To account for the non-deterministic nature of the tools, 8 interfaces are produced for each task and each condition. A human evaluator and a language model assess them blindly according to predefined criteria. The score measures the proportion of criteria met, for example, the presence of purchased items at each step of the payment process.

The figures retained are those from the human evaluation. The language model performs an independent scoring used as a control. After calibration, the decisions of the evaluators agree in about 96% of cases across both exercises.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.