ProactiveBench: AIs Fail to Ask for Help

ProactiveBench: An Unprecedented Test for Multimodal AIs

ProactiveBench is an innovative benchmark that evaluates whether multimodal language models can ask for help when visual information is lacking. Among the 22 models tested, very few demonstrated this capability, often preferring to guess or refuse to respond. However, a reinforcement learning approach could offer a solution.

When a person cannot see an object, they naturally ask for the obstacle to be removed. In contrast, AI models often hallucinate an incorrect answer or simply refuse to respond. ProactiveBench aims to test this issue by assessing whether current models can recognize their need for help and effectively request it.

ProactiveBench Testing Scenarios

The benchmark utilizes seven datasets transformed into scenarios requiring human intervention. For instance, models must identify hidden objects, clean noisy images, or interpret rough sketches. ProactiveBench contains over 108,000 images spread across 18,000 samples. An integrated filter eliminates any task that models can succeed at without assistance, thereby forcing a proactive request for additional information.

The scenarios include occluded objects, uninformative viewpoints, noisy images, and camera movements. Proactive models ask for help, while reactive ones hallucinate or abandon the task.

Model Performance and Size

Researchers tested models such as LLaVA-OV, Qwen2.5-VL, and GPT-4.1. Under normal conditions, the models succeed in 79.8% of tasks, but this figure drops by over 60% on ProactiveBench. For example, accuracy on hidden objects falls from 98.3% to just 8.2%.

Model size is not always an advantage. InternVL3-1B outperforms InternVL3-8B with 27.1% versus 12.7% success. Similarly, LLaVA-1.5-7B beats LLaVA-OV-72B with 24.8% compared to 13%. Closed models like GPT-4.1 show the best performance, although high scores may be due to data contamination.

Proactivity or Simple Guessing?

Some models appear proactive but often choose absurd options when presented. For instance, LLaVA-NeXT Vicuna increased its selection rate from 37% to 49% with incorrect choices. This indicates that their apparent proactivity is often mere guessing rather than true understanding.

Researchers subjected these models to stress tests by replacing valid proactive suggestions with absurd options, such as "Rewind the video" for a sketch task. Models that previously seemed proactive readily chose these meaningless options, proving that their supposed proactivity was merely a lower threshold for guessing.

Reinforcement Learning as a Solution

Researchers demonstrated that proactivity can be trained. By using Group Relative Policy Optimization (GRPO), they refined LLaVA-NeXT-Mistral-7B and Qwen2.5-VL-3B, improving their performance to 37.4% and 38.6%, respectively. However, a poor reward balance can lead to a drop in accuracy to 5.4%.

After training, these models surpassed each of the 22 previously tested models, including o4-mini. The learned proactivity also transferred to scenarios outside the training data. On ChangeIt, the accuracy of Qwen2.5-VL-3B increased from 12.4% to 55.6%. But if the reward balance is poorly managed, everything collapses: when proactive suggestions are rewarded the same way as correct answers, the model spams help requests endlessly, and accuracy plummets to 5.4%.

Despite these advancements, a significant gap remains compared to ideal conditions (40.7% vs. 75.1%). ProactiveBench, now open source, is presented as a first step toward models capable of recognizing their limitations and asking for help instead of guessing.

AI Models Facing Uncertainty

ProactiveBench highlights a concerning trend in AI research: multimodal language models struggle with uncertainty. The WorldVQA benchmark from Moonshot AI recently revealed that even the most advanced models plateau around 50% in visual object recognition, underscoring an embedded overconfidence.

A Stanford study on the Mirage effect reinforced this point. Multimodal models like GPT-5 and Gemini 3 Pro confidently described visual details and offered medical diagnoses even when no image was provided. On standard benchmarks, they achieve 70 to 80% of their normal performance using only textual patterns and prior knowledge, essentially simulating visual understanding without realizing that the input was missing.

Other research tells a similar story. A study on exam question difficulty revealed that language models cannot reliably assess their own limits, while researchers from Sapienza University in Rome used their "Spilled Energy" method to show that hallucinations leave measurable traces in a model's calculations—suggesting that even when models do not know they are guessing, the underlying mathematics does.