Hugging Face: Solaiman Warns About the Limits of Built-in Evaluators

Irene Solaiman, Head of Policy at Hugging Face, warns against over-reliance on current benchmarks and built-in evaluators. With fragile reproducibility, uncertain comparisons, and a lack of standards, evaluation now extends to the entire system—including harnesses and agents—resulting in increased costs and blind spots. A more uniform approach may be necessary for benchmarking to remain relevant.

Testing the System, Not Just the Model: Increasing Complexity and Costs

Evaluations are no longer limited to the model alone but extend to the entire system, including the harness and the agent. Irene Solaiman emphasizes that the harness has a decisive influence on performance: a less capable model can outperform a state-of-the-art model if it benefits from a better harness. She believes that agent evaluations are particularly challenging and costly. Today, there are more evaluations for models than for harnesses, and better methods for assessing the holistic agent are under development. The evaluation field remains poorly structured, and a more uniform industry-wide approach may be necessary for benchmarking to retain its usefulness.

Built-in Evaluators: High Demand, Questionable Neutrality

The idea of built-in evaluators, consisting of supposedly neutral third-party observers placed within AI organizations, has gained popularity in recent months. This trend follows a series of security incidents in AI labs that have reignited concerns about the potential dangers of these technologies; Hugging Face was indeed involved in one of these incidents. Rather than resorting to active government regulation to identify and mitigate risks, several labs have advocated for third-party solutions. Nevertheless, the question of impartiality and the actual performance of these evaluators continues to be debated. Anthropic's choice to select Accenture as its first evaluator has drawn criticism, with some observers arguing that Accenture lacks the necessary expertise for this role.

Reproducibility and Comparability: The Achilles' Heels of Benchmarks

Irene Solaiman believes that for tests to be reliable in the future, they must become a more coherent science, as there is currently no meaningful method to compare evaluations with one another. Trust in results primarily depends on their reproducibility, which requires sufficient access to the model; the openness of weights plays a significant role here. Reproducibility remains one of the most challenging and debated aspects of benchmarking. According to Solaiman, some evaluations quickly become ineffective, and the ecosystem is not as developed as necessary, despite strong demand for built-in evaluators. The qualifications required to effectively evaluate according to tasks need further standardization. She finally reminds us that evaluation methods remain complex and in maturation, even as many AI labs base their reputation on benchmarks presented as neutral and accurate.