ADeLe: Revolutionizing AI Performance Evaluation

ADeLe: A New Approach to Evaluating AI

Traditional benchmarks in the field of artificial intelligence often focus on the performance of models on specific tasks, but they provide little information about the underlying capabilities of these models. This is where ADeLe comes in, a tool that evaluates AI models by assigning scores not only to tasks but also to the models themselves, based on 18 core capabilities. This approach allows for a direct comparison between task requirements and model capabilities, thus offering a more comprehensive view of their performance.

By using these capability scores, ADeLe can predict model performance on new tasks with an impressive accuracy of around 88%. This includes advanced models such as GPT-4o and Llama-3.1. By building capability profiles, ADeLe identifies areas where models are likely to succeed or fail, highlighting their strengths and limitations across different tasks.

Evaluating Models with ADeLe

ADeLe assigns scores to tasks based on 18 core capabilities, such as attention, reasoning, and domain knowledge. Each task receives a value from 0 to 5 according to its specific requirements. For example, a simple arithmetic problem might receive a low score in quantitative reasoning, while an Olympiad-level math proof would receive a much higher score.

Evaluating a model across numerous tasks produces a capability profile that offers a structured view of the model's performance and its breaking points. By comparing this profile to the requirements of a new task, it becomes possible to identify specific gaps that could lead to model failure.

By applying this framework to 15 LLMs, the team constructed capability profiles using scores from 0 to 5 for each of the 18 capabilities. For each capability, the team measured how performance evolves with task difficulty and used the difficulty level at which the model has a 50% chance of success as the capability score.

In-Depth Analysis with ADeLe

The team behind ADeLe used this tool to evaluate a wide range of AI benchmarks and model behaviors to better understand what current evaluations capture and what they miss. The results show that many widely used benchmarks provide an incomplete and sometimes misleading picture of model capabilities. A more structured approach, like the one proposed by ADeLe, can clarify these gaps and help predict how models will perform in new contexts.

ADeLe highlights that many benchmarks fail to clearly distinguish the capabilities they are supposed to measure or only cover a limited range of difficulty levels. For example, a test designed to evaluate logical reasoning may also heavily depend on specialized knowledge or metacognition. Other tests focus on a narrow range of difficulty, omitting both simpler and more complex cases. By assigning scores to tasks based on the required capabilities, ADeLe makes these inconsistencies visible and provides a means to diagnose existing benchmarks and design better tools.

Accurate Performance Prediction

ADeLe does not just evaluate; it also enables predictions. By comparing a model's capability profile to the requirements of a task, it can forecast whether the model will succeed, even on tasks it has not encountered before. In experiments, this approach achieved an accuracy of around 88% for models like GPT-4o and LLaMA-3.1-405B, surpassing traditional methods. This allows for explaining and anticipating potential failures before deployment, thereby improving the reliability and predictability of AI model evaluations.

Towards a Standardized Future with ADeLe

ADeLe is designed to evolve with advancements in AI and can be extended to multimodal and embodied AI systems. It also has the potential to serve as a standardized framework for AI research, policy development, and security auditing.

More broadly, ADeLe promotes a more systematic approach to AI evaluation—an approach that explains the behavior of systems and predicts performance. This work builds on previous efforts, including Microsoft’s research on applying psychometrics to AI evaluation and recent work on societal AI, underscoring the importance of AI assessment.