AgentRx: Breakthrough in Debugging Complex AI Agents

The Challenge of Debugging AI Agents

Debugging artificial intelligence (AI) agents proves to be a daunting task. Indeed, the trajectories of these agents are often long, stochastic, and involve multiple agents, complicating the identification of the root cause of failures. Modern AI agents are characterized by actions over extended periods, a probabilistic nature where the same input can produce different outputs, and multi-agent interactions where failures can propagate, thus masking the initial cause.

Introducing AgentRx: An Innovative Solution

To overcome these challenges, the AgentRx framework has been introduced. This system focuses on identifying the first step of irreversible failure, termed "critical failure." AgentRx synthesizes executable and protected constraints from tool schemas and domain policies, then records violations with step-by-step evidence. Rather than relying on a large language model (LLM) to guess the error, AgentRx employs a structured multi-step pipeline.

A Benchmark and Taxonomy for Failures

To assess the effectiveness of AgentRx, a benchmark named AgentRx Benchmark has been published. It includes 115 manually annotated failed trajectories, covering three complex domains: τ-bench, Flash, and Magentic-One. Additionally, a nine-category failure taxonomy has been developed to help classify the types of errors encountered, ranging from "Plan Adherence Failure" to "Invention of New Information."

Significant Improvements with AgentRx

The results obtained with AgentRx are impressive. The framework improves failure localization by 23.6% and root cause attribution by 22.9% compared to prompt-based methods. By providing an auditable log of failures, AgentRx enables developers to better understand and correct errors in AI agents, surpassing traditional trial-and-error debugging.

The Automated Diagnostic Process

AgentRx treats the execution of agents as a system trace requiring validation. The framework follows a structured process that includes several steps:

• Trajectory Normalization: Heterogeneous logs from different domains are converted into a common intermediate representation.
• Constraint Synthesis: The framework automatically generates executable constraints based on tool schemas and domain policies.
• Protected Evaluation: AgentRx evaluates the constraints step by step, checking each constraint only when its guard condition applies, and produces an auditable validation log of violations supported by evidence.
• LLM-Based Judgment: Finally, an LLM judge uses the validation log and an anchored failure taxonomy to identify the critical failure step — the first irreversible error.

A New Benchmark for Agent Failures

The benchmark developed for AgentRx includes failed trajectories across three complex domains: τ-bench, which pertains to structured API workflows for retail and service tasks; Flash, which focuses on real incident management and system troubleshooting; and Magentic-One, which covers web tasks and open files using a generalist multi-agent system. A grounded theory approach was used to derive a nine-category failure taxonomy, helping developers distinguish between types of errors such as "Plan Adherence Failure" and "Invention of New Information."

Analyzing Failure Density Across Domains

In multi-agent systems like Magentic-One, trajectories often contain multiple errors, but AgentRx focuses on identifying the first critical violation. In our experiments, AgentRx demonstrated significant improvements over LLM-based prompt benchmarks: a 23.6% improvement in failure localization accuracy and a 22.9% improvement in root cause attribution.

Joining the Community: Open Source Publication

To encourage ongoing adoption and improvement, the AgentRx framework and the complete annotated benchmark are published as open source. This initiative aims to enhance the reliability of AI agents, a crucial element for their use in the real world.

Contributions and Acknowledgments

The project has benefited from the contributions of Avaljot Singh and Suman Nath, whose work has been essential to its development.