OpenAI Revolutionizes Instruction Hierarchy with IH-Challenge

OpenAI recently introduced an innovative training dataset named "IH-Challenge." This dataset utilizes reinforcement learning to teach artificial intelligence models to establish a clear hierarchy of instructions. This hierarchy starts with system security policies, followed by developer instructions, then user instructions, and finally guidelines from external tools.

The GPT-5 Mini-R model, which has been trained with IH-Challenge, demonstrates an enhanced ability to correctly prioritize instructions. This model is particularly effective at defending against prompt injection attacks, which involve concealing malicious instructions within the outputs of tools. OpenAI considers this capability essential for agentic models, which must be able to independently call tools and process external documents. To encourage research and development in this area, OpenAI has made the IH-Challenge dataset available on the Hugging Face platform.

The development of IH-Challenge addresses a fundamental problem faced by AI systems: the difficulty in discerning which instructions to follow when multiple sources of instructions conflict. Security policies, developer parameters, user requests, and information from external tools can often be at odds. When a model follows an incorrect instruction, it can lead to bypassing security policies and enable prompt injection attacks. OpenAI designed IH-Challenge to teach models to reliably prioritize trustworthy instructions over those that are not.

IH-Challenge surpasses previous approaches, such as those based on GPT-3.5 Turbo, which only supported three levels of priority and relied on LLM judges for evaluation. The new dataset adds a fourth level of hierarchy for developers and replaces error-prone evaluations with simple Python scripts for automated verification.

In the accompanying document, OpenAI identifies three major pitfalls of current training methods. First, errors in following complex instructions can be mistakenly reported as hierarchy failures. Second, instruction conflicts are often subjective, making automated evaluation challenging. Finally, models tend to learn shortcuts, such as rejecting harmless requests just to be cautious. IH-Challenge addresses these issues with deliberately simple tasks that can be automatically evaluated by scripts and do not allow trivial shortcuts.

According to OpenAI, the internal GPT-5 Mini-R model trained on IH-Challenge shows clear improvements in academic and internal benchmarks regarding the correct prioritization of instructions. The greatest gains have been observed in conflicts between developer-level instructions and user-level instructions. At the same time, the model's overall capabilities have remained largely intact.

The enhanced instruction hierarchy translates into two concrete benefits, according to OpenAI. First, the model follows security policies in the system prompt more reliably without becoming less useful overall. Second, robustness against prompt injection attacks significantly improves, particularly those that conceal malicious instructions in tool outputs. OpenAI had previously documented similar vulnerabilities in ChatGPT Atlas.

OpenAI emphasizes that this capability will become a critical security feature as models become more agentic. Models that independently call tools and read unreliable documents must be able to reliably distinguish between legitimate and manipulative instructions. By publishing IH-Challenge on Hugging Face, OpenAI hopes to stimulate further research that will further enhance the security of AI systems.