IH-Challenge: Securing LLMs with a Hierarchy of Instructions

IH-Challenge: A New Dataset for Training Models

IH-Challenge is an innovative dataset designed to enhance the hierarchy of instructions in cutting-edge language models. This dataset aims to improve the safety and robustness of models against prompt injection attacks. Artificial intelligence systems receive instructions from various sources, such as security policies embedded in system messages, developer guidelines, user requests, and information available online. Training these models to identify and prioritize the most reliable instructions among these sources is crucial for ensuring secure deployment.

Security Issues Related to Instruction Hierarchy

When models fail to correctly prioritize instructions, security and reliability issues can arise. Models may be prompted to provide prohibited content, disclose private information, or fall victim to prompt injection attacks hidden within online data. The inability to respond appropriately in these situations often stems from a misinterpretation of instructions. In cases of conflicting instructions, the model must determine which ones take precedence. If an unreliable instruction is treated as authoritative, the model risks violating the policies or intentions of developers and users.

The Importance of a Clear Instruction Hierarchy

To manage instruction conflicts, models developed by OpenAI are trained to follow a well-defined instruction hierarchy: system > developer > user > tool. Higher-priority instructions are considered more reliable. The model should only follow lower-priority instructions if they do not contradict higher-priority ones. For example, if a system message contains a security policy and a user requests the model to violate it, the model must refuse to execute that request.

Challenges of Large-Scale Training

Reinforcement learning is an effective method for teaching instruction hierarchy. However, several challenges can arise. Instruction-following failures can also be hierarchy failures. Instruction conflicts can be complex and subjective. Additionally, models tend to learn shortcuts that maximize rewards but are not always practical in real-world scenarios.

Our Approach with IH-Challenge

To overcome these challenges, we designed IH-Challenge, a reinforcement learning dataset. We adhered to key principles: tasks must be simple to follow, objectively evaluable using a Python script, and should not include trivial shortcuts that guarantee high rewards. Each task in IH-Challenge consists of a conversation with high-priority and low-priority role instruction messages, testing the model's ability to adhere to high-priority instructions.

Model Results and Robustness

By training a model on IH-Challenge, we developed an internal model called GPT-5 Mini-R. This model has shown significant improvements in performance on instruction hierarchy benchmarks. It has also demonstrated better general performance during held-out and adversarial instruction hierarchy tests while maintaining its overall utility without falling into over-refusal.

Security and Reliability Benefits

A strengthened instruction hierarchy offers numerous security benefits, particularly regarding security steerability and robustness against prompt injections.

Security Steerability

We evaluate security steerability by adding specific security specifications to categories in the system prompt and measuring the model's behavior on OpenAI's security benchmarks. The model trained with IH-Challenge shows consistent improvement in this area.

Robustness Against Prompt Injections

The instruction hierarchy is also crucial for resisting prompt injections, where malicious instructions are embedded in tool outputs. The model trained with IH-Challenge enhances its robustness against these attacks across multiple benchmarks.

Future Perspectives

As models become more autonomous, capable of calling tools, reading unreliable documents, and taking actions in the real world, the ability to consistently prioritize reliable instructions over unreliable ones becomes a fundamental security property. Strengthening the instruction hierarchy not only improves reliability but also unlocks several security gains, an increasingly important foundation as AI systems gain capabilities and autonomy.