Instruction Hierarchy in LLMs

• Instruction Hierarchy (IH) is a formalized ordering of LLM directives that ensures higher-level instructions, such as system messages, override lower-level inputs.
• It underpins robust safety, effective tool use, and conflict resolution, with benchmarks like IHEval and ManyIH-Bench quantifying compliance performance.
• Recent architectural and training interventions, including ISE, AIR, and GW-DPO, demonstrate measurable improvements in model robustness and mitigation of prompt injection risks.

An instruction hierarchy (IH) is a formalized, explicit priority ordering of directives issued to LLMs, aimed at ensuring that higher-privilege instructions (such as system or developer-level messages) are always obeyed in the presence of lower-priority—typically user, history, or tool—directives. The motivation for IH is foundational to LLM reliability: it underpins robust safety enforcement, effective agentic tool use, and resilience against prompt injection and behavioral subversion. Decades of classical security models and recent empirical studies in LLM alignment converge on the conclusion that consistent prioritization among instruction sources is both necessary and deeply challenging in high-capacity neural architectures.

1. Formalization and Principles of Instruction Hierarchy

Instruction hierarchy is defined as an explicit partial or total ordering over the set of all instructions in the model's context, based on the instruction's provenance or intended privilege. In canonical modern LLM pipelines, the four most common levels are system message (highest), user message, conversation history, and tool outputs (lowest), though finer granularity (e.g., developer, per-user, retrieved data) is appearing in advanced agents (Zhang et al., 12 Feb 2025, Zhang et al., 10 Apr 2026). The fundamental enforcement rule is as follows: when two instructions conflict, the model's output must satisfy the highest-priority instruction, ignoring or soft-failing all subordinate, incompatible directives.

Notations formalize this as a privilege function $\pi: \{I_i\} \to V$ (with $V$ totally or partially ordered), and operationally as an inference-time comparator $$F(i, j) = \operatorname{arg\,max}_{k \in \{i, j\}} \pi(k)$$. In the presence of $N$ instructions and $M$ conflict groups (subsets of mutually exclusive constraints), the model is expected to resolve each group in favor of the maximally privileged member, subject to additional tie-breaking rules (Zhang et al., 10 Apr 2026).

2. Empirical Evaluation and Benchmarking of IH Compliance

Systematic evaluation of IH compliance is nontrivial. A canonical framework embeds pairs (or larger groups) of mutually exclusive, programmatically verifiable constraints into model prompts, rotating the designation of which constraint is "primary." By measuring rates of primary-only, secondary-only, and non-compliant responses, one defines:

• Primary Obedience Rate $R_1$
• Secondary Obedience Rate $R_2$
• Non-compliance Rate $R_3$
• Priority Adherence Ratio $\mathrm{PAR} = \frac{R_1}{R_1 + R_2}$
• Constraint Bias $\mathrm{CB}$, measuring inherent model preference in no-priority settings

Key benchmarks such as IHEval (Zhang et al., 12 Feb 2025, Geng et al., 21 Feb 2025) and ManyIH-Bench (Zhang et al., 10 Apr 2026) cover thousands of test cases across instruction-following, rule adherence, tool use, adversarial conflict, and safety domains, stratified by levels and types of conflict. IHEval, for example, measures both "aligned" (no conflict) and "conflict" resolution accuracy across a four-level IH, and empirically demonstrates that even competitive open-source models achieve at most 48% accuracy in resolving conflicts, with most performant API models dropping 20-40 points when moving from aligned to conflict evaluation regimes (Zhang et al., 12 Feb 2025).

3. Failure Modes and Inherent Model Biases

Empirical results consistently show that LLMs, including frontier models, do not adequately enforce explicit instruction hierarchy under conflict. When instructions are split across privilege levels, primary obedience rates ($V$0) collapse relative to single-instruction baselines (e.g., GPT4o: 91% on single-constraint vs. 40–64% under hierarchy; open-source models fare much worse) (Geng et al., 21 Feb 2025). Models display strong, context-insensitive constraint biases: preference for lowercase over uppercase, longer outputs, inclusion/exclusion of user-specified keywords, and strong language or brevity biases often override even explicit system directives. These tendencies persist across prompt structure, context enrichment, and prompt separator schemes.

Moreover, architectural scale does not reliably correlate with improved IH performance: in some cases, smaller or less capable models outperform larger ones on IH metrics, attesting to the entrenchment of non-hierarchical, token-level statistical heuristics (Geng et al., 21 Feb 2025, Zhang et al., 12 Feb 2025). Prompt engineering (explicit constraint marking or severe language) and light fine-tuning yield only partial, brittle gains and do not generalize across conflict types (Geng et al., 21 Feb 2025).

4. Architectural and Training Interventions for Instruction Hierarchy

Research has converged on several architectural and training recipes to strengthen IH adherence in LLMs:

• Instructional Segment Embedding (ISE): Inspired by segment-type embeddings in BERT, each token is augmented with an embedding denoting its origin (system, user, tool, etc.), thus preserving privilege distinctions throughout the transformer stack. Fine-tuning with ISE boosts prompt injection robustness and general safety by up to 18.68 percentage points, with 4.1% improvement on standard instruction-following tasks (Wu et al., 2024). Combining ISE with explicit delimiters further enhances performance.
• Augmented Intermediate Representations (AIR): Extends ISE by injecting privilege-embedding signals at each transformer layer, mitigating the vanishing of the IH signal due to deep model stacking. Against gradient-based prompt-injection, AIR achieves up to 9.2× reduction in attack success rates over ISE or delimiter-based strategies, while maintaining instruction-following utility (Kariyappa et al., 25 May 2025).
• Preference Optimization (DPO, GW-DPO): Direct Preference Optimization (DPO) formalizes the preference for hierarchy-compliant responses over violations. Gravity-weighted DPO (GW-DPO) differentially scales the loss margin according to the privilege gap between conflicting instructions. Bilateral scheduling (weighting both privilege gap and victim level) yields the highest macro pairwise priority adherence (0.838) while controlling over-refusal, with ISE acting as a crucial refusal-threshold calibrator (Bolliger et al., 9 Jun 2026).
• Constraint-Based Policy Optimization (HIPO): Formulates the IH problem as a Constrained Markov Decision Process (CMDP), with system prompt compliance as a hard constraint and user utility dynamically maximized within the feasible set. This primal–dual RL achieves simultaneous Pareto improvement in system compliance and user utility, mechanistically shifting decoder attention to long-range, high-authority tokens (Chen et al., 17 Mar 2026).
• Symbolic-Neuro Reasoning (NSHA): NSHA enforces IH by parsing contexts into atomic, authority-labeled instructions, detecting contradictions, and using MaxSMT solvers to maximally satisfy high-authority constraints; these selections are then distilled into the LLM via a composite preference and semantic loss. This approach recovers up to 70.7% accuracy on safety-conflict tasks (e.g., language detection under conflicting directives) (Yang et al., 10 Apr 2026).

5. Data-Centric and Evaluation Advances

Dedicated IH-centric datasets and evaluation frameworks play a central role in both diagnosing deficiencies and catalyzing algorithmic progress:

• IH-Challenge: Programmatically constructs multi-role, multi-constraint adversarial and benign contexts, accompanied by Pythonic auto-graders for objective compliance measurement. Fine-tuning LLMs on IH-Challenge data robustly improves IH adherence by 10–12 percentage points on both in-distribution and OOD conflict tasks, simultaneously lowering unsafe behavior to under 1% (Guo et al., 11 Mar 2026). The anti-overrefusal split is critical to prevent models from over-generalizing refusals, a common shortcut in naive preference optimization.
• ManyIH-Bench: Targets the Many-Tier IH setting (up to 12 privilege levels, reflecting heterogeneous agentic use cases). Empirical findings indicate frontier LLMs struggle (≤42.7% accuracy even for closed-source leaders) and that model decisions are highly sensitive to privilege encoding and the number of tiers (Zhang et al., 10 Apr 2026).
• Instruction Data Construction: Hierarchical labeling systems (e.g., InfinityInstruct-Subject) assign explicit fine-grained and domain-level tags to instructions, allowing systematic stratification and diagnosis of model strengths/deficiencies. Closed-loop pipelines integrate these tags into seed selection, data evolution, and model diagnosis, driving monotonic increases in coverage, depth, and downstream evaluation improvement (Du et al., 9 Jul 2025).

6. Open Challenges and Prospective Directions

While certain interventions—ISE, AIR, GW-DPO, NSHA, HIPO—close significant fractions of the IH compliance gap, complete, robust, and generalizable IH enforcement remains unresolved. Open challenges include:

• Brittleness in Multi-Tier and Multi-Source Settings: Models exhibit declining accuracy and instability as the number of privilege levels increases or as the surface realization of privilege values changes (Zhang et al., 10 Apr 2026).
• Architectural Scalability and Inference Efficiency: Embedding-based interventions add minimal parameter overhead but deep privilege signaling (e.g., AIR) requires careful optimization to avoid utility losses.
• Expectation vs. Deterministic Guarantees: Current reinforcement learning formulations (e.g., HIPO) guarantee system compliance in expectation, not per-instance. Real-time, deterministic constraint satisfaction or proactive rejection is an open area.
• Security Risks at the System Level: Strict IH can create new vulnerabilities if attackers gain control over high-privilege (e.g., system) instruction segments; access control and auditing must supplement model-external defense (Chen et al., 17 Mar 2026).
• Out-of-Distribution Generalization: Adaptive and rare adversarial conflict patterns are incompletely defeated by current methods, motivating research on adversarial co-training, symbolic-neural hybrids, and learning-based proxy reward models (Guo et al., 11 Mar 2026, Yang et al., 10 Apr 2026).

Proposed research frontiers encompass architecture-level supports for hundreds of privilege levels, symbolic-neural constraint solvers, reinforcement learning curricula covering the full space of privilege relations, dynamic privilege assignment at inference time, and hybrid approaches integrating structural ordering invariance (Zhang et al., 10 Apr 2026, Bolliger et al., 9 Jun 2026, Yang et al., 10 Apr 2026).

7. Summary Table: IH Mechanisms and Empirical Payoff

IH Intervention	Core Technique	Quantitative Improvement
ISE (Wu et al., 2024)	Input segment embedding	+15.75~18.68 pp robust acc; utility +4.1 pp
AIR (Kariyappa et al., 25 May 2025)	Layerwise privilege embeddings	1.6–9.2× ASR reduction/gr. attacks
GW-DPO (Bolliger et al., 9 Jun 2026)	Gravity-weighted preference loss	Macro PPA 0.838, WHS 0.885; ORR 0.057
HIPO (Chen et al., 17 Mar 2026)	CMDP safe RL	r_sys/r_user: 0.70/0.47+; system utility up
IH-Challenge (Guo et al., 11 Mar 2026)	Adversarial training	+10–12 pp IH robustness, unsafe ↓6.6→0.7%
NSHA (Yang et al., 10 Apr 2026)	MaxSMT solver-distillation	Conflict rec.: up to 70.7% from ~4.5% base
ManyIH-Bench (Zhang et al., 10 Apr 2026)	Multi-tier IH, agentic eval	Models ≤42.7% at scale, urgent research need

Robust instruction hierarchy is a distinctive, safety-critical, and still-incomplete dimension of LLM controllability. Progress now relies on mechanistically grounded architecture modifications, principled training objectives, adversarial evaluation, and theoretically sound symbolic reasoning integration—all subject to ongoing empirical scrutiny in multi-level, multi-source, adversarial, and benign real-world settings.