Instruction Brittleness in Machine Learning

• Instruction brittleness is the sensitivity of machine learning models to minor prompt variations, leading to inconsistent outputs and misaligned rewards.
• It manifests across LLMs and RL agents in diverse scenarios, severely affecting evaluation metrics, fairness, and system reliability in high-stakes applications.
• Dynamic prompting, ensemble evaluations, and refined reward designs are practical strategies to mitigate brittleness and enhance model generalization.

Instruction brittleness denotes the susceptibility of machine learning agents—particularly LLMs and reinforcement learning (RL) agents—to substantial output or performance variation in response to minor changes in instruction phrasing or natural-language prompts. It manifests as output instability, misaligned reward attribution, diminished generalization, and unpredictable fairness under small, semantically equivalent perturbations, posing significant challenges in both high-stakes (e.g., clinical) and research settings. Instruction brittleness is observed across a range of architectures, training protocols, and real-world tasks, with implications for evaluation, model tuning, and deployment safety.

1. Definition and Conceptual Foundations

Instruction brittleness is characterized by high sensitivity of model output or agent behavior to superficial or non-adversarial modifications of task instructions. Within RL, Huang et al. (Huang et al., 2023) describe a scenario where agents trained to follow natural-language instructions are misled by scalar reward functions $R(s,a;I)$ that relax strict task constraints, rewarding not only fully-aligned but also partially-aligned trajectories. As a result, agents cannot reliably distinguish fully correct from partially correct solutions, leading to slow or failed learning, frequent exploitation of loopholes, and increased data requirements for learning an $\epsilon$-optimal policy.

In LLMs, instruction brittleness is measured empirically as a performance gap (e.g., accuracy, AUROC, F₁) between outputs produced under original vs. paraphrased or reworded prompts for semantically identical tasks (Carranza, 8 Oct 2025, Arroyo et al., 2024). Li et al. (Li et al., 9 Sep 2025) extend the definition, showing that both LLMs and humans exhibit sensitivity to label set perturbations, but LLMs are uniquely fragile to surface-form changes (e.g., typos, label order).

2. Formalizations, Metrics, and Experimental Protocols

Instruction brittleness is operationalized using range and variance metrics over output scores, as well as divergence-based measures:

• Performance Range and Variance: For prompt set $\{I_1, ..., I_N\}$, model $M$, and task $T$, define range $\Delta_T(M) = \max_i R_i - \min_i R_i$, and variance $$\text{Var}(R) = \frac{1}{N-1}\sum_{i=1}^N (R_i - \bar{R})^2$$, where $R_i$ is model performance under $I_i$ (Arroyo et al., 2024).
• Distributional Divergence: Jensen–Shannon divergence (JSD) is used to quantify shifts in output distributions when comparing base and perturbed instructions for both LLMs and humans (Li et al., 9 Sep 2025).
• Accuracy Gap: $$\Delta_{\text{acc}} = \text{Acc}_{\text{orig}} - \text{Acc}_{\text{para}}$$ across original and paraphrased items (Carranza, 8 Oct 2025).
• RL Convergence Signal: In actor–critic setups, brittleness is revealed by a diminished signal-to-noise ratio and bias in policy gradient updates due to partial-match rewards (Huang et al., 2023).

Experimental protocols include stress-testing on paraphrased benchmark items (Carranza, 8 Oct 2025), multi-prompt evaluation on clinical NLP tasks (Arroyo et al., 2024), and direct comparison between LLMs and human annotators under prompt variation (Li et al., 9 Sep 2025).

3. Origins and Mechanisms

Instruction brittleness is rooted in several architectural, algorithmic, and data-driven mechanisms:

• Reward Perturbation (RL): Scalar reward compression in LRS leads to positive feedback for suboptimal behaviors, especially when similarity-based or binary classifier heads allow partial matches to exceed reward thresholds (Huang et al., 2023). The loosening of action, state, and temporal constraints (e.g., $$\Pi_L \supsetneq \Pi_{C_a}\cap\Pi_{C_s}\cap\Pi_{C_t}$$) expands the set of policies receiving nonzero reward, poisoning the gradient and slowing convergence.
• Surface-Form Sensitivity (LLMs): Minor edits in instructional language, such as synonym swaps, label order rearrangements, or paraphrasing, precipitate significant accuracy drops, as models latch onto n-gram matches, formatting cues, or learned shortcuts (Carranza, 8 Oct 2025, Li et al., 9 Sep 2025). This is exacerbated when training data lacks sufficient variance in instruction phrasing.
• Overfitting in Domain Models: Instruction-brittleness is pronounced in domain-specific LLMs (e.g., clinical models) trained on synthetic or literature-derived text, which overfit to a narrow phrase distribution and fail to generalize to diverse real-world prompt styles (Arroyo et al., 2024).
• Static Prompt Limitations in Symbolic Tasks: Fixed prompt strategies in sequence-generation or equation discovery tasks (e.g., PDE identification) cause models to plateau at suboptimal outputs contingent on initial phrasing (Qu et al., 31 Dec 2025).

4. Empirical Manifestations and Quantitative Findings

Empirical studies provide strong evidence for the prevalence and impact of instruction brittleness:

• RL agents with LRS: In Montezuma’s Revenge, PPO+RND agents with reward shaping for partial matches (simulated rule 2: $\alpha=0.5$) saw area under the curve (AUC) plummet to 0.05–0.18, compared to pure RL baselines achieving 0.55–0.82 and 100% success. Only strictly matched reward rules recover performance (Huang et al., 2023).
• LLMs under Paraphrase Stress: Accuracy declines by 6–10 percentage points between original and paraphrased question stems in ARC-Easy/-Challenge. The largest brittleness observed is $\Delta_{\text{acc}} = 0.10$ (Carranza, 8 Oct 2025).
• Clinical NLP Prompts: Performance ranges $\Delta$ of up to 0.60 AUROC and 0.40 F₁ are found between equivalent clinical prompts, with notable fairness disparities (race $\Delta$ up to 0.35 AUROC; sex $\Delta$ 0.19) induced solely by prompt choice (Arroyo et al., 2024).
• Human vs LLM Sensitivity: Humans show mean JSD $\approx$ 0.33 across instructions, LLMs $\approx$ 0.22. Both are most sensitive to label set changes, but LLMs uniquely respond to typographical and order perturbations (Li et al., 9 Sep 2025).
• Equation Discovery: NeuroSym-BO, a dynamic prompt selection framework, surpasses fixed prompts in test $R^2$ by 5–11% and consistently identifies sparser, more accurate PDEs, validating that adaptive instruction tuning mitigates brittle plateaus (Qu et al., 31 Dec 2025).

5. Implications for Evaluation, Generalization, and Fairness

Instruction brittleness has profound implications for benchmarking, deployment, and risk assessment:

• Evaluation Contamination: Benchmark scores may overstate generalization if high performance results from memorization or exploitation of surface-form shortcuts; paraphrase-aware evaluation is required (Carranza, 8 Oct 2025).
• Deployment Uncertainty: In clinical settings, brittleness can induce unacceptably high output and fairness variability for small, non-adversarial instruction edits, threatening real-world reliability (Arroyo et al., 2024).
• Human Variability as Reference: Certain forms of brittleness (semantic label changes) mirror genuine human annotation variance, but LLM hypersensitivity to minor surface cues (e.g., typos, punctuation) is excessive and does not correspond to legitimate human cognitive behavior (Li et al., 9 Sep 2025).
• Dynamic Prompting as Remedy: Adaptive, closed-loop selection of instructional strategies (e.g., via Bayesian Optimization over a library) both improves generalization and attenuates brittle behavior (Qu et al., 31 Dec 2025).

6. Mitigation Strategies and Best Practices

Multiple mitigation approaches have been proposed:

• Multi-label and Vectorized Reward Construction: In RL, explicitly scoring action, state, and temporal satisfaction, rather than collapsing constraints to a scalar, reduces false positives (Huang et al., 2023).
• Trajectory-Level Preference Learning: Transitioning from step-wise shaping to upvoting matched trajectories or offline optimization avoids error accumulation due to reward loosening (Huang et al., 2023).
• Prompt Auditing and Ensemble: Evaluating models on sets of plausible prompt variants, reporting performance ranges, and aggregating outputs over prompt ensembles enhances robustness (Arroyo et al., 2024).
• Surface-Form Data Augmentation: Incorporating meaning-preserving paraphrases during instruction-tuning, or regularizing outputs against syntactic variations, builds surface invariance (Carranza, 8 Oct 2025).
• Soft Prompt Embeddings: Optimizing embeddings to collapse representations of equivalent instructions mitigates embedding-level variance (Arroyo et al., 2024).
• User Education and Template Validation: Informing end-users (e.g., clinicians) about prompt sensitivity; alerting when prompts deviate from validated forms (Arroyo et al., 2024).
• Consistency in Label Framing: Fixing label sets and numeric scales avoids semantic drift in classification tasks (Li et al., 9 Sep 2025).

7. Ongoing Challenges and Research Directions

Priority areas for future work include:

• Scaling dynamic instruction frameworks to higher dimensions and more complex domains (e.g., chaotic, stochastic systems in symbolic discovery) (Qu et al., 31 Dec 2025).
• Developing paraphrase-adversarial fine-tuning protocols to directly minimize output brittleness across diverse linguistic forms (Carranza, 8 Oct 2025).
• Joint human–LLM sensitivity profiling to isolate legitimate from spurious brittleness, informing robust evaluation standards (Li et al., 9 Sep 2025).
• Fairness stratification in instruction-tuned models to prevent demographic disparities caused by prompt artifacts (Arroyo et al., 2024).
• Decoding strategies and self-consistency studies exploring the impact of temperature, top-$k$ sampling, and other generation hyperparameters on brittleness (Li et al., 9 Sep 2025).

Instruction brittleness remains a central obstacle in the quest for reliable, generalizable instruction-following agents and models. Its mitigation demands careful attention to reward design, evaluation methodologies, prompt diversity, and user practice, as well as ongoing research into dynamic adaptation and fairness.