Superficial Alignment Hypothesis in LLMs

• Superficial Alignment Hypothesis is the idea that alignment techniques, like supervised fine-tuning with refusal responses, induce only surface-level behavioral changes without improving underlying reasoning or factuality.
• Empirical evidence shows that removing alignment examples improves performance across benchmarks such as MMLU, BBH, HumanEval, and DROP, indicating a measurable performance drop from superficial alignment.
• Mitigation strategies including dataset filtering, style-matched safety augmentation, and neuron-level interventions are proposed to counteract the detrimental effects of superficial alignment in LLMs.

The Superficial Alignment Hypothesis (SAH) posits that, in the context of LLMs, alignment procedures—especially those relying on supervised fine-tuning (SFT) datasets that incorporate refusals or safety-motivated responses—tend to impart only surface-level behavioral changes. Instead of imbuing models with genuinely improved reasoning, factuality, or deep safety understanding, these approaches teach models to mimic refusal patterns and other stylistic signals. Empirical evidence shows that this "surface" alignment can significantly degrade model performance across reasoning and code-generation benchmarks and leaves models vulnerable to adversarially structured or stylistically manipulated prompts. SAH underpins a growing body of research that explores both the limitations of existing alignment practices and methodologies for isolating, quantifying, and remedying superficial alignment effects (Bekbayev et al., 2023).

1. Formal Definitions and Theoretical Framing

SAH is defined as the hypothesis that alignment examples in SFT—especially those resembling content-safety refusals—induce LLMs to learn surface-level mappings (e.g., mapping many queries to refusal templates) while failing to improve or even degrading substantive task performance. Formally, for a reasoning benchmark $B$, the performance drop induced by alignment poison is

$$\Delta_B = \mathrm{Score}_{\mathrm{unaligned}(B)} - \mathrm{Score}_{\mathrm{aligned}(B)}$$

with relative degradation

$$\delta_B = 100 \times \frac{\Delta_B}{\mathrm{Score}_{\mathrm{unaligned}(B)}}$$

(Bekbayev et al., 2023).

At the algorithmic level, alignment by SFT or RLHF is conceptualized as analogous to data poisoning: an influx of uninformative, refusal-style examples dilutes the information in the training signal, akin to the effect of adversarial examples in classical supervised learning. This dilution does not create new capabilities but suppresses productive mappings between inputs and helpful outputs. Some formalizations model alignment as narrowing the model’s output support strictly to those assistant-style behaviors already present in the pretrained distribution (Lake et al., 2024), while others analyze it as a transformation that can be captured by low-rank or linear modifications to the model's output layer (Chen et al., 7 Feb 2025).

2. Experimental Methodologies and Metrics

A range of ablation and comparative experiments underpin SAH:

• Dataset construction: Many studies construct parallel aligned (with refusals/content-safety) and unaligned (refusals removed) SFT datasets. LLMs (frequently LLaMA-2/3 or Mistral) are fine-tuned on these, with all other variables held constant (Bekbayev et al., 2023).
• Evaluation benchmarks: Standard downstream assays include MMLU, BigBench Hard (BBH), HumanEval (code generation), DROP (reading comprehension), as well as safety-sensitive benchmarks like AdvBench (Bekbayev et al., 2023, Chen et al., 7 Feb 2025).
• Metrics: Performance drop $\Delta_B$ and relative degradation $\delta_B$, attack success rate (ASR) in jailbreak settings, KL divergence between aligned and superficially aligned head outputs, win-rate under pairwise LLM-judgment, and self-similarity scores for diversity (Bekbayev et al., 2023, Lake et al., 2024, Xiao et al., 9 Jun 2025).

Empirical strategies also include measuring models' attention to superficial style cues in prompts—quantified as

$\Delta A = A_s - A_i$

where $A_s$ is the model's self-attention to style tokens and $A_i$ to intent tokens. Statistically significant correlations between $\Delta A$ and increased ASR affirm the risk of superficial alignment via style (Xiao et al., 9 Jun 2025).

3. Core Empirical Evidence Across Tasks

Empirical results across multiple studies systematically demonstrate the risks and prevalence of superficial alignment:

• Task performance: On LLaMA 2 7B, removal of refusal/alignment examples yields improvements across MMLU (8.1% rel.), BBH (4.1%), HumanEval (33.3%), and DROP (24.3%) (Bekbayev et al., 2023).

| Benchmark | With Alignment | No Alignment | Absolute Δ_B | Relative δ_B | |-----------|---------------|--------------|--------------|--------------| | MMLU | 45.63 | 49.31 | 3.68 | 8.1% | | BBH | 34.28 | 35.69 | 1.41 | 4.1% | | HumanEval | 9.15 | 12.20 | 3.05 | 33.3% | | DROP | 22.61 | 28.10 | 5.49 | 24.3% |

• Style-based vulnerabilities: Fine-tuning on specific response styles leads to a sharp increase in ASR for jailbreak attacks when test queries match the fine-tuned style; a correlation of $\rho=0.571$ ($$\Delta_B = \mathrm{Score}_{\mathrm{unaligned}(B)} - \mathrm{Score}_{\mathrm{aligned}(B)}$$0) is observed between attention to style cues and vulnerability (Xiao et al., 9 Jun 2025).
• Superficial knowledge transfer: Linear modifications to the output head recapitulate over 50%—and up to 100%—of alignment performance on safety and style tasks, but fail to capture deeper reasoning ability (Chen et al., 7 Feb 2025).

4. Formal Operationalizations & Broader Generalization

Recent refinements cast SAH in broader algorithmic and information-theoretic terms:

• Task complexity: The minimal bit-length program needed to reach a given performance—when given access to the pretrained LLM—collapses by orders of magnitude with alignment/post-training. Pretraining stores most relevant information, while SFT or in-context methods merely access it via short programs (e.g., 4,358 bits for 72.2% GSM8k on Olmo3 32B), unifying SFT, prompt engineering, and parametric adaptation under a single metric (Vergara-Browne et al., 17 Feb 2026).
• Distributional support: SAH posits that aligned output distributions are simply filtered and aggregated subsets of the base distribution. All behaviors of RLHF-tuned LLMs can be elicited from the base model using in-context alignment or retrieval-augmented prompting, demonstrating that alignment does not generate new content (Lake et al., 2024).
• Multilingual setting: When fine-tuning Llama 3 8B on multilingual Alpaca (1K, 10K, 52K subsets), larger, noisier SFT sets degrade win-rate, confirming that IFT predominantly aligns the pre-existing knowledge to preferred formats rather than imparting new knowledge. The effect size varies with pretraining coverage across languages (Zhao et al., 19 Sep 2025).

5. Distinction Between Superficial and Deep Alignment

SAH demarcates superficial (style, format, simple refusal, output-level safety) alignment from deep (reasoning, multi-token causal, new knowledge) alignment:

• Superficial alignment: Encoded by shallow modifications in the final projection (output head) without altering the Transformer backbone. Captures restyling, politeness, refusal, and list/poem formatting, and saturates within a handful of examples (Chen et al., 7 Feb 2025, Raghavendra et al., 2024, Xiao et al., 9 Jun 2025).
• Deep alignment: Necessary for nontrivial reasoning, arithmetic, and real causal integration. Requires modification of the backbone, demonstrated by a continued scaling curve of task performance as a function of SFT dataset size (power-law exponents $$\Delta_B = \mathrm{Score}_{\mathrm{unaligned}(B)} - \mathrm{Score}_{\mathrm{aligned}(B)}$$1 for major tasks) (Raghavendra et al., 2024).
• Fragility and reversibility: Alignment effects are easily reversed by subsequent SFT due to a “rebound force” proportional to the narrowness of the aligned posterior; however, a latent “rehearsal priming effect” means that prior alignment can be quickly reacquired when re-exposed (Huang et al., 18 May 2026).

6. Mitigation Strategies and Emerging Remedies

Remediation approaches focus on excising superficial alignment signals or supplementing them with deeper safety reasoning objectives:

• Dataset filtering: Remove refusals and alignment-style responses from SFT datasets to prevent “poisoning” and recover lost performance (Bekbayev et al., 2023).
• Explicit safety signals: Add an auxiliary binary classification head, trained to detect malicious intent, into the model. This enables the LLM to explicitly reason about safety, rather than rely on diluted, implicit signals, and demonstrably improves robustness under adversarial prompts (to $$\Delta_B = \mathrm{Score}_{\mathrm{unaligned}(B)} - \mathrm{Score}_{\mathrm{aligned}(B)}$$20.4% ASR from >90% under strong attacks) (Li et al., 19 May 2025).
• Style-matched safety augmentation: Augment alignment datasets with a small set of adversarially constructed safety-aware examples in each major style to prevent overfitting to superficial cues (Xiao et al., 9 Jun 2025).
• Neuron-level interventions: Freezing or pruning specific sets of neurons (e.g., Exclusive Safety Units and related complex units identified via ablation) allows retention of safety alignment under further downstream fine-tuning and preserves utility (Li et al., 2024).
• Leveraging modular superficial alignment: Extracted superficial “heads” can be transferred or reattached to recover alignment in corrupted or adversarially attacked models—a cost-effective and rapid restoration mechanism (Chen et al., 7 Feb 2025).

7. Scope, Critique, and Implications for Alignment Research

While SAH is strongly supported in areas of style, format, and basic safety, recent scaling-law studies dispute its universality. Objective measurement of reasoning, coding, and knowledge integration shows monotonic performance improvements with increased SFT data, and post-training can impart genuinely new inference structures not present during pretraining (Raghavendra et al., 2024). Thus, SAH is best construed as a lower bound on alignment efficacy: alignment is often superficial, especially where objectives target output surface properties or safety without explicit structure, but can be made deep through principled architectural or objective interventions. Ongoing work operationalizes SAH via task complexity, semantic recoverability, alignment dynamics, and neuron-level ablation, providing comprehensive diagnostics for alignment strategies and motivating next-generation techniques that move beyond superficiality (Bekbayev et al., 2023, Lake et al., 2024, Chen et al., 7 Feb 2025, Vergara-Browne et al., 17 Feb 2026, Huang et al., 18 May 2026).