Layered Input/Output Validation

• Layered Input/Output Validation is a mechanism in LLMs where separate circuits compute arithmetic solutions and validate alignment between computed and presented outputs.
• Experimental paradigms leverage edge attribution patching and behavioral tests, revealing variable error detection accuracies (60%–99%) across different models.
• Targeted interventions like consistency-head patching and residual-stream injection effectively bridge the validation gap, significantly boosting error detection.

Layered input/output validation is a mechanistic phenomenon observed in LLMs, whereby distinct circuits—operating in separate layers—are responsible for computing outputs (such as arithmetic results) and for validating the alignment between computed results and presented answers. In contemporary instruction-tuned decoder-only Transformers, arithmetic computation is concentrated in higher layers, while validation of outputs is mainly performed by attention heads in intermediate layers, termed "consistency heads." This architectural dissociation leads to a "validation gap," where models can internally compute correct results while relying only on surface-level input/output alignment for validation, resulting in systematic failures to detect certain errors (Bertolazzi et al., 17 Feb 2025).

1. Architectures, Datasets, and Experimental Paradigm

The mechanistic analysis of layered input/output validation has focused on four instruction-tuned decoder-only Transformer models:

Model	Layers	Heads	Hidden Size
Qwen-2.5-1.5B-Instruct	28	12	1536
Qwen-2.5-Math-1.5B-Instruct	28	12	1536
Llama-3.2-3B-Instruct	28	24	3072
Phi-3-Mini-4k-Instruct	32	32	3072

Arithmetic error detection capabilities are probed using eight “fill-in” templates for simple addition word problems. Each template produces 6,000 pairs of clean (erroneous) and corrupt (no-error) prompts, covering both incorrect intermediate results and incorrect final answers. Behavioral tests yield detection accuracies ranging from 60% to 99% across models, demonstrating baseline capacity for error detection.

Circuit identification is achieved via edge attribution patching (EAP), isolating the minimal set of edges sufficient to recover model logit-difference behavior between valid and invalid outputs.

2. Division of Computation and Validation Across Layers

The computation of arithmetic outputs and the validation of those outputs are realized by functionally and geographically dissociated subgraphs.

• Computation Subgraph: Circuits encoding the solution token (e.g., producing “13” for “5 + 8 =”) are found overwhelmingly in the upper layers, generally layers 20–28. Linear probing of residual streams demonstrates near-perfect sum decodability from layer 20 onwards; below these layers, the required semantic information is not yet linearly accessible.
• Validation Subgraph: Consistency heads, implementing input/output validation (invalid/valid), are localized in mid-layers—layers 10–14 in Qwen models and layers 4–10 in Llama/Phi. These heads specialize in surface-level digit alignment checks between computed results and presented answers.

Formally, for attention head $h$, result position $i$, and answer position $j$, the average attention pattern is: $$A^h(i \to j) = \mathbb{E}_{\text{prompts}}\,\mathrm{softmax}\left(Q_h k_h^\top/\sqrt{d}\right)_{i,j}$$ A consistency head exhibits

$$A^h(\mathtt{[result\text{-}first]} \to \mathtt{[answer\text{-}first]}) \gg 0,\quad A^h(\mathtt{[result\text{-}second]} \to \mathtt{[answer\text{-}second]}) \gg 0$$

only when the digits align, and at least one drops if digits are misaligned.

3. Circuit Analysis by Edge Attribution and Faithfulness

Edge attribution patching precisely maps attention heads and residual stream pathways to the functions of computation and validation.

• Edge Attribution Patching (EAP): For each model edge $e$ and metric $\mathcal{P}$ (average logit difference between “valid” and “invalid”), the edge's attribution is

$$\Delta_e \approx (X_\text{clean})_e - (X_\text{corr})_e \cdot \frac{\partial \mathcal{P}}{\partial e}\Big|_{X_\text{corr}}$$

Edges are ranked by $|\nabla_e \mathcal{P}|$ to identify functional circuits.

• Faithfulness: A circuit $\mathcal{C}$ is faithful if

$$\mathrm{Faithfulness} = \frac{1}{N} \sum_{i=1}^N \frac{\mathcal{C}(X_{i,\text{clean}}) - \mathcal{C}(X_{i,\text{corr}})}{M(X_{i,\text{clean}}) - M(X_{i,\text{corr}})} \times 100$$

falls within $[99\%, 101\%]$.

• Soft Template Intersection: For eight templates, soft intersection yields

$f(e) = \frac{1}{8} \sum_{i=1}^8 \mathds{1}_{\mathcal{C}_i}(e), \quad \mathcal{C}^{(\tau)} = \{e \mid f(e) \ge \tau\}$

enabling a trade-off between circuit size and faithfulness.

Causal tests confirm that patching as few as six consistency heads suffices to restore error detection on challenging “consistent-error” cases, whereas patching random heads has no effect.

4. Successes and Limitations: Behavioral and Mechanistic Findings

Successful detection occurs when result and answer tokens are inconsistent:

• Result Error Only: With an input such as “… 5+8 = 16. Thus, Jane has 13 apples. …”, the layer 12 consistency head attends “1” (of 16) to “1” (of 13) but not “6” to “3”, yielding "invalid."
• Answer Error Only: In “… 5+8 = 13. Thus, Jane has 16 apples. …”, the head attends “1”→“1”, not “3”→“6”, also predicting "invalid."
• Consistent Error Failure: In “… 5+8 = 16. Thus, Jane has 16 apples. …”, heads attend both digits correctly, resulting in a false “valid” prediction.

Linear probes of residual activations below layer 20 decode sums at near-chance, confirming that validation is initiated before the computation subgraph’s output is accessible.

5. Structural Dissociation and the Validation Gap

A two-stage hierarchy characterizes layered input/output validation.

• Top-Heavy Computation: Arithmetic subgraph activity and decodable sum information are concentrated in the highest layers (20–28).
• Mid-Layer Validation: Consistency heads operate in layers 10–14, well before the correct sum is reflected in activations.

This partition causes the validation operation to rely on superficial digit-alignment heuristics rather than direct assessment of true computed results. As a result, validation heads never “see” the final computed number during their operation. This "validation gap" manifests in models correctly performing internal arithmetic but failing to identify errors when both result and answer are mutually consistent but incorrect.

6. Bridging Techniques and Interventions

Directly addressing the validation gap, two interventions have proven effective:

• Consistency-Head Patching: Transplanting six heads associated with error detection from an “error-result” instance into a “consistent-error” instance increases error detection accuracy from ~12% to ~95%, without affecting simpler detection cases.
• Residual-Stream Injection: Adding the hidden activation at layer 22, [result-first] token position, into layer 1, [result-second] token gives mid-layer validation units access to the actual computed value: $$r^1_{\text{[result‐second]}} \leftarrow r^1_{\text{[result‐second]}} + H^{22}_{\text{[result‐first]}}$$ This yields an ~80-point accuracy increase on the consistent-error set, with no adverse effect on other error types. Generalizing: $$\mathrm{res}^l_{t_2}\mapsto\mathrm{res}^l_{t_2} + \alpha\,H^{l'}_{t_1}\quad(l'<l,\,\alpha\approx1)$$ Such patching closes the validation gap by enabling validation heads to condition on internal computations, not just surface-level cues.

7. Implications and Research Trajectories

Layered input/output validation exposes a structural pitfall in LLMs: dissociated computation and validation circuits enable correct internal processing without robust error flagging. This suggests avenues for architectural modification, such as residual-steam bridging, and motivates future studies on mechanistic transparency of self-correction and internal error detection. These findings delineate that bridging previously disjoint subgraphs is both necessary and sufficient to robustly detect inconsistencies, highlighting the need for integrating validation mechanisms with deep semantic computation (Bertolazzi et al., 17 Feb 2025).