Horizontal Attention Gradient Dynamics

• Horizontal Attention Gradient is the derivative of loss with respect to attention scores, clarifying how error signals redistribute across tokens.
• The mechanism reallocates attention mass from tokens with negative advantage to those with positive advantage, resembling an EM-like specialization process.
• It enables practical interventions like attention-patching to steer model updates, enhancing interpretability and aligning gradient flow with Bayesian reasoning.

Horizontal attention gradients characterize the mechanism by which transformer attention heads redistribute error signals across positions (tokens) in sequence models during backward propagation. In contrast to vertical gradients, which aggregate error signals across queries for a fixed key or value, horizontal gradients explain how a given query updates its allocation of attention across all available keys and values. These gradients offer a principled account of how attention routing, content specialization, and internal geometry evolve during optimization, connecting the mathematical details of gradient flow to statistical procedures such as the expectation-maximization (EM) algorithm and Bayesian reasoning.

1. Definitions and Fundamental Quantities

Consider a single-head attention block with input sequence length $T$ and input vectors $x_j \in \mathbb{R}^{d_x}$. Projections yield query, key, and value vectors: $q_i = W_Q x_i \in \mathbb{R}^{d_k}$, $k_j = W_K x_j \in \mathbb{R}^{d_k}$, and $v_j = W_V x_j \in \mathbb{R}^{d_v}$. The attention scores are $s_{ij} = q_i^\top k_j / \sqrt{d_k}$, producing attention weights via softmax: $$\alpha_{ij} = \exp(s_{ij}) / \sum_{r=1}^T \exp(s_{ir})$$. Context vectors are formed as $g_i = \sum_{j=1}^T \alpha_{ij} v_j$, which feed into output logits and probabilities.

With cross-entropy loss $L = -\sum_{i=1}^T \log p_{i, y_i}$, define the upstream gradient at the context as $$u_i = \partial L / \partial g_i = W_O^\top(p_i - e_{y_i}) \in \mathbb{R}^{d_v}$$. Compatibility scores between the gradient direction and each value vector are $x_j \in \mathbb{R}^{d_x}$0; their expectation under the attention distribution is $x_j \in \mathbb{R}^{d_x}$1.

2. Horizontal Gradient Law: Derivation and Implications

The "horizontal attention gradient" refers specifically to $x_j \in \mathbb{R}^{d_x}$2, the gradient of the loss with respect to the attention score at position $x_j \in \mathbb{R}^{d_x}$3 for fixed query $x_j \in \mathbb{R}^{d_x}$4 and varying key $x_j \in \mathbb{R}^{d_x}$5. By direct differentiation, the gradient takes the form:

$x_j \in \mathbb{R}^{d_x}$6

This formula is succinctly termed the advantage-based routing law (Aggarwal et al., 27 Dec 2025). Here, $x_j \in \mathbb{R}^{d_x}$7 is the attention "responsibility" that query $x_j \in \mathbb{R}^{d_x}$8 assigns to value $x_j \in \mathbb{R}^{d_x}$9; $q_i = W_Q x_i \in \mathbb{R}^{d_k}$0 quantifies how beneficial increasing $q_i = W_Q x_i \in \mathbb{R}^{d_k}$1 would be relative to the backward signal; and $q_i = W_Q x_i \in \mathbb{R}^{d_k}$2 sets a local baseline for comparison.

In gradient descent, attention mass is reallocated from positions with negative advantage ($q_i = W_Q x_i \in \mathbb{R}^{d_k}$3) toward those with positive advantage ($q_i = W_Q x_i \in \mathbb{R}^{d_k}$4). This results in a dynamic feedback process wherein both score allocations and value vector contents evolve jointly.

3. Reversed Attention and Gradient Flow Across Tokens

The horizontal gradient flow is further illuminated by the notion of "Reversed Attention" (Katz et al., 2024). Starting from the output $q_i = W_Q x_i \in \mathbb{R}^{d_k}$5 (row-wise dot products), and denoting $q_i = W_Q x_i \in \mathbb{R}^{d_k}$6, one obtains the gradient with respect to the attention matrix $q_i = W_Q x_i \in \mathbb{R}^{d_k}$7 as $q_i = W_Q x_i \in \mathbb{R}^{d_k}$8. Backpropagating through the softmax yields, on each row,

$q_i = W_Q x_i \in \mathbb{R}^{d_k}$9

Forming the Reversed Attention matrix $k_j = W_K x_j \in \mathbb{R}^{d_k}$0,

$k_j = W_K x_j \in \mathbb{R}^{d_k}$1

This matrix captures, for every query $k_j = W_K x_j \in \mathbb{R}^{d_k}$2, the "horizontal" reallocation of gradient signal across key positions $k_j = W_K x_j \in \mathbb{R}^{d_k}$3. $k_j = W_K x_j \in \mathbb{R}^{d_k}$4 is lower-triangular for causal models and is typically sparse, pinpointing the most influential token pairs for gradient-based adjustment.

4. Specialization and Positive Feedback: EM Analogy

The coupled update laws for scores (horizontal) and values (vertical) induce a specialization effect, whereby queries increasingly focus attention on the values most aligned with their error signals, and those values adapt to serve the needs of their principal queries. The process exhibits a two-timescale structure analogous to EM in mixture models (Aggarwal et al., 27 Dec 2025):

• E-step: The attention weights $k_j = W_K x_j \in \mathbb{R}^{d_k}$5 act as soft responsibilities, dynamically favoring columns $k_j = W_K x_j \in \mathbb{R}^{d_k}$6 with above-average compatibility for the current query $k_j = W_K x_j \in \mathbb{R}^{d_k}$7.
• M-step: Value vectors $k_j = W_K x_j \in \mathbb{R}^{d_k}$8 aggregate upstream gradients $k_j = W_K x_j \in \mathbb{R}^{d_k}$9, weighted by the current attention assignments, shifting toward the "centers" of their assigned queries.

Empirically, attention weights stabilize rapidly (fast E-step), locking the attention pattern early in training, while values continue drifting (slow M-step), supporting ongoing error reduction and model calibration.

5. Updating Values and Controlling Attention Directly

The update rule for value vectors under SGD with learning rate $v_j = W_V x_j \in \mathbb{R}^{d_v}$0 is:

$v_j = W_V x_j \in \mathbb{R}^{d_v}$1

This aggregates contributions from all queries according to their current assignment to value $v_j = W_V x_j \in \mathbb{R}^{d_v}$2. In practice, attention-patching exploits the interpretable structure of Reversed Attention $v_j = W_V x_j \in \mathbb{R}^{d_v}$3 for direct intervention (Katz et al., 2024): replacing the forward attention matrix $v_j = W_V x_j \in \mathbb{R}^{d_v}$4 with $v_j = W_V x_j \in \mathbb{R}^{d_v}$5 allows external steering of attention flow at inference-time, without updating weights.

Experimental evidence demonstrates that such attention-patching can match or outperform few-shot prompting in in-context learning tasks, and that the magnitude and sparsity in $v_j = W_V x_j \in \mathbb{R}^{d_v}$6 allow prioritized, fast localization of important head-token interactions.

6. Geometry, Bayesian Manifolds, and Statistical Interpretation

Horizontal attention gradients play a key role in sculpting the internal geometry of transformer representations. They drive keys and queries to span nearly orthogonal axes in key space, sharpening the competition among value vectors and yielding low-dimensional subspaces where residual errors are minimized. These subspaces correspond to posterior-entropy manifolds as predicted by Bayesian inference models (Aggarwal et al., 27 Dec 2025). Once the attention assignment $v_j = W_V x_j \in \mathbb{R}^{d_v}$7 stabilizes, further changes are restricted to this subspace, cementing the connection between gradient optimization and statistical reasoning.

A plausible implication is that horizontal attention gradients provide the mechanism that unifies optimization (via gradient flow), emergent geometry (via manifold formation), and functional capacity (in-context probabilistic reasoning) in transformer architectures.

7. Interpretability, Practical Analysis, and Future Directions

Horizontal attention gradients offer an interpretable, fine-grained lens on transformer dynamics. The sparsity of Reversed Attention $v_j = W_V x_j \in \mathbb{R}^{d_v}$8 highlights key token pairs responsible for model updates, supporting efficient head ranking and intervention. The ability to inject $v_j = W_V x_j \in \mathbb{R}^{d_v}$9 in forward passes enables accurate behavior modification without retraining or parameter adjustment.

Ongoing research continues to elucidate the hierarchy and interaction among horizontal and vertical gradients, the role of coupled specialization dynamics in complex sequence learning, and the correspondence between empirical manifolds and Bayesian posteriors. These results offer foundational insights for both theoretical analysis and practical engineering of interpretable, controllable attention mechanisms.