---
title: 'HVDA: Horizontal-Vertical Detail Attention'
url: https://www.emergentmind.com/topics/horizontal-vertical-detail-attention-hvda
type: topic
---

# HVDA: Horizontal-Vertical Detail Attention

Horizontal–Vertical Detail Attention (HVDA) is an architectural module designed to augment self-attention mechanisms in Transformers by incorporating two orthogonal attention strategies: horizontal attention, which recalibrates multi-head outputs before projection, and vertical attention, which re-weights feature channels post-projection via explicit channel-wise modelling. Both mechanisms aim to enrich feature representation and token dependency modeling with negligible computational and parameter overhead, and can be modularly integrated into standard Transformer blocks [2207.04399].

## 1. Formal Definitions and Module Motivation

Let $X \in \mathbb{R}^{n \times D}$ denote the sequence of $n$ input tokens, each with $D$-dimensional features. Multi-head Scaled Dot-Product Attention (SDPA) produces $M$ parallel head outputs $H_1,\ldots,H_M$, $H_m \in \mathbb{R}^{n \times D_v}$, with typical dimensionalities $D_k = D_v = D/M$ per head.

- **Horizontal attention** introduces a learned re-weighting vector $\alpha = [\alpha_1, \ldots, \alpha_M] \in \mathbb{R}^{n \times M}$ (per token) to emphasize "more informative" heads before the usual linear projection, replacing the concatenation and projection step with:
  $$
  \text{Concat}(\alpha_1 \cdot H_1, \ldots, \alpha_M \cdot H_M) W^M
  $$
  instead of $\text{Concat}(H_1, \ldots, H_M) W^M$.

- **Vertical attention** recalibrates the $D$ channels of $Y^M$ (the projected multi-head output) via a channel-wise gating vector $\beta \in [0,1]^{n \times D}$, yielding rescaled output $Y^v = \beta \odot Y^M$.

The stated motivation is to enhance representation distinctiveness, model informative head outputs more selectively, and capture inter-channel dependencies [2207.04399].

## 2. Mathematical Formulation

The HVDA module formalizes attention recalibration via explicit learned transformations.

### 2.1 Horizontal Attention

Given $X\in \mathbb{R}^{n \times D}$ and $\{H_m\}_{m=1}^M$:

- For each head $m$:
  $$
  A_m = \text{ReLU}(H_m W^{a1} + X W^{a2}) \in \mathbb{R}^{n \times D_v}
  $$
  $$
  B_m = A_m W^b + b^b \in \mathbb{R}^{n \times 1}
  $$
- Stack $B_m$ across heads: $B = [B_1, \ldots, B_M] \in \mathbb{R}^{n \times M}$.

- Compute softmax over heads:
  $$
  \alpha = \text{Softmax}(B) \in \mathbb{R}^{n \times M}
  $$
  with $\sum_m \alpha_{n,m} = 1$ for each token $n$.

- Re-weight each head: $H'_m = \alpha_m \odot H_m$

- Concatenate $H'_m$: $H' = \text{Concat}(H'_1, \ldots, H'_M) \in \mathbb{R}^{n \times (M D_v)}$

- Project: $Y^H = H' W^M \in \mathbb{R}^{n \times D}$

Key matrices:

| Symbol    | Shape                            | Role                          |
|-----------|----------------------------------|-------------------------------|
| $W^{a1}$  | $\mathbb{R}^{D_v \times D_v}$    | Transform $H_m$               |
| $W^{a2}$  | $\mathbb{R}^{D \times D_v}$      | Transform $X$                 |
| $W^b$     | $\mathbb{R}^{D_v \times 1}$      | Final linear head score       |
| $b^b$     | $\mathbb{R}^{1}$                 | Head score bias               |
| $W^M$     | $\mathbb{R}^{(M D_v) \times D}$  | Output projection             |

### 2.2 Vertical Attention

Given $Y^M \in \mathbb{R}^{n \times D}$ and $X \in \mathbb{R}^{n \times D}$:

- Compute squeezed representation:
  $$
  U = \text{ReLU}(X W^{u1} + Y^M W^{u2}) \in \mathbb{R}^{n \times D_a}
  $$
- Gating vector:
  $$
  \beta = \text{Sigmoid}(U W^u + b^u) \in \mathbb{R}^{n \times D}
  $$
- Channel-wise recalibration:
  $$
  Y^v = \beta \odot Y^M \in \mathbb{R}^{n \times D}
  $$

Key matrices:

| Symbol    | Shape                     | Role                                 |
|-----------|---------------------------|--------------------------------------|
| $W^{u1}$, $W^{u2}$ | $\mathbb{R}^{D \times D_a}$  | Linear projections          |
| $W^u$     | $\mathbb{R}^{D_a \times D}$ | Mapping to gating vector             |
| $b^u$     | $\mathbb{R}^D$               | Channel recalibration bias           |
| $D_a$     | $< D$ (e.g., $D/4$)          | Channel squeeze dimension            |

## 3. Integration within Transformer Architectures

HVDA is integrated into the standard Transformer block as a modular augmentation or replacement of the conventional multi-head attention sublayer. The following steps comprise the forward pass:

1. **Multi-head SDPA**: Compute $M$ parallel head outputs $H_1,\ldots,H_M$ from the input $X$.
2. **Horizontal Attention**: If enabled, compute per-token, per-head weights $\alpha$ and recalibrate $H'_m=\alpha_m\odot H_m$. Concatenate and project to obtain $Y^M$.
3. **Vertical Attention**: If enabled, compute channel-wise gating $\beta$ and element-wise modulate $Y^M$ to yield $Y^v$.
4. **Residual and Layer Normalization**: Apply $Z = \text{LayerNorm}(X + Y^v)$.
5. **Feed-forward Sublayer**: Apply standard feed-forward and residual structure on $Z$.

A concise pseudocode, as described in the original source [2207.04399], is:

```python
def TransformerBlock_HV(X):
    # 1. Multi-head SDPA
    for m in 1…M:
        H[m] = ScaledDotProdAttn(X·Wq_m, X·Wk_m, X·Wv_m)
    # 2. Horizontal attention
    if use_horizontal:
        for m in 1…M:
            A[m] = ReLU(H[m]·Wa1 + X·Wa2)
            B[m] = A[m]·Wb + bb
        Bcat = concat_along_head(B[1],…,B[M])
        α = Softmax(Bcat, axis=head)
        for m in 1…M:
            Hʹ[m] = α[:, m] ∘ H[m]
        Hcat = concat_along_feature(Hʹ[1],…,Hʹ[M])
    else:
        Hcat = concat_along_feature(H[1],…,H[M])
    Yᴹ = Hcat·WM
    # 3. Vertical attention
    if use_vertical:
        U = ReLU(X·Wu1 + Yᴹ·Wu2)
        β = Sigmoid(U·Wu + bu)
        Yᵛ = β ⊙ Yᴹ
    else:
        Yᵛ = Yᴹ
    # 4. Residual + LayerNorm
    Z = LayerNorm(X + Yᵛ)
    # 5. Feed-forward sublayer
    return FFN(Z) + Z
```

## 4. Hyperparameters and Configuration

Key hyperparameters for deploying HVDA are:

- $M$: Number of SDPA heads, typically unchanged from the baseline Transformer configuration.
- $D_a$: "Channel squeeze" dimension used in vertical attention ($D_a < D$, e.g., $D_a = D / 4$).
- $D_k = D_v$: Key/query and value dimensions per head, commonly set as $D / M$.

No constraints are imposed on the standard architectural parameters of the Transformer aside from the introduction of the extra weights referenced above.

## 5. Computational Overhead and Complexity

HVDA is characterized by minimal computational and storage overhead:

- The core cost of multi-head SDPA remains $O(MN^2D)$ per block, requiring $2M D^2$ parameters.
- **Horizontal attention** introduces approximately $O(N M D_v + N D D_v)$ computations per token, with parameter increment of approximately $2D^2 + D$ for $D_v = D$.
- **Vertical attention** incurs an additional $O(N D D_a + N D D_a)$ cost per token and about $3D^2$ extra parameters for $D_a = D$.
- Both modules affect only $O(D^2)$ storage. Empirically, the observed floating-point operations (FLOPs) and parameter overhead are less than $10\%$ of the baseline [2207.04399].

A table summarizing complexity increments:

| Module           | Time Complexity           | Param Increase (if $D=D_v=D_a$) |
|------------------|--------------------------|----------------------------------|
| Vanilla SDPA     | $O(MN^2D)$               | $2MD^2$                          |
| + Horizontal     | $O(MD)$ per token        | $2D^2 + D$                       |
| + Vertical       | $O(MD)$ per token        | $3D^2$                           |

*This suggests that HVDA can be incorporated into existing architectures with negligible relative resource increase.*

## 6. Modularity and Applicability

HVDA is described as highly modular, enabling insertion into a wide variety of Transformer models to yield performance gains in supervised learning tasks. The augmentation is compatible with vanilla Transformers and does not require modification to the SDPA core or positional encoding. The mechanisms for horizontal and vertical attention can be enabled or disabled independently in each block, facilitating flexible architectural experimentation [2207.04399].

## 7. Empirical Observations and Generalization

The authors demonstrate that Transformers equipped with HVDA modules exhibit high generalization capability across different supervised tasks, with only minor increases in computational cost or parameter count. The code for reference implementation is provided in the supplementary material of the original report, enabling straightforward integration and replication. A plausible implication is that these selective re-weighting and channel recalibration mechanisms may enhance feature expressivity and robustness without compromising efficiency [2207.04399].

Source: https://www.emergentmind.com/topics/horizontal-vertical-detail-attention-hvda