---
title: 'HySAN: Branch-masked Self-Attention'
url: https://www.emergentmind.com/topics/branch-masked-self-attention-hysan
type: topic
---

# HySAN: Branch-masked Self-Attention

Branch-masked Self-Attention, formalized as the Hybrid Self-Attention Network (HySAN), is an augmentation of standard self-attention for neural sequence modeling, specifically developed to address the lack of explicit local context sensitivity and content-dependent relative positional awareness in Transformer architectures. HySAN introduces a set of masked self-attention branches—each enforcing a structural prior such as locality or directionality—followed by a parameter-efficient fusion mechanism, yielding statistically significant gains in neural machine translation (NMT) across multiple benchmarks [1811.00253].

## 1. Motivation and Problem Formulation

The Transformer model utilizes scaled dot-product self-attention, computing for each token a global weighting over all other tokens via
$$
\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,
$$
with $Q, K, V \in \mathbb{R}^{n \times d_k}$ for sequence length $n$. This architecture captures global dependencies but exhibits two notable limitations:
- **Relative Position Insensitivity:** Even when supplemented with absolute positional encodings, the dot-product attention mechanism treats all pairwise token relationships as symmetric, precluding explicit modeling of “left” versus “right” context.
- **Absence of Explicit Local Focus:** Self-attention does not naturally bias toward local structures, unlike convolution or recurrence, making it less effective in capturing immediate semantic dependencies essential for linguistic phenomena in NMT.

HySAN remedies these via branch-masking, where each self-attention branch is masked to a specific contextual region (global, local, forward, backward), facilitating content- and order-sensitive representations.

## 2. Architecture: Branch-Masked Self-Attention

All branches operate on the same projected $Q, K, V$, diverging solely via an additive mask $M_b \in \mathbb{R}^{n \times n}$. For branch $b$,
$$
\text{Attention}_b(Q, K, V) = \text{softmax}\left(\frac{QK^\top + M_b}{\sqrt{d_k}}\right)V.
$$

**Branches and Corresponding Masks:**

| Branch   | Mask Description (Encoder)                                  | Effect                                 |
|----------|------------------------------------------------------------|----------------------------------------|
| Global (G)     | $M_{G}(i,j) = 0$ for all $i, j$                               | Standard global SAN                    |
| Local ($L_k$)  | $M_{L_k}(i,j) = 0$ if $|i-j| \leq k$, $-\infty$ otherwise      | Attend within window $[i-k, i+k]$      |
| Forward (FW)   | $M_{FW}(i,j) = 0$ if $j \leq i$, $-\infty$ otherwise          | Attend to tokens at positions $\leq i$ |
| Backward (BW)  | $M_{BW}(i,j) = 0$ if $i \leq j$, $-\infty$ otherwise          | Attend to tokens at positions $\geq i$ |

- In the **decoder**, local branches are masked causally: $M_{D_k}(i,j) = 0$ if $i-j \leq k$ and $j\leq i$ (ensuring no information leakage).

This architecture enables extraction of global, local, and directionally-aware contextual representations in parallel via different branches.

## 3. Squeeze Gate Fusion: Gated Sum

Each branch yields an output $x_b \in \mathbb{R}^{n \times d_v}$. Fusion across $l$ branches is performed via a lightweight “gated sum”:
$$
\text{out} = \sum_{b=1}^l x_b \odot SG(x_b)
$$
where $SG(x_b)$ is a per-token gate, defined by:
- Squeeze via $W_1$ ($d_v \to d_v/r$), ReLU, then expand via $W_2$ ($d_v/r \to 1$)
- $g_b(i) = \sigma(W_2 \,\text{ReLU}(W_1 x_b(i,:)))$ for each position $i$
- $\sigma$ is the sigmoid function.

Alternative fusion schemes—simple sum or concatenation plus projection—were empirically inferior in BLEU improvement and parameter efficiency. The squeeze gate adds two feed-forward layers per branch with minimal overhead.

## 4. Integration into Transformer Layers

HySAN replaces the self-attention module in each Transformer encoder layer (and in the encoder-side self-attention sublayer in the decoder) with a multi-head hybrid SAN. Each head $h$ computes:
1. Per-head projections: $Q_h, K_h, V_h$.
2. Shared score matrix $S = Q_h K_h^\top / \sqrt{d_k}$.
3. For each branch $b$: 
   - $S_b = S + M_b$
   - $A_b = \text{softmax}(S_b)$
   - $x_{h,b} = A_b V_h$
4. Fuse across $b$ by gated-sum to produce $x_h$.
5. Concatenate all $H$ heads and apply output linear transformation $W^O$.

Standard residual connections, layer normalization, and position-wise feed-forward layers follow as in the canonical Transformer. In the decoder, only global, causal local, and forward branches are utilized to maintain autoregressive masking.

## 5. Hyperparameterization and Implementation

Key hyperparameters for HySAN are as follows:

| Model         | $d_{model}$ | # Layers | # Heads | Encoder Branches                | Decoder Branches     |
|---------------|-------------|----------|---------|-------------------------------|----------------------|
| Small (IWSLT14) | 256         | 2        | 4       | G, FW, BW, L1, L2, L5          | G, FW, L1            |
| Base (WMT)     | 512         | 6        | 8       | G, FW, BW, L1, L2, L5          | G, FW, L1            |
| Big (WMT17)    | 1024        | 6        | 16      | G, FW, BW, L1, L2, L5          | G, FW, L1            |

- Local window radii $k$ evaluated at 1, 2, 5.
- Squeeze-gate reduction ratio $r \approx 16$ or 8.
- Regularization utilizes standard attention and feed-forward dropout.

## 6. Experimental Results and Ablation Studies

HySAN was evaluated on IWSLT14 German→English, WMT14 English→German, and WMT17 Chinese→English datasets. Optimization followed standard Adam with $(\beta_1, \beta_2, \epsilon) = (0.9, 0.98, 10^{-9})$; the learning rate was scaled as $d_{model}^{-0.5} \min(\text{step}^{-0.5}, \text{step} \cdot \text{warmup}^{-1.5})$, and decoding used beam=4 and length penalty=0.6.

Notable findings:
- **Ablation (IWSLT14, small):**
  - Global (baseline): BLEU=31.27
  - +FW: 31.50 (+0.23)
  - +BW: 31.83 (+0.56)
  - +Local $k=1$: 31.55 (+0.28)
  - All branches with gated sum: 32.28 (+1.01)
  - Without positional embeddings, baseline BLEU $\approx$ 15.6, HySAN $\approx$ 30.7 (+15), indicating that local and directional masks encode order partially independent of explicit embeddings.

- **WMT14 En→De:**
  - Base: Transformer=27.3, HySAN=27.9 (+0.6)
  - Big: Transformer=28.4, HySAN=28.8 (+0.4)

- **WMT17 Zh→En (big):**
  - Transformer=24.2, HySAN=25.27 (+1.07)

The consistent improvements across tasks and model sizes underscore the effectiveness of branch-masked mechanisms and gated fusion for NMT [1811.00253].

## 7. Significance and Empirical Observations

By introducing structured prior constraints through masking at each self-attention sublayer and selectively fusing diverse contextual views, HySAN systematically addresses two core deficiencies of canonical self-attention: insufficient local context aggregation and indifference to relative order. The ability of HySAN to recover strong performance without any positional encoding, as evidenced by the ablation with “NoPos” input, demonstrates the potency of directional and local attestation in capturing ordering cues. The approach achieves these gains with minimal architectural modification and parameter increase, making it an efficient and generalizable extension to Transformer-based models in sequence transduction tasks [1811.00253].

Source: https://www.emergentmind.com/topics/branch-masked-self-attention-hysan