---
title: Local Bidirectional Attention (LBA)
url: https://www.emergentmind.com/topics/local-bidirectional-attention-lba
type: topic
---

# Local Bidirectional Attention (LBA)

Local Bidirectional Attention (LBA) is an attention mechanism that synthesizes local and bidirectional context, enabling efficient and expressive modeling of sequential data and multimodal word-level correspondences. It is employed in high-efficiency long-range parsers such as BLRP for textual and vision data [2404.05210], as well as in cross-lingual style transfer frameworks for automatic dubbing [2305.05203]. LBA constructs local sliding-window or local cross-lingual attention blocks and aggregates bidirectional latent representations, capturing both short-range dependencies and global structure with computational scalability.

## 1. Formal Definition and Core Principles

LBA builds on the principle of local attention, restricting each input segment’s receptive field to a short neighborhood, and bidirectional passes, whereby local features are synthesized in both forward and backward directions and merged through a global latent representation. This is instantiated in BLRP by partitioning the input $X \in \mathbb{R}^{N \times d}$ into $T$ non-overlapping segments $X_1, ..., X_T$ of length $t$ and constructing, for each segment $i$, a window $W_i$ containing $w$ tokens from adjacent segments:
$$
W_i = [X_{i-1}^{(s)}; X_i; X_{i+1}^{(p)}]
$$
where $X_{i-1}^{(s)}$ are the last $\lfloor w/2 \rfloor$ tokens of the previous segment, and $X_{i+1}^{(p)}$ are the first $\lfloor w/2 \rfloor$ tokens of the next segment. Standard scaled-dot-product self-attention is applied within each window.

In cross-lingual sequence modeling [2305.05203], LBA takes two word-level feature matrices (source and target sequences), projects keys and values for each, and computes a shared bidirectional attention matrix:
$$
A_{1 \rightarrow 2} = f_1(s_1^{t}) f_2(s_2^{t})^\top
$$
$$
A_{2 \rightarrow 1} = A_{1 \rightarrow 2}^\top
$$
Row-wise softmax yields attention weights in both directions, supporting simultaneous fusion of multimodal features across languages.

## 2. Mathematical Formulation

### BLRP Local Sliding-Window Attention
For each segment $i$, queries, keys, and values are:
- $Q_i = q^S(X_i) \in \mathbb{R}^{t \times d}$
- $K_i = k^S(W_i) \in \mathbb{R}^{w \times d}$
- $V_i = v^S(W_i) \in \mathbb{R}^{w \times d}$

The local attention output is:
$$
\Theta^{\rm SELF}(X_i) = \operatorname{softmax}\Bigl(\frac{Q_i K_i^\top}{\sqrt d}\Bigr) V_i
$$
with each pre-softmax element:
$$
s_{j,m} = \frac{1}{\sqrt d} \, q^S(x_{(i-1)t + j}) \cdot k^S(w_m)
$$
Softmax normalization over window positions $m$ yields attention weights $\alpha_{j,m}$:
$$
\bigl[\Theta^{\rm SELF}(X_i)\bigr]_{j,:} = \sum_{m=1}^w \alpha_{j,m} v^S(w_m)
$$

### Bidirectional Latent-Space Synthesis
A latent block $L \in \mathbb{R}^{l \times d}$ is updated via two passes:
- Forward: $L_i^F = \Theta_L^{CROSS}[L_{i-1}^F, [\tilde X_i^F ; L^{\rm INIT}]]$
- Backward: $L_i^B = \Theta_L^{CROSS}[L_{i+1}^B, [\tilde X_i^B ; L^{\rm INIT}]]$

Each cross-attention step uses scaled-dot-product attention analogous to self-attention.

### Cross-Lingual Bidirectional Attention
Local bidirectional attention between source and target sequences uses two sets of keys/values $([f_1(s_1^t), s_1^m], [f_2(s_2^t), s_2^m])$:
- $A_{1 \rightarrow 2}$ yields source $\rightarrow$ target alignment.
- $W_{1 \rightarrow 2} = \operatorname{softmax}(A_{1 \rightarrow 2})$
- Summarized outputs: $O_{1 \rightarrow 2} = W_{1 \rightarrow 2}^\top s_1^m$ and $O_{2 \rightarrow 1} = W_{2 \rightarrow 1}^\top s_2^m$
Concatenated with textual features and projected to generate predicted local style tokens.

## 3. Algorithm and Implementation

Pseudocode for BLRP’s LBA module:

```python
# Input: X ∈ R^{N×d}, segment length t, window size w, latent size l
# Output: L_final ∈ R^{l×d}

def BLRP_LocalBidirectional(X):
    segments = split(X, t)
    L_init_F = Theta_proj_forward(X)
    L_init_B = Theta_proj_backward(X)
    L_F_prev = L_init_F
    L_B_next = L_init_B
    
    # Forward pass
    for i in range(1, T+1):
        W_i = get_window(segments, i, w)
        S_i = Softmax((q^S(segments[i]) @ k^S(W_i).T) / sqrt(d)) @ v^S(W_i)
        Xtilde_i_F = Θ_X^CROSS(S_i, L_F_prev)
        L_F_i = Θ_L^CROSS(L_F_prev, concatenate(Xtilde_i_F, L_init_F))
        L_F_prev = L_F_i
    
    # Backward pass
    for i in reversed(range(1, T+1)):
        W_i = get_window(segments, i, w)
        S_i = Softmax((q^S(segments[i]) @ k^S(W_i).T) / sqrt(d)) @ v^S(W_i)
        Xtilde_i_B = Θ_X^CROSS(S_i, L_B_next)
        L_B_i = Θ_L^CROSS(L_B_next, concatenate(Xtilde_i_B, Xtilde_i_F, L_init_B))
        L_B_next = L_B_i

    return L_final = L_B_1
```

Key implementation recommendations include segment/window size $t = w \approx 100$, latent block size $l = t$, overlapping tokens at segment boundaries, independent forward/backward projection heads, and AdamW optimization with fine-tuned learning rates and hyperparameters [2404.05210].

## 4. Computational Efficiency

LBA achieves substantial improvements in runtime and memory usage relative to full self-attention. For BLRP:
- **Full self-attention:** $O(N^2 d)$ time, $O(N^2)$ memory.
- **Local attention (LBA):** For $T = N/t$ segments each attending a window $w \approx t$,
  $O(N t d)$ total time.
- **Cross-attention:** $O(N l d)$; if $l = O(t)$, overall $O(N t d)$, nearly linear in $N$ for fixed $t$.

This scaling allows efficient processing of very long sequences ($N$ up to 16k), with only linear memory growth, in contrast to quadratic blowup for vanilla Transformers [2404.05210].

## 5. Empirical Performance and Ablation Studies

LBA contributes significant gains in accuracy and modeling fidelity across tasks.

### BLRP (Long-Range-Arena Text Benchmarks):
- ListOps accuracy: 41.43% (Longformer: 37.5%, TLB: 38.2%)
- Text classification: 82.83% (Longformer: 66.0%, TLB: 82.08%)
- Retrieval: 83.43% (Longformer: 81.79%, TLB: 76.91%)

Ablations confirm LBA’s effectiveness:
- Small segment sizes (e.g., $t=10$) degrade performance ($<$38% accuracy).
- Bidirectional updates outperform unidirectional passes, evidenced by ListOps accuracy drops into the high 39s for unidirectional variants.
- Linear scaling up to sequence lengths $N=16\,000$ without GPU memory exhaustion [2404.05210].

### Automatic Dubbing (Cross-lingual Multi-scale Style Transfer):
- Mel-spectrogram MSE (en→zh): FastSpeech 2 baseline 4.694, duration-transfer 3.695, multi-scale transfer with LBA 1.392.
- MOS for style: None 3.16±0.08, duration-only 3.92±0.07, multi-scale (GST+LST via LBA) 4.12±0.07.
- User preference: Multi-scale LBA solution preferred by 65.5%.
- Removing local LBA raises mel-MSE from 1.39 to 1.68, confirming its necessity for style fidelity [2305.05203].

## 6. Architectural Variants and Cross-Task Generalization

LBA enables versatile pattern modeling. In BLRP, short-range dependencies are modeled with sliding windows, while recurrent latent blocks synthesize context bidirectionally. In cross-lingual frameworks, LBA mediates word-level style and text correspondence across languages by simultaneously computing source-to-target and target-to-source attention and enabling joint training objectives.

A shared pair of projection networks and attention blocks enforce a unified semantic-style mapping. Bidirectionality is realized not only by using both sequence orientations, but also by tying the model parameters and alignment matrices across translation directions.

## 7. Practical Recommendations and Limitations

Select window/segment sizes matched to the task domain, typically $t=w=100$ for long-text or word-level applications. Latent block size should equal segment length ($l=t$) for optimal accuracy. Utilize two independent dynamic-projection heads for forward and backward passes. When implementing, include overlapping tokens at segment boundaries to mitigate loss of context. Depth per layer should consist of two local self-attention operations and two cross-attentions. AdamW optimizer with batch size 32, embedding size 64, hidden size 128, and 8 attention heads is recommended for LRA tasks; learning rate $4 \times 10^{-4}$, $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$, weight decay $0.8$, and linear decay scheduling yield strong convergence [2404.05210].

A plausible implication is that LBA’s local partitioning is sensitive to hyperparameter choices, particularly segment size and overlap. Under-sizing severely limits expressiveness, whereas excessive overlap or overly large windows may erode the method’s scalability. Modelers should balance locality against latent block dimension to avoid performance collapse.

---

In summary, Local Bidirectional Attention integrates local, context-sensitive modeling with bidirectional latent-space aggregation, demonstrably improving long-sequence parsing efficiency and performance as well as multimodal cross-lingual alignment for tasks requiring rich local structure and global context synthesis [2404.05210, 2305.05203].

Source: https://www.emergentmind.com/topics/local-bidirectional-attention-lba