---
title: Bi- and Uni-Directional Attention in Neural Models
url: https://www.emergentmind.com/topics/bi-directional-and-uni-directional-attention
type: topic
---

# Bi- and Uni-Directional Attention in Neural Models

Bi-directional and uni-directional attention are foundational mechanisms in modern neural architectures for sequence modeling, notably in NLP, biological, and robotic domains. Uni-directional (causal) attention restricts each token to attend only to its past and self, thus imposing an inherent ordering compatible with autoregressive generation. Bi-directional attention, in contrast, allows each token to access the full context on both sides, supporting tasks such as masked language modeling and semantic probing that benefit from full-context inference. Recent research rigorously characterizes their mathematical structures, comparative representational properties, and applications in diverse domains, as well as provides new insights into their inductive biases and computational trade-offs.

## 1. Formal Definitions and Core Mechanisms

### Uni-directional (Causal) Attention

Uni-directional attention enforces a lower-triangular masking scheme. For a sequence of length $n$, the causal mask $M\in\mathbb{R}^{n\times n}$ is defined as:
\[
M_{ij} =
  \begin{cases}
    0, & j \le i \\
    -\infty, & j > i
  \end{cases}
\]
Self-attention with this mask (cf. Eq. (2) in [2510.01652]) is given by:
\[
\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V
\]
This structure restricts each position $i$ to attend only to positions up to $i$, supporting left-to-right generation and encoding explicit ordinal information via the mask’s triangular structure, even without explicit positional embeddings [2604.18603].

### Bi-directional (Full-context) Attention

In bi-directional attention (e.g., BERT-style), every token potentially attends to every other, using an all-zeros mask:
\[
M'_{ij} = 0\quad \forall \,i,j
\]
This enables access to both left and right contexts within each attention layer, supporting tasks requiring full-context inference such as MLM, semantic similarity, or probing tasks [2510.01652, 2307.04057].

## 2. Bidirectional Attention: Architectural and Statistical Characterizations

Bidirectional attention can be formalized as a statistical mixture-of-experts (MoE) system. "Bidirectional Attention as a Mixture of Continuous Word Experts" establishes that a single-layer, single-head bidirectional self-attention model with positional encodings and MLM is mathematically equivalent to a continuous bag-of-words (CBOW) model, but with soft, data-dependent MoE weights:
\[
\text{logit}_k = \sum_{j=1}^S \pi_j(\bar{X}) f_j(\bar{X})_k
\]
where the mixture weights $\pi_j$ and experts $f_j$ are learned from token+position embeddings, and the cross-entropy loss coincides with a weighted MoE-variant of CBOW [2307.04057].

With multiple heads/layers, bidirectional attention corresponds to stacked and deep mixtures of MoEs, each allowing tokens to aggregate information from all positions with data-driven expert routing.

### Dual Triangle Attention (DTA)

Dual Triangle Attention (DTA) introduces an architectural variant that splits each head into two sub-heads—one with a past-and-self mask, another with a future-and-self mask. Each sub-head computes directional attention along its mask (lower- or upper-triangular), restoring the positional inductive bias of causal attention in both directions without requiring additional learned parameters or positional embeddings. This maintains efficient context aggregation and strong performance in both NLP and bioinformatics sequence tasks [2604.18603].

## 3. Contextual and Developmental Properties

### Information Flow and Inductive Bias

Causal masking supplies an ordinal signal that enables Transformers to recover absolute position information without positional embeddings. Standard (fully bidirectional) self-attention is permutation-invariant in the absence of position encodings, removing any implicit order. DTA restores a coarse positional prior via mask geometry, complementary to mechanisms such as Rotary Positional Embeddings (RoPE) [2604.18603].

### Task-driven Dynamic Reweighting (Robotics and BU/TD Fusion)

In developmental robotics, bi-directional attention can fuse bottom-up (BU, saliency-driven) and top-down (TD, prediction-driven) streams. In A³RNN, BU saliency maps and TD queries are fused via cross-attention, supporting a developmental trajectory from exploration (BU-driven) to exploitation (TD-driven). This bi-directional loop enables mutual refinement, producing more coherent and robust attention patterns compared to uni-directional approaches [2510.10221].

| Setting              | Mechanism                             | Information Flow        |
|----------------------|---------------------------------------|------------------------|
| NLP (BERT/LLM2Vec)   | Bidirectional Self-Attention (+MNTP)  | Any↔Any (Full context) |
| Robotics (A³RNN)     | Cross-attn fusion (BU↔TD)             | BU→TD and TD→BU        |
| Causal Transformers  | Uni-directional Mask                  | Left-context only      |

## 4. Empirical Results and Practical Implications

Extensive probing and masked modeling results establish several key findings:

- Uni-directional LLMs cannot faithfully encode left-context information for embedding/probing tasks; BERT-Base achieves 94.2% on left-context subject-animacy, while causal Llama yields only 54.7% [2510.01652].
- Simple bidirectional mask removal ("Bi + MNTP") in decoder-only models recovers right-context but trades off left-context; full context is only restored by contrastive fine-tuning (SimCSE or NLI-supervised) [2510.01652].
- DTA matches or slightly exceeds standard bidirectional attention on long-context and MLM benchmarks, while requiring no positional embeddings and maintaining efficient parameterization [2604.18603].
- In tabular data, the Mixture-of-Experts view enables an extension ("ATN") that outperforms transformer-based tabular baselines under distribution shift conditions [2307.04057].
- In developmental robotics, bi-directional (BU ↔ TD) fusion yields more stable and interpretable trajectories, with cross-attention supporting adaptive refinement not recoverable under BU→TD or TD→BU alone [2510.10221].

## 5. Computational and Design Trade-Offs

Bidirectional attention, by removing the causal mask, increases the number of nonzero dot-products in each head from $\tfrac12 n(n+1)$ to $n^2$, yielding approximately $2\times$ the FLOPs and memory costs for the attention computation and gradients [2510.01652]. Inference latency grows by a similar factor, though optimized kernels (e.g., FlashAttention, flex_attention) may mitigate this penalty [2604.18603]. DTA offers hardware-efficient, parameter-neutral bidirectional attention via fused kernel implementation.

A plausible implication is that for applications requiring full-context encoding (e.g., embeddings, clustering, or regression/classification with full-sequence context), bi-directional attention—supplemented by contrastive or task-specific refinement—is essential. For text generation, the loss of explicit autoregressive structure may degrade generation quality, suggesting hybrid architectures that switch masks (causal vs. bidirectional) depending on stage (encode vs. decode) [2510.01652].

## 6. Theoretical and Representational Consequences

From a statistical perspective, bidirectional attention inherits the expressivity of mixture-of-experts routing at every token position, supporting diverse syntactic and semantic generalization [2307.04057]. However, the emergence of classic linear word analogies in the embedding space requires additional symmetry and independence conditions not present in standard CBOW/SGNS models. This explains empirically why BERT/GPT embeddings do not always yield clean analogy structures even though they excel in contextual semantic tasks [2307.04057].

In robotics, bi-directional fusion of BU and TD streams aligns with the free-energy principle, as the system transitions from saliency-driven exploration to top-down goal-driven perception, mediated by mutual information flow in cyclic action-perception loops [2510.10221].

## 7. Open Directions and Hybrid Designs

Recent empirical and theoretical work highlights several design axes for future attention mechanisms:

- Development of attention modules (e.g., DTA) that interpolate between causal and bidirectional inductive biases, combining explicit mask geometry with learned embedding techniques [2604.18603].
- Hybrid mask-switching architectures supporting generation and encoding within a shared backbone [2510.01652].
- Fine-grained fusion of multiple information streams beyond language, such as the simultaneous integration of saliency and prediction in robotics, suggesting further generalization of cross-modal and bi-directional attention paradigms [2510.10221].

The study and continued evolution of bi-directional and uni-directional attention inform the design of more context-sensitive, data-efficient, and robust neural models across multiple domains.

Source: https://www.emergentmind.com/topics/bi-directional-and-uni-directional-attention