Bi- and Uni-Directional Attention in Neural Models

• Bi-directional attention is a mechanism that allows each token to attend to all others in a sequence, enabling full-context inference for tasks like masked language modeling.
• Uni-directional attention restricts token interactions to past inputs with a causal mask, ensuring explicit ordering for autoregressive generation.
• Advanced variants like Dual Triangle Attention merge past and future cues by splitting attention heads, optimizing performance across diverse domains.

Bi-directional and uni-directional attention are foundational mechanisms in modern neural architectures for sequence modeling, notably in NLP, biological, and robotic domains. Uni-directional (causal) attention restricts each token to attend only to its past and self, thus imposing an inherent ordering compatible with autoregressive generation. Bi-directional attention, in contrast, allows each token to access the full context on both sides, supporting tasks such as masked language modeling and semantic probing that benefit from full-context inference. Recent research rigorously characterizes their mathematical structures, comparative representational properties, and applications in diverse domains, as well as provides new insights into their inductive biases and computational trade-offs.

1. Formal Definitions and Core Mechanisms

Uni-directional (Causal) Attention

Uni-directional attention enforces a lower-triangular masking scheme. For a sequence of length $n$, the causal mask $M\in\mathbb{R}^{n\times n}$ is defined as: $$M_{ij} = \begin{cases} 0, & j \le i \ -\infty, & j > i \end{cases}$$ Self-attention with this mask (cf. Eq. (2) in (Feng et al., 2 Oct 2025)) is given by: $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V$$ This structure restricts each position $i$ to attend only to positions up to $i$, supporting left-to-right generation and encoding explicit ordinal information via the mask’s triangular structure, even without explicit positional embeddings (Halle et al., 9 Apr 2026).

Bi-directional (Full-context) Attention

In bi-directional attention (e.g., BERT-style), every token potentially attends to every other, using an all-zeros mask: $M'_{ij} = 0\quad \forall \,i,j$ This enables access to both left and right contexts within each attention layer, supporting tasks requiring full-context inference such as MLM, semantic similarity, or probing tasks (Feng et al., 2 Oct 2025, Wibisono et al., 2023).

2. Bidirectional Attention: Architectural and Statistical Characterizations

Bidirectional attention can be formalized as a statistical mixture-of-experts (MoE) system. "Bidirectional Attention as a Mixture of Continuous Word Experts" establishes that a single-layer, single-head bidirectional self-attention model with positional encodings and MLM is mathematically equivalent to a continuous bag-of-words (CBOW) model, but with soft, data-dependent MoE weights: $$\text{logit}_k = \sum_{j=1}^S \pi_j(\bar{X}) f_j(\bar{X})_k$$ where the mixture weights $\pi_j$ and experts $f_j$ are learned from token+position embeddings, and the cross-entropy loss coincides with a weighted MoE-variant of CBOW (Wibisono et al., 2023).

With multiple heads/layers, bidirectional attention corresponds to stacked and deep mixtures of MoEs, each allowing tokens to aggregate information from all positions with data-driven expert routing.

Dual Triangle Attention (DTA)

Dual Triangle Attention (DTA) introduces an architectural variant that splits each head into two sub-heads—one with a past-and-self mask, another with a future-and-self mask. Each sub-head computes directional attention along its mask (lower- or upper-triangular), restoring the positional inductive bias of causal attention in both directions without requiring additional learned parameters or positional embeddings. This maintains efficient context aggregation and strong performance in both NLP and bioinformatics sequence tasks (Halle et al., 9 Apr 2026).

3. Contextual and Developmental Properties

Information Flow and Inductive Bias

Causal masking supplies an ordinal signal that enables Transformers to recover absolute position information without positional embeddings. Standard (fully bidirectional) self-attention is permutation-invariant in the absence of position encodings, removing any implicit order. DTA restores a coarse positional prior via mask geometry, complementary to mechanisms such as Rotary Positional Embeddings (RoPE) (Halle et al., 9 Apr 2026).

Task-driven Dynamic Reweighting (Robotics and BU/TD Fusion)

In developmental robotics, bi-directional attention can fuse bottom-up (BU, saliency-driven) and top-down (TD, prediction-driven) streams. In A³RNN, BU saliency maps and TD queries are fused via cross-attention, supporting a developmental trajectory from exploration (BU-driven) to exploitation (TD-driven). This bi-directional loop enables mutual refinement, producing more coherent and robust attention patterns compared to uni-directional approaches (Hiruma et al., 11 Oct 2025).

Setting	Mechanism	Information Flow
NLP (BERT/LLM2Vec)	Bidirectional Self-Attention (+MNTP)	Any↔Any (Full context)
Robotics (A³RNN)	Cross-attn fusion (BU↔TD)	BU→TD and TD→BU
Causal Transformers	Uni-directional Mask	Left-context only

4. Empirical Results and Practical Implications

Extensive probing and masked modeling results establish several key findings:

• Uni-directional LLMs cannot faithfully encode left-context information for embedding/probing tasks; BERT-Base achieves 94.2% on left-context subject-animacy, while causal Llama yields only 54.7% (Feng et al., 2 Oct 2025).
• Simple bidirectional mask removal ("Bi + MNTP") in decoder-only models recovers right-context but trades off left-context; full context is only restored by contrastive fine-tuning (SimCSE or NLI-supervised) (Feng et al., 2 Oct 2025).
• DTA matches or slightly exceeds standard bidirectional attention on long-context and MLM benchmarks, while requiring no positional embeddings and maintaining efficient parameterization (Halle et al., 9 Apr 2026).
• In tabular data, the Mixture-of-Experts view enables an extension ("ATN") that outperforms transformer-based tabular baselines under distribution shift conditions (Wibisono et al., 2023).
• In developmental robotics, bi-directional (BU ↔ TD) fusion yields more stable and interpretable trajectories, with cross-attention supporting adaptive refinement not recoverable under BU→TD or TD→BU alone (Hiruma et al., 11 Oct 2025).

5. Computational and Design Trade-Offs

Bidirectional attention, by removing the causal mask, increases the number of nonzero dot-products in each head from $M\in\mathbb{R}^{n\times n}$0 to $M\in\mathbb{R}^{n\times n}$1, yielding approximately $M\in\mathbb{R}^{n\times n}$2 the FLOPs and memory costs for the attention computation and gradients (Feng et al., 2 Oct 2025). Inference latency grows by a similar factor, though optimized kernels (e.g., FlashAttention, flex_attention) may mitigate this penalty (Halle et al., 9 Apr 2026). DTA offers hardware-efficient, parameter-neutral bidirectional attention via fused kernel implementation.

A plausible implication is that for applications requiring full-context encoding (e.g., embeddings, clustering, or regression/classification with full-sequence context), bi-directional attention—supplemented by contrastive or task-specific refinement—is essential. For text generation, the loss of explicit autoregressive structure may degrade generation quality, suggesting hybrid architectures that switch masks (causal vs. bidirectional) depending on stage (encode vs. decode) (Feng et al., 2 Oct 2025).

6. Theoretical and Representational Consequences

From a statistical perspective, bidirectional attention inherits the expressivity of mixture-of-experts routing at every token position, supporting diverse syntactic and semantic generalization (Wibisono et al., 2023). However, the emergence of classic linear word analogies in the embedding space requires additional symmetry and independence conditions not present in standard CBOW/SGNS models. This explains empirically why BERT/GPT embeddings do not always yield clean analogy structures even though they excel in contextual semantic tasks (Wibisono et al., 2023).

In robotics, bi-directional fusion of BU and TD streams aligns with the free-energy principle, as the system transitions from saliency-driven exploration to top-down goal-driven perception, mediated by mutual information flow in cyclic action-perception loops (Hiruma et al., 11 Oct 2025).

7. Open Directions and Hybrid Designs

Recent empirical and theoretical work highlights several design axes for future attention mechanisms:

• Development of attention modules (e.g., DTA) that interpolate between causal and bidirectional inductive biases, combining explicit mask geometry with learned embedding techniques (Halle et al., 9 Apr 2026).
• Hybrid mask-switching architectures supporting generation and encoding within a shared backbone (Feng et al., 2 Oct 2025).
• Fine-grained fusion of multiple information streams beyond language, such as the simultaneous integration of saliency and prediction in robotics, suggesting further generalization of cross-modal and bi-directional attention paradigms (Hiruma et al., 11 Oct 2025).

The study and continued evolution of bi-directional and uni-directional attention inform the design of more context-sensitive, data-efficient, and robust neural models across multiple domains.