---
title: 'MoDA: Mixture-of-Depths Attention in Transformers'
url: https://www.emergentmind.com/topics/mixture-of-depths-attention-moda
type: topic
---

# MoDA: Mixture-of-Depths Attention in Transformers

Mixture-of-Depths Attention (MoDA) is a class of architectural and algorithmic innovations for transformer models that integrates dynamic, data-driven compute allocation along the depth (layer) axis, aiming to improve computational efficiency, scalability, and representational fidelity especially as models become deeper or handle multimodal, long-sequence, or complex reasoning tasks. MoDA encompasses both token-sparse routing—where tokens are adaptively processed at select layers—and depth-aware attention, where attention heads retrieve information from activations across multiple layers. These mechanisms address the inefficiencies and information bottlenecks inherent in dense, fixed-depth transformer architectures. MoDA is now central to the efficient scaling of both unimodal and multimodal transformers, as evidenced by recent advances in unified large models and depth-recurrent architectures.

## 1. Core Mechanisms and Formulations

The fundamental principle of Mixture-of-Depths Attention is to decouple the notion that all tokens must be processed in all layers, and, in recent depth-aware variants, to enable each attention head to attend over both the current layer’s and preceding layers’ key-value pairs.

### Token Routing and Gating

In classic MoDA routing, at each transformer layer, a lightweight router network computes a per-token importance score. Tokens with high scores are processed by the full attention and feed-forward subnet; others are routed via the residual connection only, bypassing heavy computation. The routing mechanism is typically implemented as:
\[
r_i = \sigma(W_r x_i + b_r)
\]
where \(x_i\) is the token embedding and \(r_i\) is its scalar importance score. A hard threshold or percentile-selection (top-k or threshold-p) is used to decide which tokens to process:
\[
\tilde x_i = \begin{cases}
x_i + D(x_i), & r_i \geq o_s \\
x_i,          & r_i < o_s
\end{cases}
\]
where \(D(\cdot)\) denotes the attention+FFN block [2502.06474, 2404.02258, 2412.04449, 2410.14268].

### Depth-Aware Attention

Recent MoDA variants enable direct attention over intermediate representations from multiple preceding layers, augmenting sequence attention with “depth” attention [2603.15619, 2601.21582]. At layer \(l\), each head jointly attends to the current sequence and to a set of depth-indexed key-value memories. This is formalized as:
\[
\mathrm{MoDA}_h(Q_h^l, K_{\text{all}}, V_{\text{all}}) = \mathrm{softmax}\left(\frac{Q_h^l K_{\text{all}}^\top}{\sqrt{d}} + \mathcal{M}\right) V_{\text{all}}
\]
where \(K_{\text{all}}\) and \(V_{\text{all}}\) concatenate current and all previous layer KVs, and \(\mathcal{M}\) masks for causality and validity.

## 2. Architectural Variants and Mathematical Details

MoDA encompasses several design families, including but not limited to:

| Variant/Family        | Routing Principle         | Depth Mixing             |
|----------------------|--------------------------|--------------------------|
| Token Pruning MoDA   | Per-token router (MLP/linear) selects tokens to process per layer [2404.02258, 2502.06474] | No                     |
| Attention-based MoDA (A-MoD) | Uses normalized attention scores from previous layer as routing scores (no extra parameters) [2412.20875] | No                |
| Threshold-p Gating   | Replaces fixed top-k with adaptive thresholding for better effectiveness [2410.14268] | No                  |
| Depth-aware MoDA     | Every attention head accesses activations from all or a subset of previous layers (depth-key/value) with a unified softmax [2603.15619, 2601.21582] | Yes                 |
| Multimodal MoDA      | Separate routers or schedules for vision and language tokens, task- or modality-aware [2502.06474, 2412.04449, 2410.13859] | Optional            |

*“A-MoD” and “MoDA” are sometimes used synonymously for attention-routed token pruning, but recent works in depth-recurrent architectures [2603.15619, 2601.21582] use MoDA to mean genuine depth attention.*

### Layer- and Token-level Routing

- **Top-k Routing:** Selects exactly \(k\) tokens per layer via scoring; enables static computation graphs, but is rigid [2404.02258].
- **Threshold-p Routing:** Dynamically determines, per layer, how many tokens are kept based on a score threshold so the retained fraction is ≈\(p\); reduces routing artifacts and matches input heterogeneity [2410.14268].
- **Task-/Modality-Aware Routers:** Separate scoring networks/thresholds per data source or objective, exploiting heterogeneity in token redundancy [2502.06474, 2412.04449, 2410.13859].

### Depth Attention and Recurrent Mixtures

- **Full Depth Attention:** Each attention head can mix information from all prior layer activations for its token position, dynamically retrieving the most relevant depth [2603.15619].
- **Hybrid Mixture-of-Attentions:** Modular stacking of sequence attention, depth attention, and sparse MoE expert attention in a single block; enables cross-depth retrieval and capacity scaling [2601.21582].

## 3. Efficiency, Scalability, and Hardware Considerations

MoDA achieves substantial reductions in computational cost, GPU memory, and wall-clock training/inference time by bypassing redundant computations at both the token and layer level.

- **Token skipping** yields FLOPs and memory savings scaling with the skipped fraction. In UniMoD [2502.06474], 15–40% training/inference FLOPs are saved with negligible or even improved benchmark accuracy; in p-MoD, up to 44% reduction in inference FLOPs and 54% lower KV cache usage is achieved in multimodal decoders [2412.04449].
- **A-MoD** (attention-driven routing) introduces zero additional parameters and negligible overhead beyond leveraging already-computed attention matrices; this not only simplifies implementation but also accelerates convergence (2× faster in transfer settings) and can yield up to 2% accuracy improvements at equal FLOPs compared to standard MoD [2412.20875].
- **Depth-attentive MoDA** incurs only a moderate computational overhead (+3.7% FLOPs for 1.5B LLMs) while improving average perplexity by 0.2 and increasing downstream accuracy by 2.11% [2603.15619].
- **Hardware-efficient deployments** rely on custom memory layouts, chunk-aware indexing, and group query attention packing to achieve over 97% of vanilla FlashAttention-2 throughput even for 64K sequence length [2603.15619].

Token skipping and depth attention together allow both FLOP budget predictability and adaptive, data-dependent compute allocation. These characteristics are crucial for scaling to deeper architectures and longer sequences.

## 4. Empirical Findings and Applications

### Performance and Efficiency Metrics

- On unified multimodal benchmarks (Show-o, Emu3), task-aware MoDA layers in UniMoD maintain or improve accuracy with 15–40% fewer FLOPs. For example, Emu3's MMU benchmark rises from 881.3→901.0 score at 40% lower FLOPs [2502.06474].
- In computer vision, DeiT/ViT models with A-MoD routed pruning attain up to +2% accuracy over baselines at matched FLOPs and require half the epochs to reach peak transfer learning accuracy [2412.20875].
- Multimodal LLMs using progressive retention schedules (p-MoD) halve FLOPs and cache with no accuracy loss across 14 tasks [2412.04449].
- Depth-wise attentive LLMs (MoDA, Dreamer) exhibit robust gains in language modeling and downstream reasoning, requiring up to 8× fewer training tokens for equivalent accuracy (e.g., +11–19 points improvement in 0-shot math reasoning vs matched LA baselines) [2601.21582, 2603.15619].
- γ-MoD demonstrates over 50% speedup in inference and training with <2% drop in accuracy by converting 80–90% of layers to MoD blocks using the ARank redundancy metric [2410.13859].

### Token/Layer Redundancy and Routing Insights

- Token redundancy is highly layer- and task-dependent; generation tasks exhibit higher ARank (lower redundancy), while understanding tasks allow more aggressive pruning.
- Skipping early layers dramatically harms accuracy, confirming their representational importance; redundancy generally grows with depth [2502.06474].
- Depth-attentive heads display diverse attention distributions over sequence and depth slots, dynamically retrieving contextually useful features and mitigating attention sinks [2603.15619].

## 5. Practical Deployment and Tuning

### Integration Procedures

- MoDA and its attention-based variant (A-MoD) can be integrated into pretrained transformers without architecture retraining. Token routing uses either newly-trained lightweight routers, attention-based scoring, or shared routers tuned with small auxiliary losses [2412.20875, 2410.13859].
- Layer selection for MoD conversion is optimized via redundancy metrics such as ARank [2410.13859, 2502.06474]. Typically, the majority of intermediate layers can be safely converted to MoD variants.
- In multimodal transformers, per-task or per-modality routers decouple redundancy schedules, preserving cross-task performance and stability [2502.06474]. Auxiliary losses enforce global or per-layer compute budgets.

### Optimization Details

- Choice of skip ratio, thresholding method (top-k vs threshold-p), and gating normalization (e.g., TanhNorm) is critical. Optimal settings may include interleaving routed with dense layers and scheduling layer-wise retention via shifted-cosine or learned profiles [2412.04449, 2410.14268].
- Shared routers with masking (γ-MoD) are preferred for stability when retrofitting to large MLLMs [2410.13859].

## 6. Connections to Related Frameworks

- MoDA is distinct from Mixture-of-Experts (MoE): MoDA routes tokens along the depth axis, skipping entire blocks, whereas MoE routes tokens to different experts for MLP but executes full attention/MLP for all tokens [2412.04449].
- Depth attention generalizes DenseNet-style cross-layer concatenation, achieving adaptation and feature retrieval with linear, rather than quadratic, overhead [2603.15619].

## 7. Future Directions and Outlook

- **Hardware and Scaling:** Further optimization of CUDA kernels, fused depth/sequence attention, and advanced memory management are anticipated to extend MoDA to trillion-parameter and ultra-long-context regimes [2603.15619].
- **Architectural Extensions:** MoDA's depth-aware principles may generalize to cross-modal retrieval (vision, speech), bounded-sliding depth memories, or hybrid data- and depth-based retrieval [2603.15619].
- **Theoretical Questions:** Investigating the optimal balance of depth/sequence attention and expressivity-redundancy tradeoffs as depth increases, as well as robust redundancy metrics for task-adaptive MoDA scheduling, remains an open area.

MoDA establishes a new paradigm in efficient transformer design: by merging depth-aware computation and token-adaptive pruning, it enables deeper, more effective, and computationally tractable models across domains and modalities [2603.15619, 2502.06474, 2412.20875, 2412.04449, 2601.21582, 2410.13859, 2410.14268, 2404.02258].

Source: https://www.emergentmind.com/topics/mixture-of-depths-attention-moda