---
title: 'AFD: Decoupling Attention and FFN'
url: https://www.emergentmind.com/topics/attention-ffn-disaggregation-afd
type: topic
---

# AFD: Decoupling Attention and FFN

Attention-FFN Disaggregation (AFD) refers to architectural and/or system-level strategies that explicitly decouple the attention and feed-forward network (FFN) components within Transformer-based models or other multi-stage neural architectures. Originally, these two components were tightly coupled within each processing block, but recent research shows that disaggregation enables targeted optimization, efficiency gains, enhanced interpretability, and improved task or system-level performance. Approaches to AFD range from model-internal reparameterizations and pipeline specialization to distributed inference system design, with relevance in vision, language, speech, and system deployment contexts.

## 1. Conceptual Foundation and Motivation

The attention and FFN submodules in Transformers have fundamentally different computational profiles. Attention layers require intensive memory access and key–value (KV) cache management, with relatively fewer parameters, while FFNs, particularly when implemented as (sparse) Mixtures-of-Experts (MoE), are heavily parameter- and compute-bound but stateless across tokens. AFD is motivated by the observation that treating these modules as monolithic blocks results in missed opportunities for hardware specialization, computational redundancy, and sub-optimal model dynamics.

Disaggregation aims to exploit this asymmetry by:

- Scheduling attention and FFN operations on different hardware or subsystems with tailored parallelism and batching strategies.
- Enabling separate feature extraction and processing pipelines for multiple object detection subtasks, as in the case of classification and localization branches [2011.14667].
- Reducing redundancy and resource utilization inefficiency in distributed inference and serving systems by task-specific decomposition [2507.19427, 2503.20552].
- Enhancing modularity for interpretability, debiasing, and expert parameter-sharing [2405.20612, 2505.07260].

## 2. Model-Internal Disaggregation in Few-Shot Object Detection

In adaptive fully-dual network (AFD-Net) for few-shot object detection, disaggregation is directly encoded in the feature-processing and aggregation paths [2011.14667]:

- **Dual Query Encoder (DQE):** Produces separate region-of-interest (RoI) feature vectors $\{r_i^{cls}, r_i^{reg}\}_{i=1}^n = \mathcal{E}(\mathcal{R}(\mathcal{B}(Q)))$, splitting processing for classification and regression.
- **Dual Attention Generator (DAG):** Yields subtask-specific class-attentive support features via convolutional and fully-connected encoders, adaptively fused by learnable weights $\lambda^t_{conv}$, $\lambda^t_{fc}$ for $t \in \{\text{cls}, \text{reg}\}$.
- **Dual Aggregator (DA):** Jointly aggregates these representations for each subtask through separate elementwise operations and FC layers, enhancing task-specific meta-feature activation.
- **Adaptive Fusion Mechanism (AFM):** Fuses information from both task branches to improve feature richness for both classification and localization.

Separate features are used for each subtask, followed by distinct downstream detectors, yielding enhanced performance—especially in data-constrained regimes—demonstrating the utility of disaggregation for task specialization and feature expressiveness.

## 3. System-Level AFD in Distributed Serving and Cost-Efficient Decoding

In large-scale language model serving (e.g., Step-3 [2507.19427] and Adrenaline [2503.20552]), AFD manifests as an explicit systems co-design for throughput and resource utilization:

- **Subsystem Specialization:** The serving architecture places attention computations (memory-intensive, key–value cache–dependent) and FFN/MoE computations (compute-intensive, batch-friendly) on separate pools or groups of GPUs.
- **Pipelined Token Processing:** Each decoded token transits sequentially through specialized hardware stages: attention, (network) communication, and FFN, with interleaved execution to minimize time-per-output-token (TPOT) latency.
- **Mathematical Isolation of Costs:** Decoding cost is split as:  
  $$ \text{Attention Cost} = \max(\text{FLOP}_{\text{Attn}} \times U_{\text{FLOP}},\ \text{Byte}_{\text{KV}} \times U_{\text{byte}}) + \text{FLOP}_{\text{Linear}} \times U_{\text{FLOP}} $$
  $$ \text{FFN Cost} = \text{FLOP}_{\text{FFN}} \times U_{\text{FLOP}} $$
- **Resource Reallocation:** Hardware for attention is selected for memory bandwidth, while FFN hardware is optimized for arithmetic intensity and batch throughput.

Empirical results show that AFD enables Step-3 to reach a decoding throughput of up to 4,039 tokens/s/GPU under a 50 ms TPOT SLA for 4K context, significantly surpassing strong baselines like DeepSeek-V3 and Qwen3 MoE 235B, with only 32–48 GPUs required [2507.19427].

## 4. Analytical and Mathematical Formulations

Attention-FFN Disaggregation is not limited to hardware allocation but is often formalized at the level of architectural design:

| Component            | Mathematical Formulation                                    | Disaggregation Role            |
|----------------------|-------------------------------------------------------------|-------------------------------|
| Dual Query Encoder   | $\{r_i^{cls}, r_i^{reg}\} = \mathcal{E}(\mathcal{R}(\mathcal{B}(Q)))$ | Feature splitting per task     |
| Dual Attention Gen.  | $\{a_j^{cls}, a_j^{reg}\} = \mathcal{G}(\mathcal{B}([S_j, M_j]))$ | Subtask-aware attention        |
| Dual Aggregator      | $r_{i,j}^t = [f_m(r_i^t \odot a_j^t), f_s(r_i^t - a_j^t), r_i^t]$  | Task-specific aggregation      |
| System pipeline cost | $\text{Total Cost} = \text{Attention Cost} + \text{FFN Cost}$      | Resource-targeted deployment   |

This explicit separation in math and compute mapping underpins both the accuracy and throughput improvements observed.

## 5. Architectural and Algorithmic Innovations

AFD enables the implementation of advanced architectural strategies:

- **Expert Parameter Sharing:** In UMoE, attention and FFN modules are unified as a set of shared experts dispatched via a top-$k$ router, leveraging the insight that token mixing (attention weights) and per-token processing (FFN) are amenable to a unified two-matrix structure [2505.07260].
- **Sparsity and Redundancy Control:** Statistical top-$k$ operators enable both FFN and attention layers to realign active computation to a small fraction of tokens/neurons, as in the Spark Transformer [2506.06644].
- **Interpretable Disaggregation for Debiasing:** Masking attention heads and FFN vectors by their contribution to output bias in LLMs enables targeted mitigation strategies without retraining [2405.20612].

## 6. Comparative Performance and Impact

Empirical studies across modalities demonstrate that AFD leads to concrete improvements:

- In object detection, explicit dual-branch processing yields stronger novel-class generalization and allows the effective fusion of class- and localization-specific features, achieving higher mAP, especially in challenging few-shot regimes [2011.14667].
- In distributed LLM inference, Step-3’s AFD underpins new cost-efficiency frontiers (up to 4,039 tokens/s/GPU) by optimizing hardware utilization per stage and context length [2507.19427].
- Under AFD, parameter counts, FLOPs, or inference wall-time may be reduced without sacrificing (and sometimes even improving) predictive performance, as is the case in EfficientASR [2404.19214] and Spark Transformer [2506.06644].

## 7. Broader Implications and Future Research Directions

Attention-FFN Disaggregation represents a paradigm shift from monolithic Transformer design toward modular, task- or hardware-aligned processing and opens several research directions:

- **Adaptive modularity:** Dynamic routing and expert sharing based on task context, token characteristics, or resource profiling.
- **Interpretable control:** Identification and manipulation of internal model components for transparency, debiasing, or reliability without retraining [2405.20612].
- **Scalable system co-design:** Joint optimization of model architecture and serving systems via pipeline parallelism, heterogeneous hardware deployment, and disaggregated compute mapping [2507.19427].
- **Generalization across domains:** Application of AFD principles in vision, language, speech, and multimodal frameworks, leveraging its ability to reconcile efficiency and expressiveness.

Potential limitations include increased system complexity, the need for precise communication scheduling and synchronization, and initial partitioning/prioritization overheads, which require careful empirical tuning and profiling. Nonetheless, the growing body of research demonstrates that AFD is indispensable for achieving state-of-the-art efficiency and flexibility in large-scale model deployment and advanced task specialization.

Source: https://www.emergentmind.com/topics/attention-ffn-disaggregation-afd