Core Attention Disaggregation (CAD)

• CAD is a scalable technique that decouples the quadratic softmax(QK^T)V computation from the rest of the model to enable efficient long-context LLM training.
• It partitions the core attention into CA-tasks that are dynamically scheduled across dedicated GPU servers, ensuring balanced compute and memory utilization.
• Implemented in systems like DistCA, CAD has demonstrated up to 1.35× improved training throughput on extremely long sequences and large GPU clusters.

Core Attention Disaggregation (CAD) is a technique that decouples the computation of the core attention mechanism—specifically softmax($QK^{\mathrm{T}}$)%%%%1%%%%—from the remainder of the model graph, enabling more efficient and scalable training for long-context LLMs. This separation addresses the major challenge of quadratic compute scaling in core attention compared to the near-linear scaling of other model operations, such as feed-forward and token-wise projection layers. By abstracting core attention as stateless and composable, CAD transforms an otherwise monolithic computation into a set of balanced, parallelizable tasks, substantially enhancing memory and compute utilization in multi-GPU environments. The technique is instantiated in systems like DistCA, which have been shown to improve training throughput by up to 1.35× on extremely long sequences and large GPU clusters (Zhuang et al., 20 Oct 2025).

1. Fundamental Principles of CAD

Traditional transformer training co-locates all layer computations—including core attention—on the same device. While this is effective for moderate context lengths, it becomes suboptimal when dealing with highly variable and long input sequences. The quadratic growth of the core attention operation $O(N^2)$ (where $N$ is sequence length) results in load imbalance across data and pipeline parallel groups, creating "stragglers" that slow global training.

Core Attention Disaggregation is predicated on two technical observations:

• Statelessness: Core attention is parameter-free and only requires minimal transient state, such as per-row softmax statistics. This differentiates it from layers with persistent parameters and heavy activations.
• Composability: Modern high-efficiency attention kernels (e.g., FlashAttention) maintain performance even when processing fused batches of token-level shards of arbitrary lengths.

Thus, CAD partitions the core attention workload into individual tasks called "CA-tasks," each corresponding to attention over specific token shards, and dispatches them to a pool of dedicated attention servers. These can be GPU nodes time-sharing their role between both memory-intensive and compute-intensive operations.

2. CAD Implementation: DistCA System Architecture

DistCA exemplifies the CAD paradigm. The processing flow can be summarized as follows:

1. Context-independent layers (projection, normalization, feed-forward) operate as usual on the model's input or previous activations.
2. Partitioning: The output after these layers is segmented into CA-tasks by token count, with shards sized to kernel tile multiples (e.g., 128 tokens).
3. Dispatching: A dynamic scheduler balances CA-tasks across a pool of attention servers, which may be physical or virtual GPU partitions.
4. Attention Servers: Each server receives a batch of CA-tasks. Modern fused-kernel implementations allow arbitrary-length shards to be processed together efficiently.
5. Ping-pong execution: Microbatches are split into two nano-batches ("Ping" and "Pong"); while one is computed, communication of the other’s queries/keys/values is pipelined to overlap with computation, minimizing additional latency.

DistCA supports in-place execution, meaning a device alternates between serving context-independent layer computation and core attention, thereby optimizing memory utilization for very long context training.

3. Load Balancing, Scheduling, and Communication

CAD offers explicit mechanisms for balancing both compute and memory. Straggler elimination is achieved by dynamically re-batching CA-tasks so that total FLOPs per server are nearly equalized. The system is agnostic to document packing variability in the data loader; sequences can be arbitrarily long or short.

Mathematically, the compute per token is:

$$\mathrm{FLOPs}(l) = \alpha \cdot l^2 + \beta \cdot l,$$

where $\alpha$ corresponds to core attention and $\beta$ describes context-independent layers. By extracting and externally scheduling the $\alpha l^2$ term, load imbalance driven by variable sequence length is eliminated.

Communication between worker GPUs and attention servers is hidden by dual nano-batch pipelining—shown in upper bound analyses such as

$s \leq 2 \frac{(tB-h_q)}{h_\mathrm{kv}} - 1$

where $t$ is per-token compute time, $B$ is bandwidth, and $h_q$, $h_\mathrm{kv}$ are header sizes.

4. Performance and Scalability

Quantitative results on up to 512 H200 GPUs and context lengths of up to 512,000 tokens indicate that CAD (as implemented in DistCA) yields up to 1.35× faster end-to-end training throughput (Zhuang et al., 20 Oct 2025). The system achieves near-perfect balance between compute and memory across replica groups—a significant advance for the efficient scaling of long-context LLM training.

Elimination of stragglers and dynamic partitioning of CA-tasks yield stable per-step durations even under document packing strategies that would otherwise introduce substantial variance into microbatch completion times. By maintaining activation footprint locality, DistCA also avoids memory divergence that arises in models co-locating all layer computations.

5. Technical Formulation and Task Partitioning

The operation performed by the disaggregated CA-task is the standard attention computation:

$$\mathbf{O} = \operatorname{softmax}(\mathbf{Q}\mathbf{K}^{\mathrm{T}}) \mathbf{V}$$

for each shard. Explicit separation yields advantages beyond load balancing: communication is limited to $Q$, $K$, and $V$ tensors; there is no need to transmit gradients or parameter states. The composable property ensures that, for a batch of documents with varying lengths, token-wise outputs can be sequentially or concurrently processed and re-fused as needed.

CA-tasks may be dynamically scheduled or fused into composite batches, further reducing the risk of under-utilization and maximizing kernel occupancy.

6. Implications and Future Directions

Core Attention Disaggregation (CAD) establishes a scalable framework for long-context LLM training. Its stateless, composable nature enables modular parallelism that overcomes deep scaling challenges inherent to traditional transformer architectures. It is particularly well-suited for contemporary workloads where context lengths routinely exceed 100k tokens and compute clusters comprise hundreds of heterogeneous GPUs.

A plausible implication is that CAD principles can be extended to other quadratic-cost operator scenarios in deep learning (e.g., cross-attention in mixture-of-experts models, or spatial attention in dense vision networks). As the approach requires only minimal modifications to kernel dispatch and scheduling (rather than substantial architectural revision), it provides a practical path to achieve balanced and efficient resource utilization in distributed model training.

Scaling further may require investigation into more granular task sharding, network topology-aware scheduling, and adaptive fusion algorithms as context length and batch size variability continue to increase in future generations of large models.