QO-Stationary Dataflow for CIM Attention

• QO-stationary dataflow is a strategy that pins query (Q) and output (O) tiles in compute‐in‐memory architectures to maximize on‐chip data reuse.
• It eliminates redundant memory transfers by streaming key and value tiles exactly once, reducing repeated loads by up to 97%.
• The approach fuses operations to cut buffer transpositions and DRAM traffic, yielding nearly 2× speedup and significant energy savings.

A QO-stationary dataflow ("Query-Output-stationary" dataflow) is a data movement and computation scheduling strategy introduced for attention acceleration in operator-fusion-driven compute-in-memory (CIM) architectures. QO-stationary maximizes on-chip data reuse for attention mechanisms by pinning both the query (Q) tile and the corresponding output (O) tile in CIM-local buffer throughout the full inner loop over key (K) and value (V) tiles, while K and V tiles are streamed exactly once. This scheduling eliminates repeated loading of K and V, buffer transpose of $K^\mathrm{T}$, and intermediate buffering in multi-stage attention computation, enabling highly fused and bandwidth-efficient LLM inference at the hardware level (Xuan et al., 28 Apr 2026).

1. Formal Model and Operator Flow

Consider tiled attention with $Q \in \mathbb{R}^{M \times D}$, $K,V \in \mathbb{R}^{N \times D}$, and $O \in \mathbb{R}^{M \times D}$. The matrices are partitioned rowwise: $T_Q$ Q-tiles of shape $m \times D$, $T_K$ K/V-tiles of shape $n \times D$, indexed by $t \in [0, T_Q - 1]$ and $s \in [0, T_K - 1]$. The canonical sequence of operations is:

$m \times D$8

During this schedule:

• Q and O remain stationary on-chip throughout the $Q \in \mathbb{R}^{M \times D}$0 loop.
• K and V are each brought on-chip only once per $Q \in \mathbb{R}^{M \times D}$1, never reloaded for the same Q-tile.
• No explicit buffering or materialization of $Q \in \mathbb{R}^{M \times D}$2, S, or P across the entire tile dimension; data are pipelined immediately between attention subkernels.

2. Elimination of Redundant Transfers and Transpose Access

QO-stationary fundamentally decouples the movement of inputs and weights versus outputs:

• Classical (weight-stationary) attention: Pins K and V in the array, streams Q, which causes every K/V tile to be loaded $Q \in \mathbb{R}^{M \times D}$3 times (once per Q-tile).
• QO-stationary: Pins Q tile and O tile, streams each K/V tile once, eliminating $Q \in \mathbb{R}^{M \times D}$4 redundant loads for each K_s or V_s.

Additionally, QO-stationary natively fuses the transpose for $Q \in \mathbb{R}^{M \times D}$5 by streaming K in bit-serial format through inner-product CIM (IP-CIM), using bitline/wordline remapping to avoid explicit $Q \in \mathbb{R}^{M \times D}$6 materialization in auxiliary buffers. This minimization of buffer accesses is pivotal in energy-bound accelerators.

Softmax and PV (probability-value) sub-operators are pipelined such that S and P buffering is unnecessary. Each local attention score $Q \in \mathbb{R}^{M \times D}$7 (for row $Q \in \mathbb{R}^{M \times D}$8 in Q-tile $Q \in \mathbb{R}^{M \times D}$9 and K-tile $K,V \in \mathbb{R}^{N \times D}$0) immediately enters the Softmax core, and its result $K,V \in \mathbb{R}^{N \times D}$1 is propagated to OP-CIM for $K,V \in \mathbb{R}^{N \times D}$2 accumulation in situ with streaming V_s.

3. Bandwidth, Buffer, and On-Chip Data Reuse Analysis

Quantitative Model

Let $K,V \in \mathbb{R}^{N \times D}$3 be as above, with tile sizes $K,V \in \mathbb{R}^{N \times D}$4 ($K,V \in \mathbb{R}^{N \times D}$5, $K,V \in \mathbb{R}^{N \times D}$6).

• On-chip reuse factor for K/V:

Weight-stationary: $K,V \in \mathbb{R}^{N \times D}$7, each K_s written $K,V \in \mathbb{R}^{N \times D}$8 times. QO-stationary: $K,V \in \mathbb{R}^{N \times D}$9, each K_s written once.

• DRAM traffic reduction for K (similarly for V):

$O \in \mathbb{R}^{M \times D}$0

• Bandwidth savings ratio:

$O \in \mathbb{R}^{M \times D}$1

Empirically, for $O \in \mathbb{R}^{M \times D}$2, this yields a $O \in \mathbb{R}^{M \times D}$3 reduction in repeated K (and V) loads, excluding the mandatory first pass. Including partial output traffic, the reported reduction in on-chip data movement is $O \in \mathbb{R}^{M \times D}$4 (Fig. 7 in (Xuan et al., 28 Apr 2026)).

4. Integration with Hybrid IP-CIM/OP-CIM Pipeline

The QO-stationary dataflow is implemented in architectures such as FusionCIM via a hybrid pipeline:

• IP-CIM array: Holds the stationary Q-tile, streams K-tiles in, computes QK$O \in \mathbb{R}^{M \times D}$5.
• Softmax core: Receives per-row $O \in \mathbb{R}^{M \times D}$6 scores in bit-serial fashion, performs pattern-aware online Softmax with opportunistic partial normalization given early row-max detection.
• OP-CIM array: Holds the stationary O-tile, receives P and V streams, accumulates PV in situ.

The entire $O \in \mathbb{R}^{M \times D}$7-indexed workflow is scheduled in parallel across hardware engines (HEs), with each HE loading and pinning Q and O, streaming through all K and V once, issuing results to global buffer at the end of each Q-tile.

Pattern-aware K/V tile scheduling enables the Softmax unit to often observe the maximum score earlier in the pipeline, reducing total exponentiation and rescale events by up to $O \in \mathbb{R}^{M \times D}$8 (Fig. 9). In QO-stationary, this directly reduces critical path latency.

5. Comparative Analysis with Classical Dataflow Schemes

A synthesis of classical attention-focused dataflows:

Dataflow	Stationary On-Chip	Streamed Data	Tile Write Count (K/V)	Tile Write Count (Q)
Weight-stationary	K, V in array/CM	Q, O	$O \in \mathbb{R}^{M \times D}$9	$T_Q$0
Output-stationary	O (partial sums)	Q, K, V	$T_Q$1	$T_Q$2
Row-stationary	Mixed (row granularity)	Mixed	sub-optimal	sub-optimal
QO-stationary	Q, O	K, V	$T_Q$3	$T_Q$4

QO-stationary is uniquely suited for tandem-fused attention, in which $T_Q$5 and $T_Q$6 occur sequentially with an in-the-loop non-linearity (Softmax). It achieves maximal stationary reuse for both the input Q tile (across all K/V tiles) and for the output O tile (across all partial-V contributions), with the minimal required load and store events.

6. Performance Characteristics: Latency, Throughput, and Energy

Time components per tile/operation:

• $T_Q$7: QK$T_Q$8 tile on IP-CIM
• $T_Q$9: Softmax (8-cycle Taylor series pipeline)
• $m \times D$0: PV tile on OP-CIM
• $m \times D$1: Stream K or V tile
• KV-stationary baseline:

Each (t, s) cycle incurs $m \times D$2 repeated loads for Q and K, as well as O.

• QO-stationary:

Each Q-tile is loaded once, each O-tile written once, each K and V streamed through a single time per tile, with inner loop fusion.

The modeled speedup is nearly 2$m \times D$3 for large $m \times D$4, with measured results at $m \times D$5 speedup for sequence length 4K tokens (LLaMA-3), and energy savings of $m \times D$6 over prior state-of-the-art (Xuan et al., 28 Apr 2026). DRAM/global-buffer activity is reduced by $m \times D$7.

7. Significance and Application Context

QO-stationary dataflow directly addresses the memory, bandwidth, and in-situ fusion challenges of large-scale attention in LLM inference. By constraining repeated access to DRAM and minimizing on-chip transfers for K and V, it enables CIM-based hardware to approach the arithmetic-to-bandwidth ratio dictated by attention's dual-GEMM structure and softmax interleaving. QO-stationary also forms the basis for further pipeline-level fusions, such as pattern-aware Softmax, which gain amplified benefit under this maximally reuse-centric schedule.

In sum, QO-stationary is the centerpiece of FusionCIM attention acceleration, providing the on-chip data reuse and operator fusion necessary for near-ideal hardware scaling in bandwidth-constrained scenarios, as evidenced empirically in power, throughput, and efficiency for multi-thousand-token LLM inference workloads (Xuan et al., 28 Apr 2026).