---
title: QO-Stationary Dataflow for CIM Attention
url: https://www.emergentmind.com/topics/qo-stationary-dataflow-strategy
type: topic
---

# QO-Stationary Dataflow for CIM Attention

A QO-stationary dataflow ("Query-Output-stationary" dataflow) is a data movement and computation scheduling strategy introduced for attention acceleration in operator-fusion-driven compute-in-memory (CIM) architectures. QO-stationary maximizes on-chip data reuse for attention mechanisms by pinning both the query (Q) tile and the corresponding output (O) tile in CIM-local buffer throughout the full inner loop over key (K) and value (V) tiles, while K and V tiles are streamed exactly once. This scheduling eliminates repeated loading of K and V, buffer transpose of $K^\mathrm{T}$, and intermediate buffering in multi-stage attention computation, enabling highly fused and bandwidth-efficient LLM inference at the hardware level [2604.25317].

## 1. Formal Model and Operator Flow

Consider tiled attention with $Q \in \mathbb{R}^{M \times D}$, $K,V \in \mathbb{R}^{N \times D}$, and $O \in \mathbb{R}^{M \times D}$. The matrices are partitioned rowwise: $T_Q$ Q-tiles of shape $m \times D$, $T_K$ K/V-tiles of shape $n \times D$, indexed by $t \in [0, T_Q - 1]$ and $s \in [0, T_K - 1]$. The canonical sequence of operations is:

```
for t = 0 .. T_Q-1:
    load Q_t once
    initialize O_t ← 0
    for s = 0 .. T_K-1:
        stream K_s → compute S_{t,s} = Q_t K_s^T
        S_{t,s} → rowwise SoftMax → P_{t,s}
        stream V_s and P_{t,s} → accumulate O_t += P_{t,s} V_s
    write back O_t
```

During this schedule:
- Q and O remain stationary on-chip throughout the $s$ loop.
- K and V are each brought on-chip only once per $s$, never reloaded for the same Q-tile.
- No explicit buffering or materialization of $K^T$, S, or P across the entire tile dimension; data are pipelined immediately between attention subkernels.

## 2. Elimination of Redundant Transfers and Transpose Access

QO-stationary fundamentally decouples the movement of inputs and weights versus outputs:
- **Classical (weight-stationary) attention:** Pins K and V in the array, streams Q, which causes every K/V tile to be loaded $T_Q$ times (once per Q-tile).
- **QO-stationary:** Pins Q tile and O tile, streams each K/V tile once, eliminating $T_Q - 1$ redundant loads for each K_s or V_s.

Additionally, QO-stationary natively fuses the transpose for $Q K^T$ by streaming K in bit-serial format through inner-product CIM (IP-CIM), using bitline/wordline remapping to avoid explicit $K^T$ materialization in auxiliary buffers. This minimization of buffer accesses is pivotal in energy-bound accelerators.

Softmax and PV (probability-value) sub-operators are pipelined such that S and P buffering is unnecessary. Each local attention score $S_{t,s}[i,:]$ (for row $i$ in Q-tile $t$ and K-tile $s$) immediately enters the Softmax core, and its result $P_{t,s}[i,:]$ is propagated to OP-CIM for $O_t$ accumulation in situ with streaming V_s.

## 3. Bandwidth, Buffer, and On-Chip Data Reuse Analysis

### Quantitative Model

Let $M, N, D$ be as above, with tile sizes $m, n$ ($T_Q = M/m$, $T_K = N/n$).

- **On-chip reuse factor for K/V:**  
  Weight-stationary: $R_K = 1$, each K_s written $T_Q$ times.  
  QO-stationary: $R_K = T_Q$, each K_s written once.

- **DRAM traffic reduction for K (similarly for V):**
  $$
  \begin{align*}
    \text{Without QO}:&\, T_Q \times T_K = \frac{M}{m} \frac{N}{n} \\
    \text{With QO}:&\, T_K = \frac{N}{n} \\
    \text{Traffic reduction:}&\, (T_Q-1) \times T_K = \left(\frac{M}{m}-1\right)\frac{N}{n}
  \end{align*}
  $$

- **Bandwidth savings ratio:**  
  $BW_{savings} = 1 - \frac{1}{T_Q} = 1-\frac{m}{M}$

Empirically, for $T_Q \approx 32$, this yields a $\sim 97\%$ reduction in repeated K (and V) loads, excluding the mandatory first pass. Including partial output traffic, the reported reduction in on-chip data movement is $64.3\text{–}71.0\%$ (Fig. 7 in [2604.25317]).

## 4. Integration with Hybrid IP-CIM/OP-CIM Pipeline

The QO-stationary dataflow is implemented in architectures such as FusionCIM via a hybrid pipeline:

- **IP-CIM array:** Holds the stationary Q-tile, streams K-tiles in, computes QK$^{\text{T}}$.
- **Softmax core:** Receives per-row $S_{t,s}$ scores in bit-serial fashion, performs pattern-aware online Softmax with opportunistic partial normalization given early row-max detection.
- **OP-CIM array:** Holds the stationary O-tile, receives P and V streams, accumulates PV in situ.

The entire $t$-indexed workflow is scheduled in parallel across hardware engines (HEs), with each HE loading and pinning Q and O, streaming through all K and V once, issuing results to global buffer at the end of each Q-tile.

Pattern-aware K/V tile scheduling enables the Softmax unit to often observe the maximum score earlier in the pipeline, reducing total exponentiation and rescale events by up to $61.4\%$ (Fig. 9). In QO-stationary, this directly reduces critical path latency.

## 5. Comparative Analysis with Classical Dataflow Schemes

A synthesis of classical attention-focused dataflows:

| Dataflow       | Stationary On-Chip         | Streamed Data           | Tile Write Count (K/V) | Tile Write Count (Q) |
|----------------|---------------------------|-------------------------|------------------------|----------------------|
| Weight-stationary   | K, V in array/CM | Q, O                     | $T_Q$                  | $T_K$                |
| Output-stationary   | O (partial sums)      | Q, K, V                 | $T_Q$                  | $T_K$                |
| Row-stationary      | Mixed (row granularity) | Mixed                   | sub-optimal            | sub-optimal          |
| QO-stationary       | Q, O                  | K, V                    | $1$                    | $T_K$                |

QO-stationary is uniquely suited for tandem-fused attention, in which $QK^T$ and $PV$ occur sequentially with an in-the-loop non-linearity (Softmax). It achieves maximal stationary reuse for both the input Q tile (across all K/V tiles) and for the output O tile (across all partial-V contributions), with the minimal required load and store events.

## 6. Performance Characteristics: Latency, Throughput, and Energy

Time components per tile/operation:
- $T_{IP}$: QK$^{\text{T}}$ tile on IP-CIM
- $T_{SM}$: Softmax (8-cycle Taylor series pipeline)
- $T_{OP}$: PV tile on OP-CIM
- $T_{mem}$: Stream K or V tile

- **KV-stationary baseline:**  
  Each (t, s) cycle incurs $T_Q \times T_K$ repeated loads for Q and K, as well as O.

- **QO-stationary:**  
  Each Q-tile is loaded once, each O-tile written once, each K and V streamed through a single time per tile, with inner loop fusion.

The modeled speedup is nearly 2$\times$ for large $T_Q$, with measured results at $1.98\times$ speedup for sequence length 4K tokens (LLaMA-3), and energy savings of $3.86\times$ over prior state-of-the-art [2604.25317]. DRAM/global-buffer activity is reduced by $46\text{–}59\%$.

## 7. Significance and Application Context

QO-stationary dataflow directly addresses the memory, bandwidth, and in-situ fusion challenges of large-scale attention in LLM inference. By constraining repeated access to DRAM and minimizing on-chip transfers for K and V, it enables CIM-based hardware to approach the arithmetic-to-bandwidth ratio dictated by attention's dual-GEMM structure and softmax interleaving. QO-stationary also forms the basis for further pipeline-level fusions, such as pattern-aware Softmax, which gain amplified benefit under this maximally reuse-centric schedule.

In sum, QO-stationary is the centerpiece of FusionCIM attention acceleration, providing the on-chip data reuse and operator fusion necessary for near-ideal hardware scaling in bandwidth-constrained scenarios, as evidenced empirically in power, throughput, and efficiency for multi-thousand-token LLM inference workloads [2604.25317].

Source: https://www.emergentmind.com/topics/qo-stationary-dataflow-strategy