FlashAttention for Scalable Vector Architectures

Abstract: Inference with transformer models on CPUs is increasingly important, especially for Small LLMs (SLMs), where vector architectures are emerging as a promising execution substrate. The attention module is a major bottleneck due to high memory bandwidth requirements; FlashAttention mitigates this by fusing operations to improve data locality and reduce intermediate memory traffic. In this paper, we present FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector register utilization and memory access locality. We integrate FlashAttention-V into ggml within llama.cpp and evaluate it on TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M using gem5 and a Banana Pi BPI-F3. On the Banana Pi BPI-F3, we confirm that loop reordering and loop unrolling across attention heads are effective optimization principles, scaling performance gains with larger models and most pronounced with short contexts and during decoding. Simulation-based analysis shows that FlashAttention-V achieves 22x-42x speedup over scalar FlashAttention at 512-bit VL in prefill, with an additional 2x-2.5x gain scaling to 64 lanes and 4096-bit VL. During decode, FlashAttention-V achieves 8x-11x speedup using 512-bit vector lengths over scalar FlashAttention, with performance showing diminishing sensitivity to vector width and lane count due to single-token, memory-bound execution. We further identify structural bottlenecks in Q8_0 quantized linear layers that limit arithmetic amortization under long-vector execution, consistent across RVV and Arm SVE, indicating that current quantization formats pose a fundamental challenge to long-vector scalability.

• The paper introduces FlashAttention-V, a blocked and fused attention kernel that packs multiple attention heads into wide vector registers, achieving up to 12×–14× physical speedups and 22×–42× simulated prefill speedups over scalar FlashAttention.
• The method combines cache-aware tiling, loop reordering, head unrolling, register-resident online softmax, and GQA-aware key/value reuse to improve vector utilization when the vector length exceeds the attention head dimension.
• The evaluation shows that prefill benefits from latency-aware vectors near 4096 bits, while single-token decode quickly saturates and Q8_0 quantized linear layers remain a system bottleneck because packing and reduction consume about 60% of execution cycles.

FlashAttention for Scalable Vector Architectures addresses a specific mismatch between transformer attention and vector-length-agnostic CPUs. Existing CPU and RVV implementations generally vectorize along the attention-head dimension $D$, typically 64 or 128 elements. Consequently, vector lengths greater than $D$ cannot expose additional intra-head parallelism. The paper proposes FlashAttention-V, a blocked and fused attention implementation that instead exploits parallelism across independent attention heads, allowing vector registers to remain productive when the hardware vector length exceeds the head dimension (2608.18656).

Problem formulation and design objective

The underlying computational problem is exact scaled dot-product attention with online softmax. FlashAttention avoids materializing the full score and probability matrices by tiling the $QK^{\mathsf{T}}$ and $QV$ computations, while maintaining running maxima, normalization factors, and output accumulators. This reduces intermediate memory traffic, following the IO-aware principles established by FlashAttention (Dao et al., 2022) and the memory-efficient online-softmax formulation of Rabe and Staats (Rabe et al., 2021).

The remaining challenge on scalable vector processors is not only memory traffic but also vector utilization. A conventional implementation processes one head at a time and performs dot products, scaling, rescaling, and value accumulation over $D$. For FP16, $D=64$ corresponds to a 1024-bit vector and $D=128$ to a 2048-bit vector. Wider RVV or Arm SVE registers therefore contain unused capacity unless the implementation discovers a second parallel dimension.

FlashAttention-V treats attention heads as the relevant additional dimension. Its design combines:

• cache-aware blocking over sequence and key tiles;
• loop reordering to exploit contiguous cross-head tensor layout;
• loop unrolling over attention heads to expose ILP;
• inter-head packing of multiple heads into one vector register;
• vector-register retention of intermediate softmax state;
• GQA-aware reuse of shared keys and values;
• separate execution regimes for $VL \leq D$ and $VL > D$.

The approach is implemented in ggml within llama.cpp using RVV and Arm SVE intrinsics. The evaluation covers TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M, with both GQA and MHA configurations. It uses a Banana Pi BPI-F3 for silicon-based validation, QEMU for functional portability checks, and gem5 for cycle-level RVV scalability analysis up to 8192-bit vector lengths (2608.18656).

Inter-head packing and kernel organization

When $VL \leq D$, FlashAttention-V retains conventional vectorization within one head but adds loop reordering, unrolling, blocking, and GQA-specific data reuse. When $D$0, it computes multiple heads concurrently. If a vector register can accommodate $D$1 head rows, the implementation packs those rows into one register. This transforms unused vector capacity into independent attention work without changing the mathematical decomposition of multi-head attention.

Figure 1: Inter-head packing maps elements from multiple attention heads into one FP16 vector register to utilize vector widths larger than the head dimension.

The implementation keeps query fragments and online-softmax state in vector registers across inner-loop iterations. For MHA, keys and values can be loaded contiguously across heads. For GQA, a shared key or value vector is loaded once and replicated across the corresponding query heads using vector slide operations. This is important because merely packing query heads without eliminating redundant K/V loads would leave much of the memory-bandwidth cost unchanged.

The algorithm also makes the unrolling factor dependent on vector width, element precision, head dimension, register utilization, and the number of available heads. This introduces an architectural constraint: vector-width scalability depends not only on the vector unit but also on whether the model contains enough heads to populate the registers. The paper demonstrates this explicitly for Pythia-410M, whose 16 MHA heads provide less packing and unrolling capacity than the 32-query-head models.

Hardware and evaluation methodology

The experimental methodology deliberately separates measured hardware results from simulated scalability results. The Banana Pi BPI-F3 has a 256-bit RVV vector length and therefore operates in the $D$2 regime for the evaluated models. It tests the practical effects of loop reordering, unrolling, cache blocking, and GQA reuse, but cannot validate long-vector inter-head packing directly.

The gem5 experiments use an in-order RVV MinorCPU with vector lengths from 512 to 8192 bits. The default gem5 SIMD functional unit assigns fixed latency to vector operations, which would make longer vectors unrealistically inexpensive. The authors therefore introduce a latency-aware model in which SIMD latency grows with the number of physical vector-lane passes. A 512-bit-per-cycle unit is used as the reference, with analyses corresponding to 8, 16, 32, and 64 vector lanes.

This distinction is consequential. Results under fixed operation latency are presented as upper-bound speedups, whereas results under vector-length-proportional latency are intended to capture the throughput-versus-latency trade-off of long-vector microarchitectures. The simulation remains a model rather than a substitute for measurements on processors with 4096- or 8192-bit vector registers. In addition, the evaluation is single-core and uses batch size one, so it does not establish multicore or batched-serving performance.

Results on the Banana Pi BPI-F3

On the 256-bit Banana Pi platform, FlashAttention-V in FP32 improves prefill performance over scalar FlashAttention by $D$3 for TinyLlama and $D$4 for Qwen2.5. Relative to the vectorized FP32 ggml implementation, the gains are approximately $D$5 for both models. Against an optimized self-attention implementation, FlashAttention-V delivers a $D$6 speedup for TinyLlama, supporting the claim that the fused, tiled algorithm remains beneficial on a CPU platform rather than only on GPUs.

In FP16 prefill, FlashAttention-V matches ggml-vec-fp16 at 512 tokens and exceeds it at shorter contexts. At sequence lengths of 64, 128, and 256, the reported average improvement over ggml-vec-fp16 is $D$7–$D$8, with Qwen2.5 reaching $D$9–$QK^{\mathsf{T}}$0 at 64 and 128 tokens. The effect is therefore strongest when intermediate attention data and repeated memory accesses are difficult to amortize over a long sequence.

Figure 2: Prefill performance comparison between FlashAttention-V and the existing ggml FP16 vectorized implementation.

The model dependence is technically informative. Qwen2.5 benefits from $QK^{\mathsf{T}}$1, which provides more intra-head work and better amortization of blocking overhead. Pythia-410M, with only 16 heads and MHA rather than GQA, is less favorable; at 512 tokens, FlashAttention-V is approximately 7% slower than ggml-vec-fp16. This result qualifies the paper’s central scalability argument: inter-head packing is not uniformly beneficial when the model has too few heads or lacks K/V sharing.

Decode results on the physical platform show $QK^{\mathsf{T}}$2–$QK^{\mathsf{T}}$3 speedups over ggml-scalar and approximately $QK^{\mathsf{T}}$4 over the existing vectorized FP16 and FP32 implementations for the evaluated models. However, these attention-level gains do not translate proportionally to end-to-end inference. Profiling indicates that FlashAttention accounts for approximately 50% of per-layer prefill execution but only about 10% during decode, where quantized linear projections and feed-forward layers dominate. Accordingly, end-to-end throughput is often comparable to the baseline: Qwen2.5 prefill improves by approximately 4%, while TinyLlama and Llama 3.2 remain close to baseline performance. This is a direct instance of kernel-level acceleration being limited by the rest of the execution graph.

Simulated prefill scalability

Under the fixed-latency gem5 configuration, FlashAttention-V provides substantial speedups at all evaluated vector widths. At 512-bit vectors, prefill speedups over ggml-scalar range from $QK^{\mathsf{T}}$5 to $QK^{\mathsf{T}}$6 for TinyLlama, Llama 3.2, and Pythia-410M, and reach $QK^{\mathsf{T}}$7 for Qwen2.5. These results include both vector arithmetic and the effects of the redesigned memory-access pattern.

Figure 3: Prefill speedup over ggml-scalar across RVV vector lengths for four decoder-only models at a 512-token context.

Scaling from 512 to 8192 bits produces an additional approximately $QK^{\mathsf{T}}$8 gain for TinyLlama and Llama 3.2 and approximately $QK^{\mathsf{T}}$9 for Qwen2.5 under fixed operation latency. The variation follows model geometry. At 8192 bits, FP16 vectors can hold eight heads for $QV$0 and four heads for $QV$1. Qwen2.5 consequently has fewer independently packed heads per register than TinyLlama and Llama 3.2, despite its larger head dimension.

Pythia-410M reaches its maximum speedup near 4096 bits. Beyond that point, its limited 16-head count prevents further increases in the register-utilization factor and therefore restricts additional unrolling. This observation is important because it shows that scalable-vector performance is a joint property of hardware vector width and model topology; widening the vector unit cannot compensate for insufficient independent work.

With latency proportional to vector width, the paper identifies 64 lanes and a 4096-bit vector as the most favorable configuration among those studied. Eight-lane designs saturate or degrade beyond 1024 bits, whereas 16- and 32-lane designs remain effective to approximately 4096 bits. The 64-lane configuration sustains $QV$2–$QV$3 additional prefill speedup when moving from 512 to 4096 bits, with Qwen2.5 obtaining up to a further $QV$4 at 8192 bits.

Figure 4: Latency-aware prefill speedups showing the interaction between vector width, physical lane count, and vector-operation latency.

The implication is not that the widest vector is always optimal. Under realistic latency scaling, vector width must be matched to physical lane throughput. A 4096-bit register backed by 64 lanes provides sufficient parallelism to amortize vector-operation latency, while still avoiding the multiple-pass penalties that affect wider vectors on narrower implementations. The paper’s claim that 4096-bit vectors are preferable to 8192-bit vectors is therefore conditional on the modeled lane organizations and latency assumptions.

Decode behavior and the limits of single-token parallelism

Decode is structurally different from prefill. Each step processes one new token, so the query, key, and value tensors for the new token have shape $QV$5 per head. Inter-token parallelism is absent, and the principal remaining source of parallelism is across attention heads.

At 512-bit vector length, FlashAttention-V achieves $QV$6–$QV$7 speedup over ggml-scalar across the four models under the fixed-latency simulation. Increasing the vector length to 4096 bits yields only an approximately $QV$8 improvement for TinyLlama and Llama 3.2, with no significant additional gain for Qwen2.5 and Pythia-410M.

Figure 5: Decode-stage speedup over ggml-scalar across RVV vector lengths under fixed-operation-latency simulation.

The saturation is attributable to insufficient work per iteration. Packing and managing wider registers introduces overhead that cannot be amortized by a single-token computation. Under latency-aware operation modeling, decode performance becomes nearly invariant across vector widths and lane counts. This result is stronger than a simple observation of diminishing returns: for single-token decode, increasing physical vector width provides little benefit once moderate inter-head parallelism is available.

Figure 6: Latency-aware decode speedups showing the weak dependence on vector length and lane count in the single-token regime.

The paper also reports that on gem5 at 512 bits, TinyLlama FlashAttention-V takes 0.89 time units in FP32 and 0.78 in FP16 during prefill, compared with 21.38 for non-vectorized FlashAttention. In decode, the corresponding values are 0.000077 and 0.000070, compared with 0.000529 for the scalar implementation. These figures support the reported $QV$9 FP16 prefill and approximately $D$0 FP32 decode speedups, but they should be interpreted as kernel-level simulation results rather than end-to-end model latency.

Quantized linear layers as a separate scalability barrier

The paper extends its analysis beyond attention because attention acceleration has limited system-level value when projection and feed-forward layers dominate decode. The relevant operations are GEMV-like dot products over Q8_0 quantized weights. Each Q8_0 block contains 32 INT8 weights and an FP16 scale in an interleaved layout.

This representation is compact but poorly aligned with long-vector execution. A vector load that spans multiple blocks also loads scale values, requiring either explicit packing and separation or strided accesses. The authors evaluate packing with vector slide operations, output-dimension vectorization, and LMUL-based register grouping.

At 2048-bit VL, the microbenchmark attributes approximately 28% of execution cycles to packing, 32% to masked reduction, and 40% to vector MAC operations. Thus, packing and reduction consume 60% of execution time, exceeding the arithmetic component that long vectors are intended to accelerate. Register packing consequently degrades TinyLlama performance by approximately $D$1 rather than improving it. On Arm SVE, the best evaluated packing strategy still incurs approximately 20% overhead at 2048-bit vectors.

Output-dimension vectorization removes packing and reduction but produces stride-32 accesses across quantization blocks. On the Banana Pi, poorer cache-line utilization and increased memory latency outweigh the arithmetic benefit. LMUL-based strategies avoid explicit packing, but the gem5 RVV model cannot reliably simulate the required register-group type conversions for $D$2; those results therefore remain unvalidated.

The conclusion is a structural one: the Q8_0 layout couples quantization metadata to weight data in a way that makes long-vector arithmetic difficult to amortize. This limitation is reported on both RVV and Arm SVE, suggesting that it derives from the data format and memory layout rather than from one particular ISA. It also explains why substantial attention-kernel speedups produce modest decode-stage end-to-end gains.

Limitations and open questions

The strongest long-vector results are simulation-based and rely on a modified gem5 MinorCPU model. Although the latency model is motivated by lane-based vector designs such as Vitruvius+, Ara2, and AraXL, it does not establish performance on a physical processor with the modeled vector widths. The paper also uses an in-order CPU because the gem5 out-of-order RVV configuration could not complete the simulations due to memory exhaustion. Effects involving out-of-order scheduling, vector register renaming, speculative execution, and wider memory concurrency therefore remain unresolved.

The Banana Pi validation is limited to a 256-bit vector length and cannot directly test the central $D$3 packing regime in hardware. Arm SVE validation is functional and used for the quantized-layer analysis, not for performance evaluation of FlashAttention-V. The experiments further use batch size one and a single inference request, leaving the interaction between inter-request batching, head packing, cache residency, and vector scaling open.

The Q8_0 analysis identifies packing and reduction as dominant costs, but does not evaluate alternative quantization layouts or formats experimentally. In particular, it remains open whether a block format with separately contiguous scales, larger quantization blocks, or hardware-supported deinterleaving could recover long-vector arithmetic intensity without imposing unacceptable memory overhead. The paper also leaves unresolved whether the same conclusions hold for newer low-bit formats and for projection kernels optimized with LMUL register groups on real RVV hardware.

Conclusion

FlashAttention-V redesigns CPU FlashAttention around the central observation that attention heads provide exploitable parallelism when the vector length exceeds the head dimension. Loop reordering, inter-head packing, unrolling, register reuse, blocking, and GQA-aware K/V reuse deliver $D$4–$D$5 prefill speedups and $D$6–$D$7 decode speedups over scalar FlashAttention at 512-bit simulated vectors, with physical Banana Pi improvements of $D$8–$D$9.

The scalability results distinguish idealized vector-width gains from latency-aware hardware behavior. Prefill benefits from approximately 4096-bit vectors with sufficient lane throughput, whereas decode rapidly saturates because single-token execution exposes little parallel work. More fundamentally, Q8_0 quantization prevents the surrounding linear layers from exploiting long vectors: packing and masked reduction account for 60% of measured cycles at 2048-bit VL. The paper therefore establishes both an effective attention-kernel strategy and a clear system-level qualification: scalable vector hardware alone does not guarantee scalable quantized transformer inference.