Flash-ABFT: Fused Fault Detection in Transformers

• The paper introduces Flash-ABFT, a fused fault detection mechanism that computes a single online checksum spanning QKᵀ, softmax, and AV to monitor the entire attention operator.
• It leverages hardware-produced values like dot products, running log-sum-exp normalizers, and streamed row sums to detect errors without separate GEMM checks.
• Experimental results show detection rates up to 98.87% with only about 5.3% area and 1.9% energy overhead, emphasizing its efficiency for inference accelerators.

Flash-ABFT is a fused, algorithm-based fault detection mechanism for the full attention operator in Transformers. It computes a single online checksum spanning $QK^\top$, row-wise softmax, and the final $AV$ product while preserving the streaming and tiling structure of FlashAttention-style kernels. The method derives an invariant equal to the total sum of the output matrix $O$ and predicts it online from quantities already produced by the hardware—dot products, running log-sum-exp normalizers, and streamed rows of $V$—thereby avoiding separate checks of the intermediate GEMMs and the nonlinearity barrier at softmax. In reported experiments, it incurred 5.3% hardware area overhead and less than 1.9% energy overhead (Titopoulos et al., 22 Jul 2025).

1. Scope, nomenclature, and problem setting

Flash-ABFT addresses error detection in attention accelerators for Transformers and LLMs. Its immediate target is the attention layer with query, key, and value matrices $Q \in \mathbb{R}^{N \times d_k}$, $K \in \mathbb{R}^{N \times d_k}$, and $V \in \mathbb{R}^{N \times d_v}$, where each row corresponds to a token, $N$ is sequence length, $d_k$ is the key/query dimension, and $d_v$ is the value dimension. The pre-softmax score matrix is

$AV$0

with $AV$1 representing optional positional or relative bias and $AV$2 representing causal or padding masks. Attention weights are defined by row-wise softmax, $AV$3, and the output is $AV$4.

Within this setting, Flash-ABFT is not a generic ABFT wrapper around separate matrix multiplications. It is tailored to the fused attention pipeline as executed by FlashAttention-style kernels, including their streaming and tiling structure. The mechanism uses one online checksum across the entire operator rather than verifying $AV$5 and $AV$6 independently.

The term should not be conflated with the unrelated system named “Flash,” an asynchronous Byzantine fault-tolerant payment protocol based on a blocklace data structure (Lewis-Pye et al., 2023). In Flash-ABFT, “ABFT” denotes algorithm-based fault tolerance for hardware error detection in attention computation rather than asynchronous Byzantine fault tolerance in distributed systems.

2. Why conventional ABFT is insufficient for attention

Classical ABFT for a single matrix multiplication $AV$7 relies on linear invariants such as row sums, column sums, or weighted sums. With a checksum vector $AV$8, one may verify relations of the form $AV$9, or use augmented rows and columns so that the augmented product reproduces the augmented output. These checks are preserved by linear operators.

Attention disrupts this structure at two points. First, softmax is nonlinear and normalizes each row independently. Its outputs depend on $O$0 and row-local denominators $O$1, so any checksum constructed upstream of softmax cannot be propagated through exponentiation and row-wise normalization by standard linear ABFT arguments. Second, masking and optional dropout alter the structure of the attention weights. Even if softmax yields a row-stochastic matrix $O$2, multiplicative masks applied after softmax can break or modify that property unless scaled appropriately.

For this reason, per-GEMM ABFT applied separately to $O$3 and $O$4 cannot detect faults introduced inside the softmax and cannot guarantee consistency across the fused pipeline. The lost invariant is linearity: the usual “augment rows/columns” technique does not survive normalization. Flash-ABFT is designed specifically to recover a usable invariant at the level of the full attention operator rather than at the level of its constituent GEMMs.

3. Checksum formulation and online propagation

The central invariant in Flash-ABFT is the total sum of the elements of the attention output $O$5. From $O$6, the global checksum can be written as

$O$7

Defining

$O$8

the checksum is

$O$9

The same quantity admits a per-query decomposition. For query row $V$0, let $V$1 together with any bias and mask applied at position $V$2. Then

$V$3

with

$V$4

This is exactly the total sum of the $V$5-th output row:

$V$6

This decomposition makes the check implementable online during streaming execution. FlashAttention-2 already maintains, for each query row, a running maximum $V$7, a running denominator $V$8, and a running unnormalized output vector $V$9. For a single query with streamed keys and values indexed by $Q \in \mathbb{R}^{N \times d_k}$0, the standard updates are

$Q \in \mathbb{R}^{N \times d_k}$1

$Q \in \mathbb{R}^{N \times d_k}$2

$Q \in \mathbb{R}^{N \times d_k}$3

$Q \in \mathbb{R}^{N \times d_k}$4

Flash-ABFT adds a scalar accumulator $Q \in \mathbb{R}^{N \times d_k}$5 that mirrors $Q \in \mathbb{R}^{N \times d_k}$6 but replaces the streamed vector $Q \in \mathbb{R}^{N \times d_k}$7 with the row sum of that vector:

$Q \in \mathbb{R}^{N \times d_k}$8

At the end of the row,

$Q \in \mathbb{R}^{N \times d_k}$9

and the global predicted checksum is

$K \in \mathbb{R}^{N \times d_k}$0

Masking and dropout are incorporated directly into the streamed update. Masking is folded into $K \in \mathbb{R}^{N \times d_k}$1 through $K \in \mathbb{R}^{N \times d_k}$2, so a masked position with $K \in \mathbb{R}^{N \times d_k}$3 effectively contributes zero to both $K \in \mathbb{R}^{N \times d_k}$4 and $K \in \mathbb{R}^{N \times d_k}$5. If dropout is applied after softmax with keep probability $K \in \mathbb{R}^{N \times d_k}$6 and mask $K \in \mathbb{R}^{N \times d_k}$7, then $K \in \mathbb{R}^{N \times d_k}$8 and $K \in \mathbb{R}^{N \times d_k}$9 are multiplied by $V \in \mathbb{R}^{N \times d_v}$0 in the updates for $V \in \mathbb{R}^{N \times d_v}$1 and $V \in \mathbb{R}^{N \times d_v}$2, while $V \in \mathbb{R}^{N \times d_v}$3 remains unchanged.

Numerical stability follows the same log-sum-exp procedure used by FlashAttention. Whenever the running maximum increases, both $V \in \mathbb{R}^{N \times d_v}$4 and $V \in \mathbb{R}^{N \times d_v}$5 are rescaled by $V \in \mathbb{R}^{N \times d_v}$6. Flash-ABFT rescales $V \in \mathbb{R}^{N \times d_v}$7 identically. The stated equivalence between $V \in \mathbb{R}^{N \times d_v}$8 and $V \in \mathbb{R}^{N \times d_v}$9 is preserved algebraically provided the arithmetic format can represent the intermediate products and exponentials. The implementation guidance keeps $N$0, $N$1, and $N$2 in higher precision to minimize rounding-induced silent errors.

4. Detection rule and hardware realization

The predicted checksum is compared against an “actual” checksum produced directly from the computed output. After each query row is normalized as $N$3, the hardware sums the elements of that row,

$N$4

and accumulates the result into a global register $N$5. In parallel, it accumulates the predicted per-query values $N$6 into $N$7. Fault detection is triggered by the condition

$N$8

In the reported experiments, $N$9 was used to avoid false negatives due to rounding while maintaining sensitivity (Titopoulos et al., 22 Jul 2025).

The hardware additions are modest and closely aligned with the existing FlashAttention datapath. A $d_k$0-wide adder tree computes $d_k$1 for each streamed row of $d_k$2. Each active query lane receives one additional scalar register $d_k$3 and one scalar fused multiply-add update of the form

$d_k$4

or the dropout-adjusted form when dropout is enabled. A post-normalization adder tree computes $d_k$5, and two global accumulators hold $d_k$6 and $d_k$7.

The control changes are similarly limited. The finite-state machine must initialize $d_k$8 on tile entry, update it in lockstep with $d_k$9 and $d_v$0, and accumulate $d_v$1 on tile exit. The divider used to normalize $d_v$2 can also be reused or multiplexed to compute $d_v$3. The checker preserves FlashAttention’s streaming, blockwise execution and IO-awareness; checksum updates are performed tile-by-tile, and end-of-layer detection latency is described as a handful of cycles beyond the last tile’s normalization.

Precision choices are integral to the design. The compute path may use FP16 or BF16, while $d_v$4, $d_v$5, $d_v$6, $d_v$7, and $d_v$8 are kept in FP32 or double precision. Compatibility with FP32 is explicit. For INT8 or other quantized inference, the exponentials and normalizers are still required in floating-point, or fixed-point exponential approximations must be validated so that the checksum identity remains valid.

5. Fault model, experimental methodology, and measured behavior

The evaluation injects random single-bit flips during execution into registers of the FlashAttention-2 kernel and of the checker, including MAC array accumulators, local buffers, running normalizers, and checker accumulators. Input memories for $d_v$9, $AV$00, and $AV$01 are assumed to be protected by separate logic and are therefore excluded from the fault model. The methodology comprises 10,000 independent single-fault campaigns on the first attention layer across four LLMs—BERT, Phi-3-mini, Llama-3.1, and Gemma2—at sequence length $AV$02 and hidden dimensions $AV$03, using BF16 arithmetic, double-precision checksum accumulators, and $AV$04 (Titopoulos et al., 22 Jul 2025).

Three outcome categories are reported. “Detected” means $AV$05. “False Positive” denotes a checker fault that flips $AV$06 or $AV$07 and raises an alarm even though $AV$08 is correct. “Silent” covers cases where a fault produces NaN/INF or produces an incorrect $AV$09 while the checksum difference remains within $AV$10.

$AV$11	Outcome rates
64	Detected 96.94%, False Positive 2.66%, Silent 0.40%
96	Detected 97.56%, False Positive 1.99%, Silent 0.45%
128	Detected 98.45%, False Positive 1.25%, Silent 0.30%
256	Detected 98.87%, False Positive 0.62%, Silent 0.51%

The paper states that detection improves with larger hidden dimension because the checker occupies a smaller fraction of total state. It also reports that no false negatives were observed, noting that a fault in the compute path and an offsetting fault in the checker would be required to cancel at the checksum.

The overhead measurements are reported in two forms. The headline results are 5.3% area overhead and less than 1.9% energy overhead. For a 28 nm implementation with BF16 compute and double-precision checksum accumulators, the average overheads across configurations are approximately 4.55% area and approximately 1.53% dynamic power, with the per-row $AV$12 adder shared across lanes.

Qualitative baselines situate the method within the broader fault-tolerance landscape. Per-GEMM ABFT cannot verify the nonlinear softmax and therefore leaves faults in normalization and streaming updates uncovered. ECC protects storage rather than compute datapaths and does not cover compute faults. DMR and TMR incur higher area, energy, and latency penalties relative to the single-check design used here.

6. Limitations, edge cases, and development directions

Several limitations are explicit. When softmax saturates and one score dominates a row, the attention vector becomes nearly one-hot; small faults in strongly suppressed entries may have limited effect on $AV$13 and therefore on the checksum, reducing sensitivity to subtle errors. Heavy masking reduces the number of contributing terms in $AV$14 and $AV$15, so larger localized errors may be required to exceed the detection threshold. Very low-precision maintenance of $AV$16, $AV$17, or $AV$18 can increase silent faults, which is why FP32 or double precision is recommended for the running sums. Very long sequences enlarge the dynamic range of $AV$19 and $AV$20, again motivating higher-precision accumulators.

Kernel fusion and stalls are treated as compatible with the method because the checksum lane shares timing with $AV$21 and $AV$22; pipeline stalls therefore propagate consistently. Corner cases producing NaN or INF are expected to be trapped by standard floating-point exception handling, which can be integrated with the checker.

The stated application domain is primarily inference accelerators. Forward-pass checking remains valid in the presence of dropout by incorporating the dropout factor into the checksum update, but training introduces gradient backpropagation and different invariants. Extending the derivation to the backward pass is identified as future work. Other stated directions include extension to multi-head fusion with head-wise aggregation and adaptive thresholds $AV$23 that vary with precision and sequence length.

A plausible implication is that Flash-ABFT is most compelling where fused attention kernels are already present and the design objective is low-cost online detection rather than replication-based fault masking. That implication is consistent with the paper’s emphasis on preserving FlashAttention’s streaming structure, using values already produced by the hardware, and replacing multiple checks with a single end-to-end invariant (Titopoulos et al., 22 Jul 2025).