Express: Causal Attention Conversion

• Express is a framework that transforms non-causal attention approximations into causal, streaming methods tailored for autoregressive transformers.
• It employs a hierarchical cache with thinning and halving operations to compress the token prefix while managing error propagation and resource usage.
• The framework provides strong theoretical guarantees and practical speedups in long-context prefill, KV cache compression, and decode efficiency compared to prior methods.

Express is a meta-algorithm for converting a non-causal attention approximation into a causal approximation for autoregressive transformers. In the formulation of "Express Language Modeling" (Gong et al., 9 Jun 2026), it is an updatable weighted cache object used inside transformer attention to approximate causal attention while largely preserving the approximation quality of the underlying non-causal method. The framework is motivated by the mismatch between strong subquadratic approximations developed for unmasked attention and the online, cache-growing regime of language modeling, where each token may attend only to its prefix and inference cost is dominated by long-context prefill, KV cache growth, and long-form decoding (Gong et al., 9 Jun 2026).

1. Problem formulation and causal-attention setting

Express is defined around the distinction between unmasked and masked attention. The non-causal form assumes that the entire sequence is visible: $$\operatorname{Attention}(q_j,(k_i,v_i)_{i=1}^n) = \frac{\sum_{i=1}^n e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^n e^{\langle q_j,k_i\rangle/\sqrt d}},$$ whereas language modeling requires the causal form

$$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$

The causal constraint is the operational source of difficulty: the usable context grows online, the KV cache grows linearly with sequence length, and naive attention remains quadratic in context length (Gong et al., 9 Jun 2026).

The paper’s central claim is that many strong attention approximations were designed for the non-causal regime, where a single offline coreset or summarization can represent the full sequence, but this assumption fails during decoding. Express is introduced as a bridge from offline, unmasked coreset approximation to online, causal attention approximation. Its stated goals are to reuse strong existing approximations rather than redesign causal approximations from scratch, make them streaming and incrementally updatable, preserve guarantees comparable to the original non-causal method, and reduce memory and compute in long-context prefill, KV-cache management, and long-form decoding (Gong et al., 9 Jun 2026).

A common misconception is to treat Express as a standalone attention approximation family. The paper instead presents it as a conversion framework: its object of study is not a new softmax surrogate per se, but a general procedure for turning a suitable non-causal approximation into a causal one with controlled degradation in approximation quality (Gong et al., 9 Jun 2026).

2. Cache architecture and algorithmic mechanism

Algorithmically, Express is described as a streaming weighted cache with a hierarchical structure of weighted summaries or coresets of the prefix. The informal mechanism has three phases. In the exact phase, the first $n$ points are stored directly. In the thin phase, new points are processed in batches, a lightweight subsampling step reduces large incoming blocks, and a recursive compression procedure reduces them further. In the halve phase, when the cache becomes too large, Express applies the base halving algorithm twice to bring the cache back to target size, and the thinning depth parameter $m$ is increased (Gong et al., 9 Jun 2026).

The generalized Compress procedure is the principal structural device that keeps updates tractable. Rather than applying the expensive halving algorithm to all points at once, it applies halving only on smaller groups with geometrically increasing scale. The paper attributes the manageable update cost of Express to this design choice. Early points can therefore be retained exactly, later points are summarized, and the cache remains incrementally maintainable under streaming arrival of tokens (Gong et al., 9 Jun 2026).

This architecture is explicitly framed in terms of thinning and halving. A coreset is a weighted subset summarizing the prefix; thinning selects a smaller weighted subset that preserves relevant statistics; halving is a thinning step that reduces a set roughly by half while maintaining approximation quality. Express packages these operations into an updatable object that can be queried causally at every decoding step (Gong et al., 9 Jun 2026).

The paper’s resource discussion also emphasizes that the cache no longer needs to track the full prefix explicitly. Ordinary decoding requires a KV cache growing with sequence length, whereas Express maintains a compressed dynamic cache whose size scales with the target summary size rather than the entire context, which is the operational basis for its memory and query-time advantages (Gong et al., 9 Jun 2026).

3. Theoretical guarantees and approximation quality

The main theory is a meta-procedure that converts any suitable sub-Gaussian halving algorithm into a streaming thinning algorithm for causal attention. The key guarantee is that if the base halving algorithm $Halve$ has good sub-Gaussian approximation behavior, then Express produces a streaming weighted cache whose error is controlled in terms of the base algorithm’s error, with only mild inflation. The paper states that the causal conversion inflates the sub-Gaussian error by at most

$\sqrt{\frac{16}{15}(16+3m)},$

relative to a single halving phase, where $m$ is the current thinning depth (Gong et al., 9 Jun 2026).

When Express is combined with Thinformer, the resulting causal approximation achieves, for bounded inputs,

$O\!\left(\frac{\log^{3/2}(n)}{s}\right)$

approximation error with $O(s)$ memory and $O\!\big(s^2\log^2(n)\big)$ compression overhead for a sequence of length $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$0. This is the headline tradeoff of the paper: the error decays inversely with cache size $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$1, up to polylogarithmic factors in sequence length, while memory remains linear in the chosen cache budget rather than the full context length (Gong et al., 9 Jun 2026).

The theory also gives generic streaming space and runtime statements in terms of the base halving routine. If the underlying halving algorithm uses $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$2 memory, then Express uses

$$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$3

space in the streaming setting, where $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$4 is the head dimension. For the kernel-halving instantiation discussed in the paper, this becomes about $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$5. For runtime, the paper states that if the base algorithm is polynomial-time $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$6, then Express yields a causal streaming algorithm with improved runtime behavior, especially when the target cache size $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$7 is much smaller than $$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$8. For the main kernel-halving instantiation, the total compression runtime is

$$o_j= \frac{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}\,v_i}{\sum_{i=1}^j e^{\langle q_j,k_i\rangle/\sqrt d}}.$$9

These results situate Express as a controlled causalization theorem rather than an empirical-only systems heuristic (Gong et al., 9 Jun 2026).

The paper also compares its asymptotics with earlier causal approaches. It states that Express + Thinformer improves on prior causal guarantees by providing better approximation decay in $n$0, lower query cost due to a smaller cache, and lower compression overhead in heavily compressed regimes. For $n$1, the paper emphasizes an $n$2 overhead for Express versus a larger $n$3-type overhead for BalanceKV (Gong et al., 9 Jun 2026).

4. Systems implementation and I/O-aware realization

Express is paired with an efficient Triton implementation designed in the style of FlashAttention. The implementation strategy is explicitly I/O-aware: it uses tiling to reduce slow high-bandwidth-memory reads and writes, avoids materializing large kernel matrices, parallelizes across batch, head, and row blocks, parallelizes halving calls on same-sized groups for long contexts, and uses double dereferencing to avoid unnecessary data copies when indexing keys and values (Gong et al., 9 Jun 2026).

The practical consequence is that the system is tuned around memory movement rather than arithmetic complexity alone. This is an important part of the paper’s contribution, because causal compression methods can fail to produce end-to-end wins if their maintenance overhead dominates query cost. The Triton implementation is intended to make the theoretical conversion useful under realistic GPU execution constraints (Gong et al., 9 Jun 2026).

Reported performance is substantial. For unmasked attention, Triton Thinformer is stated to be faster than the original PyTorch-compiled version. At context length $n$4, the Triton implementation reaches about $n$5 speedup over FlashAttention 2 for the unmasked prefill benchmark, compared to about $n$6 for the original implementation. For masked long-context prefill, Express reaches up to $n$7 speedup over FlashAttention 2 at $n$8 tokens. The paper also reports that HyperAttention can run out of memory at $n$9 tokens on the tested GPU, while Express does not (Gong et al., 9 Jun 2026).

These implementation results matter because the paper’s thesis is not merely asymptotic. Express is positioned as a framework whose causalization guarantees, cache maintenance cost, and GPU kernel design are aligned closely enough to produce wall-clock improvements in the regimes that dominate modern inference (Gong et al., 9 Jun 2026).

5. Use in the language-modeling pipeline

The paper organizes Express around four bottlenecks in language modeling: long-context prefill, KV cache compression, long-form memory-constrained decoding, and long-form compute-constrained decoding (Gong et al., 9 Jun 2026).

Bottleneck	Role of Express	Reported outcome
Long-context prefill	Approximates masked attention during prefill	Up to $m$0 faster than FlashAttention 2 at $m$1 tokens
KV cache compression	Used during prefill and combined with compression methods	Reduces total runtime of KV computation, compression, and later decoding attention
Memory-constrained decoding	Maintains a much smaller dynamic cache	On MATH-500, matches exact attention accuracy with about $m$2 of the cache size
Compute-constrained decoding	Reduces query cost against the cache	On MATH-500, matches exact attention accuracy with about $m$3 of the compute time

In long-context prefill, Express replaces exact masked attention, whose cost is quadratic in context length, with a causal approximation that preserves quality while materially reducing runtime. In KV cache compression, the paper uses Express as a cheap approximate attention engine during prefill and combines it with existing compression methods, specifically SnapKV, StreamingLLM, and PyramidKV, to reduce the total runtime of key-value computation, compression, and later decoding attention while preserving quality (Gong et al., 9 Jun 2026).

The decoding use cases separate memory and compute as distinct bottlenecks. In memory-constrained decoding, Express is valuable because the dominant problem is lack of storage for the full KV cache; the paper reports exact-attention-matching accuracy on MATH-500 using only about $m$4 of the cache size. In compute-constrained decoding, the main issue is per-token query cost rather than storage; here the paper reports matching exact attention accuracy with only about $m$5 of the compute time, and it further notes that the runtime overhead of maintaining the cache is small relative to the query cost (Gong et al., 9 Jun 2026).

A plausible implication is that Express is meant to be deployment-modular: the same causalized cache abstraction can serve as the approximation layer in heterogeneous inference scenarios, rather than being limited to a single prefill-only or decode-only operating point.

6. Position within the broader literature

Express is situated against two main prior lines identified in the paper: HyperAttention and BalanceKV. HyperAttention is described as practical but with causal guarantees that decay more slowly in sequence length under the paper’s assumptions. BalanceKV is described as streaming but with higher space and compression costs. Express + Thinformer is presented as improving on both through better approximation decay, lower query cost due to a smaller cache, and lower compression overhead in the heavily compressed regime (Gong et al., 9 Jun 2026).

The combination with Thinformer is central. Thinformer is characterized as the state-of-the-art non-causal thinning-based attention approximation from prior work, strongest for unmasked attention; Express makes Thinformer usable for causal attention by converting its offline coreset construction into a streaming weighted cache. The paper’s formulation therefore separates two roles: Thinformer supplies a strong non-causal approximation, and Express supplies the causalization theorem and streaming cache machinery needed for language modeling (Gong et al., 9 Jun 2026).

This division of labor clarifies the scope of the contribution. Express does not replace the underlying approximation family so much as extend its admissible deployment regime. That is why the paper’s claims span theory, systems, and inference applications at once: the same causal conversion framework is intended to preserve approximation guarantees, control memory and compression cost, and unlock concrete speedups in transformer inference (Gong et al., 9 Jun 2026).

The term should also be distinguished from unrelated arXiv uses of “Express,” including a metadata-hiding communication system (Eskandarian et al., 2019) and an “express” method for calculating algebraic entropies of three-point mappings (Ramani et al., 2016). In current language-modeling usage, however, Express denotes a causal-attention conversion framework whose defining feature is the transformation of non-causal thinning-based approximations into streaming causal approximations with matching-style guarantees and deployable GPU kernels (Gong et al., 9 Jun 2026).