---
title: Blockwise Transformers
url: https://www.emergentmind.com/topics/blockwise-transformers
type: topic
---

# Blockwise Transformers

Blockwise Transformers are a class of architectures and algorithmic strategies that reorganize standard Transformer computations—especially self-attention and feedforward sublayers—into operations over contiguous blocks of tokens. This restructuring is motivated by the prohibitive memory and computational demands of vanilla Transformers on long sequences, bringing both practical linear scaling and new model parallelization capabilities while enabling exact or controlled-approximate computation. Blockwise designs underpin a variety of recent advances in long-context language modeling, scalable vision architectures, streaming sequence processing, and efficient model compression.

## 1. Core Concepts and Problem Motivation

Transformers compute self-attention as a dense $O(N^2d)$ operation over sequences of length $N$, incurring quadratic memory and compute costs that grow rapidly with context size [2310.01889][2305.19370][1911.02972]. This limitation constrains both model training and inference, especially for use cases requiring extended context (e.g., document-level or video understanding, RL with long experience windows). Blockwise Transformers address this by partitioning the input sequence into $T=N/B$ contiguous, fixed-size blocks, and re-expressing attention and feed-forward networks in terms of operations over these blocks.

Key strategies include:

- **Blockwise computation**: Partition $Q,K,V$ projections into blocks to process attention in slices, never materializing the full $N \times N$ score matrix.
- **Blockwise memory and streaming**: Retain only block-local intermediate states, reducing activation storage from $O(N^2)$ to $O(N)$ or $O(BN)$ per layer.
- **Blockwise parallelism**: Schedule computation and communication over device meshes or rings, enabling large contexts across distributed resources [2310.01889].
- **Blockwise sparsity**: Employ block-diagonal or block-permuted attention patterns for further memory and compute reduction, at modest modeling cost [1911.02972].

## 2. Blockwise Attention Mechanisms and Exactness

Blockwise self-attention variants preserve exact softmax attention by leveraging incremental, block-local updates to softmax numerators and denominators. For example, in Blockwise Parallel Transformer (BPT), for each query block $Q_i$ of size $B$, attention proceeds by iteratively accumulating the contribution of every $K_j,V_j$ block:
\[
\text{(running) } N_i \leftarrow N_i + \exp\left(\frac{Q_i K_j^T}{\sqrt{d_h}} - m_i\right) V_j; \qquad
D_i \leftarrow D_i + \exp\left(\frac{Q_i K_j^T}{\sqrt{d_h}} - m_i\right)
\]
where $m_i$ ensures numerical stability. After all blocks, $O_i = N_i / D_i$. This form is memory-equivalent to FlashAttention-style kernels and retains full expressivity [2310.01889][2305.19370].

Blockwise attention unlocks critical memory savings:
- Vanilla transformer: $O(N^2d)$ activations per layer.
- Blockwise: $O(Nd)$ for projections plus $O(Bd)$ for accumulators. For moderate $B \ll N$, this is a decisive reduction.

Blockwise attention can also be fused with immediately local feedforward computation, further capping peak memory at $O(Bd_\mathrm{ff})$ per block [2305.19370].

## 3. Ring Attention and Device-Parallel Blockwise Scaling

Ring Attention extends blockwise Transformers by sharding blocks across $D$ devices, each hosting one $(Q_h, K_h, V_h)$ block. To compute self-attention over all $K_j,V_j$, devices are arranged in a logical ring; at each of $D-1$ steps, every device concurrently:
- Processes the current KV block and updates accumulators,
- Sends its local $K,V$ to successor, and receives a new $K,V$ from predecessor,
- Fully overlaps communication and blockwise compute.

This design achieves sequence length scaling linear in device count ($N_\text{max} \propto D$), with total memory and communication costs per device independent of total sequence length:
\[
\text{Per-device memory: } 6\,b\,B\,d_h \\
\text{Per-device comm: } (D-1)\times 2\,b\,B\,d_h \text{ floats}
\]
Block size is chosen such that $B \geq F/(2B)$ (device flop/comm bandwidth), which for modern devices is efficiently satisfied for $B \sim 1\,\mathrm{k}$ [2310.01889].

Empirically, this enables training and inference for millions of tokens (e.g., 4M tokens on 1024 TPUv4 with a 13B model) without reliance on attention approximations or increased overhead, while maintaining near-ideal model FLOP utilization and scalable throughput.

## 4. Blockwise Variants and Hybrid Architectures

Blockwise designs are employed in several complementary approaches:

- **Sparse Blockwise Attention**: Techniques such as BlockBERT replace the dense $N \times N$ attention matrix by a union of local, block-diagonal submatrices and a small set of global or off-diagonal connections. This reduces time and space complexity by a factor of $n = N/B$, with empirical pretraining speedups of up to 25% and 36% peak memory savings (negligible accuracy drop) [1911.02972].
- **Block-Recurrent Transformers**: Models process blocks sequentially, maintaining recurrent states across blocks. Each block applies a vertical (self/cross) attention over the block, and horizontal (self/cross) attention over the states. LSTM-style gating provides memory, and parallelism is retained within block. Perplexity improvements and order-of-magnitude context length scaling are observed over baseline XL-like models [2203.07852].
- **Block-State Transformers (BST)**: Hybridizes blockwise self-attention (for local context) with state-space models (for global/infinite context). Block-local context states are integrated via cross-attention, and extensive parallelism is achieved across sequence blocks. BST outperforms comparable architectures on language modeling of long sequences and delivers $6{-}11\times$ speedups at the layer level for long contexts on modern hardware [2306.09539].

The following table summarizes key characteristics of selected blockwise strategies:

| Method/Variant                | Context Scale       | Memory Complexity            | Blockwise Exact  | Parallelism         | Reference      |
|-------------------------------|--------------------|-----------------------------|------------------|---------------------|---------------|
| Blockwise Parallel Transformer| $2{-}32\times$ vanilla | $O(Nd)$                      | Yes              | Per-block           | [2305.19370]  |
| Ring Attention                | $D\times$ baseline | $O(Bd)$ per device           | Yes              | Device/Ring-wise    | [2310.01889]  |
| BlockBERT                     | $n$ blocks         | $O(N^2/n)$                   | Sparse           | Per-block/free head | [1911.02972]  |
| Block-Recurrent Transformer   | $O(WN)$            | $O(WN)$                      | Yes              | Per-block           | [2203.07852]  |
| Block-State Transformer       | Subquadratic ($O(N\log N)$ SSM) | $O(NB)$ | Yes | Per-block, SSM global | [2306.09539]  |

## 5. Blockwise Transformers in Streaming and Compression

Blockwise schemes also underpin streaming and blockwise-compressed Transformer designs:

- **Blockwise Streaming Processing**: Contextual block processing with inheritance is foundational for online ASR, SLU, and simultaneous speech translation. At each layer and block, a short "context embedding" from the previous block is appended, enabling modeling of global information with minimal added memory. Masks ensure no future context is seen, and left-context inheritance supports distributed, low-latency deployment [2204.08920][1910.07204].
- **Blockwise Model Compression (BCT)**: Compression frameworks such as BCT partition all weights and activations into small blocks, applying independent low-bit quantization schemes per block (e.g., 4/8-bit elements plus per-block scales). All nonlinearities (GELU, Softmax, LayerNorm) are compressed accordingly using lookup tables and interpolation. BCT achieves up to $7.988\times$ compression with minimal GLUE accuracy drops ($<1\%$) and—unlike layerwise quantization—requires no retraining [2304.01483].

## 6. Blockwise Self-Supervised and Learning Dynamics

Blockwise self-supervised learning (BWSSL) applies local objectives per block without end-to-end gradient flow. In masked video transformers, the encoder is split into $K$ blocks, each with a local decoder and loss. Gradients are stopped between blocks, and only local block parameters are updated per loss. Empirical findings on VideoMAE-style ViTs show:
- BWSSL converges reliably and matches end-to-end masked autoencoding on linear-probe and retrieval proxies,
- High-level features become linearly accessible at mid-depth earlier under BWSSL than under end-to-end training, as shown via probing and centered kernel alignment (CKA) analyses,
- Saturation and geometry preservation arise in later blocks, with diminished marginal gains—implicating stabilization interfaces as limiting factors for further local objective-driven progress [2601.09040].

## 7. Blockwise Parallel Decoding

Blockwise strategies accelerate autoregressive decoding by predicting $b$ tokens per forward pass via parallel "proposal" heads, then verifying and accepting the longest correct prefix. This reduces the number of decode steps by factors up to $5{-}7\times$ in practice (with minimal BLEU or image quality loss), and achieves up to $4\times$ real-world speedup in both MT and image super-resolution tasks. The core algorithmic change requires only a small multi-output projection in the decoder and a modified inference loop, without modifications to encoder or attention [1811.03115].

---

Blockwise Transformers comprise a broad methodological axis encompassing parallel memory-efficient attention, model partitioning, hybridization with recurrent or state-space modules, and block-level quantization or streaming. Recent advances achieve near-infinite context scaling, high-throughput training, low-latency streaming, and sublinear memory footprints, enabling previously intractable sequence modeling workloads across modalities and application domains [2310.01889][2305.19370][1911.02972][2306.09539][2203.07852][2204.08920][2304.01483][2601.09040][1811.03115].

Source: https://www.emergentmind.com/topics/blockwise-transformers