---
title: 'Spiralformer: Dual Transformer Architectures'
url: https://www.emergentmind.com/topics/spiralformer-11185ebe-f8db-4910-8b09-983173c2313c
type: topic
---

# Spiralformer: Dual Transformer Architectures

Searching arXiv for the specified Spiralformer papers to ground the article in the latest paper records.
Searching arXiv for 2510.00982 and 2602.11698.
The term **Spiralformer** designates two distinct Transformer-derived architectures introduced in 2025–2026 under nearly identical names. In streaming automatic speech recognition, **“Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting”** defines a block-processing encoder that reduces encoding latency through **circular layer skipping** and **early exiting** [2510.00982]. In large language modeling, **“SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion”** defines a **recursive (looped) Transformer** whose central mechanism is **multi-resolution recursion**, in which loop iterations operate at progressively finer sequence resolutions [2602.11698]. The shared name reflects a recurring “spiral” computation pattern, but the two systems target different problem settings, use different computational primitives, and optimize different efficiency–quality trade-offs.

## 1. Nomenclature and scope

The speech-recognition Spiralformer is a **low latency encoder for streaming speech recognition with circular layer skipping and early exiting** [2510.00982]. Its design is explicitly motivated by **blockwise processing** in streaming ASR, where each block contains left context \(N_l\), current chunk \(N_c\), and right context \(N_r\), and where reducing token emission delay typically requires reducing \(N_c\), which in turn raises computational cost [2510.00982].

The language-model SpiralFormer is a **looped Transformer** that executes recurrence under a **multi-resolution recursion schedule** [2602.11698]. Its motivation is different: prior recursive Transformers repeatedly apply shared layers at full sequence resolution, whereas SpiralFormer varies the resolution across loop iterations so that early iterations operate on compressed representations and later iterations refine at token level [2602.11698].

A common misconception is that these papers describe the same architecture adapted across modalities. The available descriptions do not support that interpretation. The ASR model combines **layer dropping**, **cyclic layer selection**, cached intermediate outputs, and **early exiting** for blockwise speech encoding [2510.00982]. The language-model architecture instead combines **downscaling**, a **shared Transformer core**, **upscaling**, **causal right-shift**, and a **topology operator** such as **Anchor** or **MeSH** within a recursive computation loop [2602.11698]. This suggests that “Spiralformer” is presently best understood as a shared naming motif rather than a single unified model family.

## 2. Spiralformer in streaming speech recognition

In the ASR setting, streaming encoders based on Transformers or Conformers commonly use blockwise processing. The central latency quantity described in the paper is that **system-wide latency = \(N_c + N_r\)**, so smaller \(N_c\) enables more frequent output and lower latency, but also increases compute because blocks are emitted more often [2510.00982].

Spiralformer addresses this latency–compute trade-off by combining two mechanisms. The first is **Circular Layer Skipping**: instead of computing all \(I\) encoder layers for every block, only a subset is computed per block in a cyclic fashion, with a fixed **pitch** \(p\), and the identity of the computed layers shifts from block to block [2510.00982]. The paper defines the shift index for block \(b\) as
\[
s = (b-1) \bmod p
\]
and the computed layer set as
\[
\mathcal{S}_s = \{1+s, 1+p+s, 1+2p+s, \ldots\}.
\]
For each block, only
\[
\left\lfloor \frac{I}{p} \right\rfloor
\]
layers are computed [2510.00982]. Over multiple blocks, all layers are periodically computed for each input frame, completing the “spiral.”

The second mechanism is **Early Exiting**. For each block, output is taken from the last layer that was actually computed, rather than always from the final encoder layer [2510.00982]. This allows token emission without waiting for the full stack. Intermediate outputs are supervised during training through additional loss terms, summarized as
\[
\mathcal{L} = \mathcal{L}_H + \sum_{s=0}^{p-1} \mathcal{L}_H^s.
\]

The encoder recurrence for ordinary block processing is written as
\[
Z_b^{(i)} = \text{enc}^{(i)}(Z_b^{(i-1)}),
\]
whereas Spiralformer computes, for block \(b\) and layer \(i\) in the selected set,
\[
Z_b^{(i)} =
\begin{cases}
\text{enc}^{(i)}(X^{t \in b}), & i \leq p \\
\text{enc}^{(i)}(Z_b^{(i-p)}), & i > p.
\end{cases}
\]
With cached computation, the formulation becomes
\[
Z_b^{(i)} =
\begin{cases}
\text{enc}^{(i)}\big(X^{t \in b} + \tilde{Z}_{b-1}^{(i-1)}\big), & i \leq p \\
\text{enc}^{(i)}\big(Z_b^{(i-p)} + \tilde{Z}_{b-1}^{(i-1)}\big), & i > p,
\end{cases}
\]
where \(\tilde{Z}_{b-1}^{(i-1)}\) is the cached output for the previous block’s lower layer [2510.00982]. The output for a block is then taken **from the last computed layer in the set**, denoted
\[
H_b^s.
\]

Within the paper’s framing, the essential significance of Spiralformer is that frequent small-shift block emissions can be made computationally manageable by computing only part of the encoder at each block, while cached context preserves dependency structure across blocks [2510.00982]. A plausible implication is that the design moves part of the latency optimization problem from transducer emission policy into encoder scheduling itself.

## 3. Quantitative behavior in speech recognition

The paper reports results on **LibriSpeech** and **CSJ** [2510.00982]. On **LibriSpeech**, the comparison highlighted in the summary is between a baseline Conformer configuration and Spiralformer variants. The **Standard Baseline (B2)**, with **12 layers** and \(N_c=8\), has **Max theoretical latency: 640 ms**, **RTF: 0.15**, **WER (test-clean/other): 3.4% / 8.6%**, and **SWD (System Word emission Delay), P50: 589 ms** [2510.00982]. The **Spiralformer S3** configuration with \(N_c=2, p=2\) has **Max theoretical latency: 400 ms**, **RTF: 0.20**, **WER: 3.6% / 9.1%**, and **SWD(P50): 462 ms** [2510.00982]. The paper describes this as a **37.5% reduction** in maximum theoretical latency and a **21.6% reduction** in SWD(P50) relative to the baseline, while maintaining similar computational cost and word error rates [2510.00982].

The same summary reports that **S2 (\(N_c=1, p=4\)) achieves SWD(P50): 389 ms**, with the qualification that it comes with **higher RTF/less WER benefit** [2510.00982]. Table 2 is described as reporting median and 90th-percentile delays, including **FWD(P50) 420 ms** for Spiralformer S2 versus **990 ms** for B2, and **SWD(P50) 462 ms** for Spiralformer S3 versus **589 ms** for B2 [2510.00982].

On **CSJ**, the reported baseline **B3 (\(N_c=4\))** has **SWD(P50): 385 ms**, whereas **Spiralformer S3** has **SWD(P50): 358 ms**, a **7% reduction**, while maintaining similar Character Error Rates [2510.00982].

The summary also states that **RTF for Spiralformer with aggressive skipping is lower or similar to baseline, even with much smaller chunk sizes (\(N_c\))**, and that **First Word Delay and SWD are consistently reduced across Spiralformer models versus comparable compute or accuracy baselines** [2510.00982]. An ablation finding is that **training from scratch can increase emission delays**, and that **the best emission timings arise from fine-tuning from a pretrained blockwise encoder** [2510.00982].

| Configuration | Key values | Source |
|---|---|---|
| Baseline B2 | 640 ms max latency; RTF 0.15; WER 3.4% / 8.6%; SWD(P50) 589 ms | [2510.00982] |
| Spiralformer S3 | 400 ms max latency; RTF 0.20; WER 3.6% / 9.1%; SWD(P50) 462 ms | [2510.00982] |
| CSJ comparison | B3 SWD(P50) 385 ms; S3 SWD(P50) 358 ms | [2510.00982] |

These figures support the paper’s specific claim that encoding latency in blockwise streaming ASR can be reduced without a large accuracy penalty by restructuring how encoder depth is distributed across blocks rather than evaluating the full depth at every step [2510.00982].

## 4. SpiralFormer as a recursive language model

The 2026 SpiralFormer is introduced as a **recursive (or looped) Transformer architecture** that uses **multi-resolution recursion** to improve **parameter and compute efficiency** and to enable learning of **hierarchical dependencies** [2602.11698]. The motivating contrast is with standard recursive Transformers, which repeatedly apply a shared Transformer block at **full sequence resolution** during every recurrence [2602.11698].

The architecture defines, for loop iteration \(t\), a resolution \(r_t \in (0,1]\) and effective sequence length
\[
L_t = \lfloor r_t L \rfloor,
\]
where \(L\) is the token sequence length [2602.11698]. The schedule is typically **coarse-to-fine**, for example
\[
r_t = 2 r_{t-1}, \text{ with } r_0 = 1/8 \text{ or } 1/16,
\]
so early loops operate at \(1/8\) or \(1/16\) resolution and the final loop reaches full token resolution [2602.11698].

One loop iteration comprises five operations. First, **Downscale** compresses token-level hidden states \(\bm{h}^{(t)} \in \mathbb{R}^{L \times d}\) into chunk-level latents \(\bm{z}^{(t)} \in \mathbb{R}^{L_t \times d}\):
\[
\bm{z}^{(t)} = \mathcal{S}^{(t)}_{\downarrow}(\bm{h}^{(t)}; r_t).
\]
This is typically implemented as **self-aggregation**, a learnable weighted mean within each chunk:
\[
\bm{z}^{(t)}_j = \sum_{i \in \mathcal{I}_{t,j}} \alpha^{(t)}_{j,i} \cdot \bm{h}^{(t)}_i.
\]

Second, the model applies the shared **loop core** to the chunk-level sequence:
\[
\hat{\bm{z}}^{(t)} = f_{\text{loop}}(\bm{z}^{(t)}).
\]

Third, **Upscale** maps chunk-level updates back to token positions:
\[
\bm{u}^{(t)}_i = \lambda_t \cdot \beta^{(t)}_{\pi_t(i), \rho_t(i)} \cdot \hat{\bm{z}}^{(t)}_{\pi_t(i)},
\]
where \(\bm{\beta}^{(t)}\) is predicted by a lightweight router \(\mathcal{B}^{(t)}\), and \(\lambda_t = \sqrt{g_t}\) [2602.11698].

Fourth, a **causal right-shift** enforces strict autoregressive causality:
\[
\widetilde{\bm{u}}^{(t)}[i] =
\begin{cases}
0, & i < s_t \\
\bm{u}^{(t)}[i - s_t], & i \geq s_t.
\end{cases}
\]
By default, \(s_t = g_t - 1\), producing a single-token overlap between generating and receiving chunk [2602.11698].

Fifth, the running state is updated using a topology operator \(\mathcal{U}\):
\[
(\bm{h}^{(t+1)}, \mathcal{H}^{(t+1)}) = \mathcal{U}( \widetilde{\bm{u}}^{(t)}, \bm{h}^{(t)}, \mathcal{H}^{(t)}; t ).
\]
The paper identifies two topology choices: **Anchor**, which is stateless and adds to a fixed anchor state, and **MeSH**, a learned memory buffer that manages state across iterations [2602.11698].

The overall computation is summarized as **Pre-block → \(T\) multi-resolution loop steps → post-block** [2602.11698]. The intended effect is that early iterations perform low-cost global interaction on compressed representations, while later iterations perform fine-grained local refinement on token-level states.

## 5. Hierarchical specialization and empirical scaling in language modeling

A central claim of SpiralFormer is that **multi-resolution recursion enables the model to learn hierarchical dependencies by inducing iteration-wise functional specialization across different scales** [2602.11698]. The paper reports **probing evidence** for this claim. Specifically, **key-marginal entropy** of attention heads decreases with higher resolution, meaning attention becomes more selective at finer scales, and **Local Attention Mass (LAM)** increases at higher resolution, indicating that later iterations focus more on local neighborhoods [2602.11698]. The paper contrasts this with **full-resolution looping**, which does not exhibit the same systematic cross-loop specialization.

The empirical evaluation is conducted on the **Pythia suite (160M, 410M, 1B, 1.4B parameter scales)** pretrained on **the Pile (250B tokens, one epoch)** [2602.11698]. The comparison includes a **Baseline** non-recursive Transformer, **LoopedFormer** with full-resolution recursion, **SpiralFormer-B**, and **SpiralFormer-L** [2602.11698].

The principal quantitative findings are stated at a high level. **SpiralFormer-B**, with the same parameter allocation as the full-resolution recursive baseline, **consistently improves validation perplexity and few-shot task accuracy while reducing FLOPs by 7–11% versus LoopedFormer** [2602.11698]. **SpiralFormer-L**, with the same parameter count as the non-looped baseline, **reduces prefill FLOPs by 3–10%** and **improves or matches perplexity and downstream accuracy** [2602.11698]. At the **1.4B** scale, the paper reports a reduction in FLOPs from **14.08T → 13.13T** and an increase in **average 5-shot accuracy from 51.93 to 54.37** [2602.11698].

The paper further states that both SpiralFormer variants define a superior **loss–compute** and **accuracy–parameter frontier** relative to both baselines, and that the **efficiency and performance gap widens at larger scale** [2602.11698]. This suggests that the architecture’s main contribution is not merely a constant-factor efficiency gain but a different scaling behavior in recursive Transformer design.

## 6. Ablations, system implications, and comparative interpretation

The language-model paper identifies several architectural choices as important. **Multiresolution scheduling (coarse-to-fine) is essential; reversing the schedule (fine to coarse) degrades performance** [2602.11698]. **MeSH topology outperforms Anchor**, indicating the importance of a learned memory across loops [2602.11698]. **Learnable self-aggregation/upscaling is better than mean pooling/uniform upscaling**, and the **causal overlap regime is best in quality**, whereas the **parallel regime (no chunk overlap) enables system-level speedups** [2602.11698].

It also states that **right-shift scheduling is provably necessary for causality with chunked latent aggregation**, and that **chunk offset** \(\omega_t\) modulates the periodic pattern and helps ensure **uniform per-token computation** [2602.11698]. In the **no-overlap regime** \(s_t \geq g_t\), low-resolution chunked computations for earlier tokens can be overlapped with high-resolution computations for the current token, enabling inference-time pipelining [2602.11698].

By contrast, the ASR Spiralformer’s deployment implications are centered on emission timing rather than autoregressive decoding pipelines. Its key finding is that frequent block emissions with small chunk shifts can be reconciled with manageable compute by **skipping layer computation in a cyclic manner** and relying on **cached intermediate outputs** plus **early exiting** [2510.00982]. Its ablation result that **fine-tuning from a pretrained blockwise encoder** yields the best emission timings further distinguishes it from the language-model SpiralFormer, whose focus is recursive scaling behavior rather than transfer from a blockwise pretrained encoder [2510.00982].

Taken together, the two Spiralformer architectures illustrate two distinct uses of a spiral computation schedule. In the ASR system, the spiral is a **cyclic traversal of encoder depth across successive blocks** [2510.00982]. In the language model, the spiral is a **coarse-to-fine traversal of sequence resolution across recursive iterations** [2602.11698]. The shared conceptual theme is staged computation under tight efficiency constraints, but the underlying state evolution, training supervision, and deployment objectives are not the same.

## 7. Position within Transformer research

The 2025 ASR Spiralformer is situated within work on **streaming speech recognition**, **Transformer-based encoders**, **Conformers**, and **block processing**, with a specific emphasis on **encoding latency of the block processing**, a dimension the abstract describes as having received relatively little attention compared with emission latency in transducers [2510.00982]. Its contribution is therefore best interpreted as an encoder-scheduling method for real-time ASR.

The 2026 SpiralFormer is situated within **recursive/looped Transformer** research and is described as building on architectures such as the **Universal Transformer**, while also connecting to work on **latent reasoning**, memory-augmented models, and efficient LLM scaling [2602.11698]. Its stated novelty is to make **multi-resolution recursion an explicit architectural primitive** [2602.11698].

Because the two papers use near-identical names, bibliographic ambiguity is likely. For precision, the ASR model is most accurately identified by the subtitle **“Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting”** [2510.00982], whereas the recursive language model is most accurately identified by **“Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion”** [2602.11698]. A plausible implication is that future usage of the term “Spiralformer” will require modality-specific disambiguation unless one of the two lineages becomes dominant.

At present, the name therefore refers not to a single canonical Transformer variant but to two technically separate proposals: one for **low-latency blockwise speech encoding** and one for **hierarchical multi-resolution recursion in looped language models**. Their joint significance lies less in a shared implementation than in a shared methodological tendency: replacing uniformly repeated full-stack computation with structured, staged computation that redistributes model work across time, depth, or resolution [2510.00982][2602.11698].

Source: https://www.emergentmind.com/topics/spiralformer-11185ebe-f8db-4910-8b09-983173c2313c