Timesteps of Mamba Align with Human Reading Times

Abstract: This study demonstrates an alignment of per-word processing time in a popular state-space LLM Mamba and human readers. In Mamba, the recurrent state transition at each layer conceptually takes some duration of time, the discretization timestep $Δ_t$, determined dynamically in response to the input. Using a naturalistic reading dataset, we show that the per-word timestep from Mamba is a significant predictor of human reading times, and remains significant even when known predictors such as GPT-2 surprisal are controlled for. We further suggest, through formal analysis of Mamba's architecture and internal dynamics, that Mamba can serve as a new, valuable lens to look at human real-time language processing with ever-updated memory, because it allows us to look at how each module (layer) weighs short- and long-term information retention, and how noise may interact with dynamic, continuous memory representation. Code is available online.

• The paper shows that Mamba’s input-dependent timestep predicts human reading times, reaching R² = 0.22 and matching GPT-2 surprisal while explaining unique variance beyond surprisal and word-level controls.
• The paper finds that timestep patterns reflect linguistic structure and memory demands, with sentence-boundary peaks and layers 16–17 supporting long-range retrieval in passkey experiments.
• The paper proposes Mamba as an algorithmic model of incremental reading in which larger timesteps signal stronger state updates and greater transition uncertainty, while noting that causal mechanisms and generalization beyond Mamba remain unresolved.

This paper reports a word-level alignment between the internal dynamics of Mamba, a selective state-space LLM, and human reading times. The authors' central claim is that Mamba's input-dependent discretization timestep $\Delta_t$ — which governs how far each layer's hidden state evolves when processing a token — functions as an implicit per-word processing time, and that this quantity predicts human word-by-word reading times over and above established predictors such as GPT-2 surprisal. The paper combines empirical reading-time modeling with formal analyses of Mamba's memory dynamics, proposing the architecture as an algorithmic-level lens on human incremental language processing.

Background: selective state-space models and reading-time modeling

Mamba processes text recurrently, word by word, like an RNN, but its hidden-state update contains no nonlinearities, enabling parallel-scan training (2606.29904). Each of its layers implements a discretized continuous-time state-space model:

$$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$

with $\overline{A}_t = \exp(\Delta_t A)$ and $\overline{B}_t = \Delta_t W_B \bm{x}_t$, where the timestep $\Delta_t$ is itself computed from the input via a softplus. Because the transition matrix $\exp(A)$ is diagonal with elements in $(0,1]$, a large $\Delta_t$ drives $\overline{A}_t$ toward zero and scales up $\overline{B}_t$: the layer forgets prior memory and writes the current input strongly. A small $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$0 preserves memory and discards input. The timestep thus acts as a gate over retention versus update.

The linking hypothesis departs from the standard surprisal framework, in which processing cost is proportional to $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$1 (2606.29904). Prior work has shown that engineering-superior LMs are not necessarily better reading-time predictors, motivating the search for alternative linking variables at the algorithmic level. Interpreting $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$2 as processing time is natural: a larger timestep means the model evolves its continuous-time dynamics longer while holding the same state, functionally analogous to a reader dwelling longer on a cognitively demanding word.

Experimental design

The authors evaluated $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$3 (the sum of the $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$4-dimensional timestep vector per layer, aggregated to word level by maximum over subwords) against two naturalistic datasets: Natural Stories, with self-paced reading times from 181 participants over 10,245 words, and OneStop (ordinary reading subset), with eye-tracking first-pass times and regression-path durations from 180 participants over both advanced and elementary article versions. First-pass and regression-path measures were chosen because, like $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$5, they reflect processing before right-context information is accessed.

Reading times were modeled by linear regression on mean residual log reading times (after removing by-participant random effects), with 10-fold cross-validation repeated 50 times; significance of a predictor's contribution to per-word MSE was assessed by permutation tests with Holm-corrected $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$6-values across layers. Control variables included word length, corpus frequency, sentence position, Mamba surprisal, and GPT-2 surprisal, each entered at the current and two preceding words to capture spillover. Four regression configurations were tested, from an intercept-only baseline up through low-level controls plus GPT-2 surprisal. Experiments used Mamba-130M (24 layers) and Mamba-2.8B (64 layers).

Main results

The timestep $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$7 from most layers significantly predicts reading times as a sole predictor, and a subset of layers remains significant under full controls. The strongest single-layer result reaches $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$8 (layer 41 of Mamba-2.8B on Natural Stories), comparable to $$\bm{h}_t = \overline{A}_t \bm{h}_{t-1} + \overline{B}_t x_{ti}, \qquad y_{ti} = C_t \bm{h}_t + D x_{ti}$$9 for GPT-2 surprisal on the same data. On Natural Stories with Mamba-130M, layers 16 and 17 stand out: layer 17 alone achieves $\overline{A}_t = \exp(\Delta_t A)$0 against an intercept-only baseline and retains significant predictive power ($\overline{A}_t = \exp(\Delta_t A)$1MSE $\overline{A}_t = \exp(\Delta_t A)$2, $\overline{A}_t = \exp(\Delta_t A)$3) even when GPT-2 surprisal and low-level variables are included. Across all dataset/model combinations, multiple layers contribute independently — e.g., 20 layers of Mamba-2.8B are independently predictive for Natural Stories under GPT-2 surprisal controls, and 61 layers are significant for OneStop first-pass time. These results indicate that $\overline{A}_t = \exp(\Delta_t A)$4 captures variance in human reading behavior not explained by predictability-based or low-level predictors.

Follow-up correlational analyses show that $\overline{A}_t = \exp(\Delta_t A)$5 is systematically sensitive to linguistic structure: across layers it peaks at sentence beginnings, decreases with within-sentence word position, and increases with syntactic tree distance between adjacent words — consistent with large state updates at constituent boundaries. Layers 16 and 17 deviate from this pattern, correlating with story-level position rather than sentence-local features, suggesting a division of labor in which these layers maintain discourse-level information.

Probing long-range dependencies in the predictive layers

To test whether layers 16 and 17 genuinely carry long-range information, the authors knocked out individual layers' SSM blocks (zeroing $\overline{A}_t = \exp(\Delta_t A)$6) during a passkey retrieval task with inputs of 600–1,000 tokens. Knocking out layers 0, 7, 13, 16, or 17 sharply degrades retrieval accuracy; layer 16 drops to the 10% range even at 800 tokens, and layers 0 and 17 yield 0% accuracy even at 600 tokens. This corroborates the reading-time results in identifying these layers as critical for transmitting information across long contexts, although the authors caution that knockout sensitivity does not establish specialization for long-range dependencies — the layers may simply be essential for text processing generally.

A further discrepancy remains unexplained: layers 16 and 17 are strongly predictive of Natural Stories self-paced reading but not of OneStop eye-tracking measures. The authors speculate that self-paced reading, which forces memory-based retrieval of context, may amplify sensitivity to long-range information relative to eye-tracking, where readers can physically re-inspect text, but they explicitly flag this as speculation.

Mamba as a model of memory-constrained processing

Two formal analyses extend the empirical findings. First, the eigenvalues of the diagonal transition matrix $\overline{A}_t = \exp(\Delta_t A)$7 quantify per-layer memory retention: eigenvalues near one yield $\overline{A}_t = \exp(\Delta_t A)$8 regardless of $\overline{A}_t = \exp(\Delta_t A)$9 (long retention), while eigenvalues near zero force rapid forgetting. Layers 16 and 17 have eigenvalue distributions shifted toward 0.5 relative to the model-wide distribution, which peaks near zero — independently confirming their long-timescale character. The authors note the parallel to hierarchical timescale organization in neural populations, though they do not test it.

Second, the authors derive that $\overline{B}_t = \Delta_t W_B \bm{x}_t$0 corresponds to state-transition uncertainty. Introducing a Gaussian noise term into the continuous-time dynamics and applying zero-order-hold discretization, the conditional differential entropy of the transition is

$\overline{B}_t = \Delta_t W_B \bm{x}_t$1

which increases monotonically in $\overline{B}_t = \Delta_t W_B \bm{x}_t$2. Mamba's learned behavior of enlarging $\overline{B}_t = \Delta_t W_B \bm{x}_t$3 at sentence beginnings — precisely where prediction from prior context is weakest — is therefore consistent with larger timesteps encoding greater transition uncertainty. The authors stress two caveats: this interpretation requires the injected noise term and does not follow from the original deterministic model, and the uncertainty in question concerns the post-update state, not the uncertainty about upcoming input studied in the surprisal literature. They suggest this framing — noise accumulating in a continuous context representation rather than deleting discrete words — as a distinct perspective on noisy-channel models of language processing.

Relation to prior work

The study differs from Transformer-based cognitive modeling that converts intermediate representations into surprisal-like values: here the internal component $\overline{B}_t = \Delta_t W_B \bm{x}_t$4 enters the regression directly as a processing-time variable (2606.29904). Intervention methods for locating factual-information pathways in Mamba are related to the knockout analysis but have addressed only short factual sentences, not story-level text. Finally, the sentence-boundary peaks in $\overline{B}_t = \Delta_t W_B \bm{x}_t$5 resonate with findings that LMs constrain information integration at clause and sentence boundaries; however, layer 17 — the strongest Natural Stories predictor — shows no such boundary peaks, suggesting that cross-sentence information transmission is an essential component of the reading-time alignment, in contrast to strictly within-sentence integration accounts.

Limitations and open questions

The paper is candid about several gaps. It does not explain how the model learns to enlarge $\overline{B}_t = \Delta_t W_B \bm{x}_t$6 at sentence boundaries or at syntactically distant word pairs; establishing this would require training-corpus statistics and learning-dynamics analysis, which the authors leave out of scope. The knockout evidence for layer 17's long-range role does not by itself establish long-range dependency processing as the reason $\overline{B}_t = \Delta_t W_B \bm{x}_t$7 predicts reading times — a principled mechanistic explanation for the alignment is still lacking. All experiments use Mamba exclusively, so the conclusions do not transfer to other architectures; the authors justify this by noting Mamba is the only architecture whose discretization timestep is dynamically input-determined in this way. Why different layers predict reading times in self-paced versus eye-tracking paradigms also remains open.

Conclusion

This paper establishes that Mamba's input-dependent discretization timesteps, interpreted as per-word processing time, are significant predictors of human reading times across self-paced and eye-tracking datasets, with the best layers matching GPT-2 surprisal in explanatory power and contributing unique variance under full controls. Combined with the eigenspectrum analysis of per-layer memory retention and the formal link between timesteps and state-transition uncertainty, the results position selective SSMs as a tractable algorithmic-level model of incremental comprehension under continuously updated, lossy memory. The open questions — the learning origin of timestep peaks, the causal role of long-range dependencies, and generalization beyond Mamba — define the immediate agenda for this line of work.