---
title: State-Space Models with Transfer State
url: https://www.emergentmind.com/topics/state-space-models-with-transfer-state
type: topic
---

# State-Space Models with Transfer State

State-Space Models with Transfer State are state-space architectures in which the hidden state computed on one temporal segment is explicitly carried into the next segment, so that long sequences can be processed incrementally without resetting memory. In the strict sense introduced for dense video captioning, the method called **State-Space Models with Transfer State (STS)** combines the long-sequence and recurrent properties of SSMs by processing a video in snippets, computing the SSM state for one snippet, transferring that state to the next snippet, and repeating this over the full video; the paper positions this as a way to extend SSMs to “arbitrarily long videos” while enabling online dense video captioning [2509.03426]. Related work uses the same underlying idea under different names, notably temporal hidden-state propagation in video stylization and style-conditioned recurrent fusion in image style transfer, even when the exact phrase “transfer state” is not used [2503.12291] [2409.10385].

## 1. Definition and mathematical formulation

In STS, the starting point is the standard continuous-time SSM
\[
h'(t) = Ah(t) + Bx(t), \qquad y(t) = Ch(t) + Dx(t),
\]
with discretized form
\[
h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.
\]
For training-time sequence processing, the same model can be written in convolutional form
\[
y = \overline{K} * x,
\]
with kernel
\[
\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],
\]
where \(L\) is the sequence length. The key limitation identified in the STS formulation is that this convolutional usage pattern still assumes that the whole sequence is available and that its activations can be retained in memory [2509.03426].

STS changes the usage pattern rather than the underlying SSM equations. For a segment of length \(L\), it defines a hidden-state kernel
\[
\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],
\]
and, given an incoming state \(h_0\) from the previous segment, computes
\[
y = \overline{K} * x + \overline{CA^{(L - 1)}h_0},
\]
\[
h = \overline{K}_{\mathrm{hidden}} * x + \overline{A^{(L - 1)}h_0}.
\]
Here, \(\overline{K} * x\) is the contribution of the current segment, while the extra term involving \(h_0\) injects past context from earlier segments. The resulting hidden state is then passed forward as the next segment’s initial state [2509.03426].

| Work | State carrier or alternative | Primary task |
|---|---|---|
| STS [2509.03426] | SSM hidden state transferred across snippets | Dense video captioning |
| Video State-Space Fusion [2503.12291] | Temporal hidden state \(H_t\) propagated across frames | Text-driven video style transfer |
| Mamba-ST [2409.10385] | Style-driven decoder state with content-conditioned readout | Image style transfer |
| RTF [2405.06147] | No transfer state; transfer-function, state-free parallel inference | Efficient LTI SSM inference |

This suggests that “transfer state” is best understood as an execution regime in which SSM state is preserved across processing boundaries—typically chunk boundaries or frame boundaries—rather than as a new dynamical primitive.

## 2. Exactness of chunked computation

A central claim of STS is that chunking does not alter the underlying SSM computation in theory. The paper provides an equivalence argument showing that if a sequence of length \(L\) is split after \(M\) tokens, the first chunk produces a state \(h_M\), and unrolling the recurrence for the second chunk yields the same final state \(h_L\) as direct full-sequence processing. The stated conclusion is that **state transfer preserves the true SSM state**, so chunking removes the need to keep the whole video in memory without changing the computation in theory [2509.03426].

The practical consequence is that the model can process videos in smaller segments, discard processed visual tokens, keep only the current state, and continue indefinitely in principle. This directly targets two failure modes of prior long-video methods: Transformer-style quadratic scaling in sequence length, and SSM video methods that are efficient relative to Transformers but are still commonly used in a convolutional form that requires the full video sequence in memory. The same paper further notes that, although SSMs are theoretically recurrent and stateful, their state can deteriorate over very long contexts; STS is introduced specifically to make that statefulness operational for online dense captioning [2509.03426].

Because each segment inherits the accumulated state from preceding segments, STS supports online or streaming behavior. As the video arrives, each snippet is processed, outputs can be emitted at multiple time steps, future frames are not required, and the system can process arbitrarily long streams by carrying only the state forward. In the dense-captioning setting, this allows event captions to be generated on-the-fly and timestamps to be predicted locally rather than only after the full video has ended [2509.03426].

## 3. Dense video captioning architecture and empirical profile

The STS dense-video-captioning pipeline has seven stated stages: extract visual tokens from video frames using a ViT encoder; reduce token count per frame using a bottleneck Transformer; split the video into consecutive snippets; feed each snippet’s tokens through an SSM module; compute the segment’s Transfer State; pass that state to the next snippet’s SSM module; and use a local text decoder to generate captions and temporal localization outputs for each segment [2509.03426].

This architecture is designed so that the decoder at each segment has access to the accumulated context prior to that point. In the terminology of the paper, the convolutional SSM view provides efficient processing of the current segment, the recurrent SSM view provides state propagation, and STS combines both. The paper therefore frames the method as particularly suitable for online dense video captioning over long untrimmed videos, where a model must localize multiple events with start and end times and generate a caption for each event [2509.03426].

Empirically, the reported benefits are tied to scaling and online operation. The paper reports results on ViTT, YouCook2, and ActivityNet, with videos in the range of **120–315 seconds** and up to **256 frames**. An ablation on frame count shows consistent improvement as the number of frames increases, with 16 frames lower, 32 better, and 64, 128, and 256 progressively stronger. The same study states that STS achieves strong or state-of-the-art results among the compared methods on captioning metrics such as SODA, CIDEr, and METEOR, and on localization via F1 and mIoU in ablations. It also reports about **7x fewer FLOPs**, citing **STS: 1854 GFLOPs** and **SDC: 12800 GFLOPs**, and observes that using the **last token per segment** performs especially well [2509.03426].

A plausible implication is that Transfer State is not merely a memory-saving device. In this formulation it is also the mechanism by which local decoding remains conditioned on earlier video context despite snippet-level processing.

## 4. Transfer-state mechanisms in stylization models

A closely related mechanism appears in text-driven video style transfer. In the extension of StyleMamba from images to videos, the model maintains a temporal hidden state \(H_t\) and updates it by
\[
H_t = \phi(H_{t-1}, F_t, S),
\]
with
\[
\phi(H_{t-1}, F_t, S) = \mathrm{SSM}(a \cdot H_{t-1} + B \cdot F_t + \gamma \cdot S),
\]
where \(F_t\) is the frame content feature, \(S\) is the text-derived style tensor, and \(a\), \(B\), and \(\gamma\) are learnable scalars balancing the contributions of previous state, current content, and style embedding. The model processes frames sequentially: each frame \(I_t\) is encoded to \(F_t \in \mathbb{R}^{h \times w \times c}\), the prompt is encoded by a CLIP- or BERT-like text encoder into \(e_p \in \mathbb{R}^d\) and projected to \(S \in \mathbb{R}^{h \times w \times c}\), the SSM fusion module combines \(F_t\), \(S\), and \(H_{t-1}\) to obtain \(H_t\), and a decoder outputs the stylized frame \(\hat{I}_t\) [2503.12291].

Temporal coherence is reinforced by two explicit regularizers. The **Temporal Masked Directional Loss** \(L_{\mathrm{TMD}}\) extends masked directional loss to video by randomly masking entire frames or spatiotemporal patches:
\[
L_{\mathrm{TMD}} = \sum_{t=1}^{T} \sum_{(x,y)\in M_t} \left\| \hat{I}_t - V(x,y)\,G(\hat{I}_t; p) \right\|,
\]
where \(G(\hat{I}_t; p)\) is described as CLIP-based gradient guidance toward the prompt \(p\). The **Temporal Second-Order Loss** \(L_{\mathrm{TSO}}\) is
\[
L_{\mathrm{TSO}} = \sum_{t=2}^{T} \sum_{(x,y)} \left\| f(x,y)_{t-2} - 2f(x,y)_{t-1} + f(x,y)_t \right\|^2,
\]
and the overall objective is
\[
L_{\text{total}} = \lambda_1 L_{\text{content}} + \lambda_2 L_{\text{dir}} + \lambda_3 L_{\mathrm{TMD}} + \lambda_4 L_{\mathrm{TSO}}.
\]
On DAVIS the reported result is **T-SSIM \(= 0.903\)**, **CLIP Score \(= 0.684\)**, **Model Size \(= 190\) MB**; on UCF101 it is **T-SSIM \(= 0.915\)**, **CLIP Score \(= 0.693\)**, **Model Size \(= 190\) MB**. Removing \(L_{\mathrm{TSO}}\) reduces T-SSIM from \(0.903\) to \(0.877\), removing \(L_{\mathrm{TMD}}\) hurts abrupt scene changes, and using a single-layer SSM weakens temporal dependency modeling [2503.12291].

In image style transfer, Mamba-ST instantiates another transfer-state pattern. Its decoder sets
\[
B = \mathrm{Lin}_B(s), \quad \Delta = \mathrm{Lin}_\Delta(s), \quad C = \mathrm{Lin}_C(x),
\]
where \(s\) is the style embedding and \(x\) is the content embedding, and then performs
\[
h_k = \bar{A} h_{k-1} + \bar{B} s_k, \qquad y_k = C h_k.
\]
The decoder therefore uses the style stream to control state dynamics and the content stream to control readout. The paper explicitly frames this as cross-attention-like fusion without actual cross-attention and without AdaLN. It also randomly shuffles style embeddings before they enter ST-VSSM so that the state retains mainly high-level stylistic statistics rather than content layout [2409.10385].

The reported quantitative results at \(512 \times 512\) are **ArtFID: 27.11**, **FID: 16.75**, **LPIPS: 0.53**, **CFSD: 0.33**, **Time: 24.70 s for 800 images**, and **Memory: 1414 MiB**. Ablations show that using content instead of style as the selective-scan input degrades performance, four scan directions perform best, state dimension 16 is best, and removing random shuffle at inference slightly improves FID but worsens LPIPS and causes content leakage from the style image [2409.10385].

These stylization models do not define STS in the dense-captioning sense, but they do implement the same structural motif: a hidden state functions as a carrier of information that must persist across steps rather than being recomputed independently at each step.

## 5. Distinction from transfer functions and state-free inference

The phrase “transfer state” should be distinguished from **transfer function**. In the Rational Transfer Function approach, the SSM
\[
x_{t+1}=Ax_t+Bu_t,\qquad y_t=Cx_t+h_0u_t
\]
is represented by its \(Z\)-domain transfer function
\[
H(z)=h_0+C(zI-A)^{-1}B,
\]
or equivalently
\[
H(z)=h_0+\frac{b_1z^{-1}+b_2z^{-2}+\cdots+b_nz^{-n}}{1+a_1z^{-1}+a_2z^{-2}+\cdots+a_nz^{-n}}.
\]
That paper is explicit that the transfer function is a complete representation of the LTI SSM but **does not add a new “transfer state,” nor does it replace the hidden state**. Its contribution is a **state-free parallel inference** algorithm that computes the kernel spectrum directly from the transfer function using FFTs, with \(O(\ell\log\ell)\) time and \(O(\ell)\) space for the parallel path. The reported results include on average about a **35% training speed improvement over S4 layers** on Long Range Arena, an average score of **87.71** versus **86.69** for S4 and **86.72** for S4D, and **perplexity 18.0** on WikiText103 for Hyena-RTF versus **18.5** for Hyena and **18.3** for Hyena-S5 [2405.06147].

A related but distinct line of work studies fast application of the transfer operator
\[
H(z)=C(I-z^{-1}A)^{-1}B + D
\]
in the time domain. The proposed factorization
\[
(I - z^{-1}A)^{-1} = \prod_{n=0}^{\infty} \left(I + (z^{-1}A)^{2^n}\right)
\]
yields an unconditionally stable cascade implementation based on a matrix polynomial of degree \(2^{N+1}-1\). The paper states that computing \(L\) states requires **no more than \(2L\) matrix-vector multiplications**, and gives the example that \(N=14\) produces degree \(2^{15}-1=32767\) using only 15 matrix-vector multiplications by powers of \(A\) [2411.17729].

From a broader systems perspective, even classical recurrent models can be interpreted through this lens. Echo State Networks are recast as nonlinear SSMs
\[
x_{t+1}=f(x_t,u_t)+\omega_t,\qquad y_t=g(x_t,u_t)+\nu_t,
\]
with canonical leaky update
\[
x_{t+1} = (1-\lambda)x_t + \lambda\sigma(Wx_t + Uu_t + b), \qquad y_t = Cx_t + d.
\]
After small-signal linearization they admit an LTI surrogate with transfer function
\[
H(z)=C(I-z^{-1}A)^{-1}B + D,
\]
and after lifting they can be written as a linear SSM in an augmented state \(z_t=\phi(x_t)\). The paper summarizes this reframing as **ESP = ISS / contraction**, **leak = time constant**, **spectral scaling = pole placement**, and **linearization/lift = transfer function + impulse kernel** [2509.04422].

This suggests that transfer-state models and transfer-function models are complementary rather than synonymous: one emphasizes propagated state across processing boundaries, the other emphasizes equivalent input-output operators or lifted realizations.

## 6. Limitations, misconceptions, and scope

A common misconception is that any use of SSMs on long sequences automatically provides durable long-range memory. The STS formulation argues otherwise: standard SSM video methods are theoretically recurrent, but in practice they are usually run in a convolutional form that still requires the full input sequence, and their state can deteriorate over very long contexts. Transfer State is proposed specifically to operationalize recurrence across snippets while retaining manageable memory use [2509.03426].

Another misconception is that “transfer” in SSM literature always means hidden-state transfer. In video stylization, the temporal hidden state is explicitly propagated, but temporal alignment is learned implicitly rather than enforced by an explicit optical-flow warping module; the paper presents this as simpler and lighter, while also making temporal alignment implicitly learned rather than geometrically enforced [2503.12291]. In Mamba-ST, the limitation appears elsewhere: the paper notes patch discontinuities or gaps and non-uniform stylization across patches, attributing them to a recurrence/state-compression limitation inherited from RNN-like models [2409.10385].

A further distinction arises in cross-modal knowledge transfer. Event USKT uses a U-shaped State Space Model Knowledge Transfer framework for Event-to-RGB transfer, but the paper states that this is **not “state transfer” in the strict classical sense of copying hidden states between two trained sequence models**. Instead, event streams are converted into an RGB-compatible tensor \(\mathbf{X}_{\text{USKT}} \in \mathbb{R}^{3 \times 224 \times 224}\) so that a pre-trained ResNet can be reused; the reported gains with ResNet50 are **+0.95%** on DVS128 Gesture, **+3.57%** on N-Caltech101, and **+2.90%** on CIFAR-10-DVS [2411.15276].

Taken together, these results delimit the concept. In the narrow sense of STS, transfer state is the preserved SSM hidden state passed across snippets for exact chunked computation and online prediction. In a broader architectural sense, it denotes recurrent state propagation used as a carrier of temporal, stylistic, or contextual information across processing boundaries. It does not, however, subsume transfer-function parameterizations, state-free inference, or generic representation-transfer pipelines, even though all are adjacent developments in the contemporary SSM literature.

Source: https://www.emergentmind.com/topics/state-space-models-with-transfer-state