State-Space Models with Transfer State

• State-Space Models with Transfer State are architectures that transfer hidden states across segments, enabling efficient incremental processing of long sequences.
• They maintain mathematical equivalence with full-sequence computation while reducing memory overhead, making them suitable for online tasks like dense video captioning.
• The method is applied in areas such as video captioning and stylization, demonstrating improved scalability and performance with lower computational cost.

State-Space Models with Transfer State are state-space architectures in which the hidden state computed on one temporal segment is explicitly carried into the next segment, so that long sequences can be processed incrementally without resetting memory. In the strict sense introduced for dense video captioning, the method called State-Space Models with Transfer State (STS) combines the long-sequence and recurrent properties of SSMs by processing a video in snippets, computing the SSM state for one snippet, transferring that state to the next snippet, and repeating this over the full video; the paper positions this as a way to extend SSMs to “arbitrarily long videos” while enabling online dense video captioning (Piergiovanni et al., 3 Sep 2025). Related work uses the same underlying idea under different names, notably temporal hidden-state propagation in video stylization and style-conditioned recurrent fusion in image style transfer, even when the exact phrase “transfer state” is not used (Li et al., 15 Mar 2025, Botti et al., 2024).

1. Definition and mathematical formulation

In STS, the starting point is the standard continuous-time SSM

$$h'(t) = Ah(t) + Bx(t), \qquad y(t) = Ch(t) + Dx(t),$$

with discretized form

$$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$

For training-time sequence processing, the same model can be written in convolutional form

$y = \overline{K} * x,$

with kernel

$$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$

where $L$ is the sequence length. The key limitation identified in the STS formulation is that this convolutional usage pattern still assumes that the whole sequence is available and that its activations can be retained in memory (Piergiovanni et al., 3 Sep 2025).

STS changes the usage pattern rather than the underlying SSM equations. For a segment of length $L$, it defines a hidden-state kernel

$$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$

and, given an incoming state $h_0$ from the previous segment, computes

$y = \overline{K} * x + \overline{CA^{(L - 1)}h_0},$

$$h = \overline{K}_{\mathrm{hidden}} * x + \overline{A^{(L - 1)}h_0}.$$

Here, $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$0 is the contribution of the current segment, while the extra term involving $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$1 injects past context from earlier segments. The resulting hidden state is then passed forward as the next segment’s initial state (Piergiovanni et al., 3 Sep 2025).

Work	State carrier or alternative	Primary task
STS (Piergiovanni et al., 3 Sep 2025)	SSM hidden state transferred across snippets	Dense video captioning
Video State-Space Fusion (Li et al., 15 Mar 2025)	Temporal hidden state $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$2 propagated across frames	Text-driven video style transfer
Mamba-ST (Botti et al., 2024)	Style-driven decoder state with content-conditioned readout	Image style transfer
RTF (Parnichkun et al., 2024)	No transfer state; transfer-function, state-free parallel inference	Efficient LTI SSM inference

This suggests that “transfer state” is best understood as an execution regime in which SSM state is preserved across processing boundaries—typically chunk boundaries or frame boundaries—rather than as a new dynamical primitive.

2. Exactness of chunked computation

A central claim of STS is that chunking does not alter the underlying SSM computation in theory. The paper provides an equivalence argument showing that if a sequence of length $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$3 is split after $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$4 tokens, the first chunk produces a state $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$5, and unrolling the recurrence for the second chunk yields the same final state $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$6 as direct full-sequence processing. The stated conclusion is that state transfer preserves the true SSM state, so chunking removes the need to keep the whole video in memory without changing the computation in theory (Piergiovanni et al., 3 Sep 2025).

The practical consequence is that the model can process videos in smaller segments, discard processed visual tokens, keep only the current state, and continue indefinitely in principle. This directly targets two failure modes of prior long-video methods: Transformer-style quadratic scaling in sequence length, and SSM video methods that are efficient relative to Transformers but are still commonly used in a convolutional form that requires the full video sequence in memory. The same paper further notes that, although SSMs are theoretically recurrent and stateful, their state can deteriorate over very long contexts; STS is introduced specifically to make that statefulness operational for online dense captioning (Piergiovanni et al., 3 Sep 2025).

Because each segment inherits the accumulated state from preceding segments, STS supports online or streaming behavior. As the video arrives, each snippet is processed, outputs can be emitted at multiple time steps, future frames are not required, and the system can process arbitrarily long streams by carrying only the state forward. In the dense-captioning setting, this allows event captions to be generated on-the-fly and timestamps to be predicted locally rather than only after the full video has ended (Piergiovanni et al., 3 Sep 2025).

3. Dense video captioning architecture and empirical profile

The STS dense-video-captioning pipeline has seven stated stages: extract visual tokens from video frames using a ViT encoder; reduce token count per frame using a bottleneck Transformer; split the video into consecutive snippets; feed each snippet’s tokens through an SSM module; compute the segment’s Transfer State; pass that state to the next snippet’s SSM module; and use a local text decoder to generate captions and temporal localization outputs for each segment (Piergiovanni et al., 3 Sep 2025).

This architecture is designed so that the decoder at each segment has access to the accumulated context prior to that point. In the terminology of the paper, the convolutional SSM view provides efficient processing of the current segment, the recurrent SSM view provides state propagation, and STS combines both. The paper therefore frames the method as particularly suitable for online dense video captioning over long untrimmed videos, where a model must localize multiple events with start and end times and generate a caption for each event (Piergiovanni et al., 3 Sep 2025).

Empirically, the reported benefits are tied to scaling and online operation. The paper reports results on ViTT, YouCook2, and ActivityNet, with videos in the range of 120–315 seconds and up to 256 frames. An ablation on frame count shows consistent improvement as the number of frames increases, with 16 frames lower, 32 better, and 64, 128, and 256 progressively stronger. The same study states that STS achieves strong or state-of-the-art results among the compared methods on captioning metrics such as SODA, CIDEr, and METEOR, and on localization via F1 and mIoU in ablations. It also reports about 7x fewer FLOPs, citing STS: 1854 GFLOPs and SDC: 12800 GFLOPs, and observes that using the last token per segment performs especially well (Piergiovanni et al., 3 Sep 2025).

A plausible implication is that Transfer State is not merely a memory-saving device. In this formulation it is also the mechanism by which local decoding remains conditioned on earlier video context despite snippet-level processing.

4. Transfer-state mechanisms in stylization models

A closely related mechanism appears in text-driven video style transfer. In the extension of StyleMamba from images to videos, the model maintains a temporal hidden state $$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$7 and updates it by

$$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$8

with

$$h = \overline{A}h + \overline{B}x, \qquad y = \overline{C}h + \overline{D}x.$$9

where $y = \overline{K} * x,$0 is the frame content feature, $y = \overline{K} * x,$1 is the text-derived style tensor, and $y = \overline{K} * x,$2, $y = \overline{K} * x,$3, and $y = \overline{K} * x,$4 are learnable scalars balancing the contributions of previous state, current content, and style embedding. The model processes frames sequentially: each frame $y = \overline{K} * x,$5 is encoded to $y = \overline{K} * x,$6, the prompt is encoded by a CLIP- or BERT-like text encoder into $y = \overline{K} * x,$7 and projected to $y = \overline{K} * x,$8, the SSM fusion module combines $y = \overline{K} * x,$9, $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$0, and $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$1 to obtain $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$2, and a decoder outputs the stylized frame $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$3 (Li et al., 15 Mar 2025).

Temporal coherence is reinforced by two explicit regularizers. The Temporal Masked Directional Loss $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$4 extends masked directional loss to video by randomly masking entire frames or spatiotemporal patches: $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$5 where $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$6 is described as CLIP-based gradient guidance toward the prompt $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$7. The Temporal Second-Order Loss $$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$8 is

$$\overline{K} = [\overline{CB}, \overline{CAB}, \ldots, \overline{CA^{(L - 1)}B}],$$9

and the overall objective is

$L$0

On DAVIS the reported result is T-SSIM $L$1, CLIP Score $L$2, Model Size $L$3 MB; on UCF101 it is T-SSIM $L$4, CLIP Score $L$5, Model Size $L$6 MB. Removing $L$7 reduces T-SSIM from $L$8 to $L$9, removing $L$0 hurts abrupt scene changes, and using a single-layer SSM weakens temporal dependency modeling (Li et al., 15 Mar 2025).

In image style transfer, Mamba-ST instantiates another transfer-state pattern. Its decoder sets

$L$1

where $L$2 is the style embedding and $L$3 is the content embedding, and then performs

$L$4

The decoder therefore uses the style stream to control state dynamics and the content stream to control readout. The paper explicitly frames this as cross-attention-like fusion without actual cross-attention and without AdaLN. It also randomly shuffles style embeddings before they enter ST-VSSM so that the state retains mainly high-level stylistic statistics rather than content layout (Botti et al., 2024).

The reported quantitative results at $L$5 are ArtFID: 27.11, FID: 16.75, LPIPS: 0.53, CFSD: 0.33, Time: 24.70 s for 800 images, and Memory: 1414 MiB. Ablations show that using content instead of style as the selective-scan input degrades performance, four scan directions perform best, state dimension 16 is best, and removing random shuffle at inference slightly improves FID but worsens LPIPS and causes content leakage from the style image (Botti et al., 2024).

These stylization models do not define STS in the dense-captioning sense, but they do implement the same structural motif: a hidden state functions as a carrier of information that must persist across steps rather than being recomputed independently at each step.

5. Distinction from transfer functions and state-free inference

The phrase “transfer state” should be distinguished from transfer function. In the Rational Transfer Function approach, the SSM

$L$6

is represented by its $L$7-domain transfer function

$L$8

or equivalently

$L$9

That paper is explicit that the transfer function is a complete representation of the LTI SSM but does not add a new “transfer state,” nor does it replace the hidden state. Its contribution is a state-free parallel inference algorithm that computes the kernel spectrum directly from the transfer function using FFTs, with $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$0 time and $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$1 space for the parallel path. The reported results include on average about a 35% training speed improvement over S4 layers on Long Range Arena, an average score of 87.71 versus 86.69 for S4 and 86.72 for S4D, and perplexity 18.0 on WikiText103 for Hyena-RTF versus 18.5 for Hyena and 18.3 for Hyena-S5 (Parnichkun et al., 2024).

A related but distinct line of work studies fast application of the transfer operator

$$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$2

in the time domain. The proposed factorization

$$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$3

yields an unconditionally stable cascade implementation based on a matrix polynomial of degree $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$4. The paper states that computing $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$5 states requires no more than $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$6 matrix-vector multiplications, and gives the example that $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$7 produces degree $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$8 using only 15 matrix-vector multiplications by powers of $$\overline{K}_{\mathrm{hidden}} = [\overline{B}, \overline{AB}, \ldots, \overline{A^{(L - 1)}B}],$$9 (Beylkin, 2024).

From a broader systems perspective, even classical recurrent models can be interpreted through this lens. Echo State Networks are recast as nonlinear SSMs

$h_0$0

with canonical leaky update

$h_0$1

After small-signal linearization they admit an LTI surrogate with transfer function

$h_0$2

and after lifting they can be written as a linear SSM in an augmented state $h_0$3. The paper summarizes this reframing as ESP = ISS / contraction, leak = time constant, spectral scaling = pole placement, and linearization/lift = transfer function + impulse kernel (Singh et al., 4 Sep 2025).

This suggests that transfer-state models and transfer-function models are complementary rather than synonymous: one emphasizes propagated state across processing boundaries, the other emphasizes equivalent input-output operators or lifted realizations.

6. Limitations, misconceptions, and scope

A common misconception is that any use of SSMs on long sequences automatically provides durable long-range memory. The STS formulation argues otherwise: standard SSM video methods are theoretically recurrent, but in practice they are usually run in a convolutional form that still requires the full input sequence, and their state can deteriorate over very long contexts. Transfer State is proposed specifically to operationalize recurrence across snippets while retaining manageable memory use (Piergiovanni et al., 3 Sep 2025).

Another misconception is that “transfer” in SSM literature always means hidden-state transfer. In video stylization, the temporal hidden state is explicitly propagated, but temporal alignment is learned implicitly rather than enforced by an explicit optical-flow warping module; the paper presents this as simpler and lighter, while also making temporal alignment implicitly learned rather than geometrically enforced (Li et al., 15 Mar 2025). In Mamba-ST, the limitation appears elsewhere: the paper notes patch discontinuities or gaps and non-uniform stylization across patches, attributing them to a recurrence/state-compression limitation inherited from RNN-like models (Botti et al., 2024).

A further distinction arises in cross-modal knowledge transfer. Event USKT uses a U-shaped State Space Model Knowledge Transfer framework for Event-to-RGB transfer, but the paper states that this is not “state transfer” in the strict classical sense of copying hidden states between two trained sequence models. Instead, event streams are converted into an RGB-compatible tensor $h_0$4 so that a pre-trained ResNet can be reused; the reported gains with ResNet50 are +0.95% on DVS128 Gesture, +3.57% on N-Caltech101, and +2.90% on CIFAR-10-DVS (Lin et al., 2024).

Taken together, these results delimit the concept. In the narrow sense of STS, transfer state is the preserved SSM hidden state passed across snippets for exact chunked computation and online prediction. In a broader architectural sense, it denotes recurrent state propagation used as a carrier of temporal, stylistic, or contextual information across processing boundaries. It does not, however, subsume transfer-function parameterizations, state-free inference, or generic representation-transfer pipelines, even though all are adjacent developments in the contemporary SSM literature.