TS-Mamba: Efficient Online Video Super-Resolution

• The paper demonstrates TS-Mamba's main contribution as an online VSR architecture that uses trajectory-aware token selection with shifted SSMs to efficiently aggregate long-term spatio-temporal information from past frames.
• The methodology combines Hilbert scanning for locality preservation and a token-based aggregation strategy, achieving state-of-the-art performance with a 22.7% MACs reduction.
• Empirical results on datasets like REDS4 and Vid4 confirm TS-Mamba’s effectiveness in reconstructing high-resolution frames under real-time, causal constraints.

Searching arXiv for the cited TS-Mamba and closely related papers to ground the article. Trajectory-aware Shifted SSMs (TS-Mamba) is an online video super-resolution (VSR) architecture that combines trajectory-aware token selection with shifted state space modeling to aggregate long-term spatio-temporal information under a causal constraint, using only temporally previous frames to reconstruct the current high-resolution frame (Zhu et al., 14 Aug 2025). In the cited literature, the term also appears as a broader design pattern that combines trajectory-aware alignment, shifted local context aggregation, and selective state space model (SSM) propagation; in that sense, Gather-Scatter Mamba formalizes a related “Trajectory-aware Shifted SSMs (TS-Mamba)” pipeline for VSR through alignment-aware propagation and shifted local aggregation (Ko et al., 1 Oct 2025).

1. Problem setting and research motivation

Online VSR aims to reconstruct the current high-resolution frame from its low-resolution input and only temporally previous frames, which imposes a causal constraint and is motivated by real-time scenarios such as live conferencing and streaming, where latency and computational complexity must remain low (Zhu et al., 14 Aug 2025). A central technical difficulty is temporal alignment and aggregation. Most prior online methods align the current frame to a single neighboring previous frame with lightweight motion compensation, which restricts modeling to short-term dependencies and leaves long-range temporal information underexploited.

TS-Mamba is positioned against two established limitations. First, extending online VSR to longer temporal spans typically increases complexity sharply, jeopardizing real-time deployment. Second, while SSMs and Mamba provide linear-time sequence modeling with a global receptive field, direct application to vision requires additional mechanisms to preserve spatial continuity and select relevant temporal evidence. TS-Mamba addresses this by combining long-term trajectory modeling, token selection, and shifted scanning (Zhu et al., 14 Aug 2025).

The paper characterizes TS-Mamba as the first SSM-based online VSR architecture that aggregates long-term spatio-temporal information from multiple previous frames at the token level. Its headline empirical claims are state-of-the-art performance in most cases against six online VSR benchmark models and over $22.7\%$ complexity reduction in MACs (Zhu et al., 14 Aug 2025). This suggests that the contribution is not merely substituting attention with Mamba, but redefining the unit of temporal aggregation from whole aligned frames to trajectory-linked tokens.

2. Core formulation: trajectories, token selection, and reconstruction

TS-Mamba operates on a current low-resolution frame $I^{LR}_t$ and a buffer of $T$ temporally previous low-resolution frames $\{I^{LR}_k, k \in [t-T,t-1]\}$. A convolutional frontend $G(\cdot)$ with $N_1$ residual blocks extracts features and token grids from the current frame and the buffered frames: $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$ where $N$ is the number of tokens per frame (Zhu et al., 14 Aug 2025).

The model then constructs per-pixel trajectories over the current and previous frames,

$\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$

with $(x_i^k, y_i^k) \in [1,H] \times [1,W]$, and the endpoint $I^{LR}_t$0 corresponding to $I^{LR}_t$1. For each current token $I^{LR}_t$2, TS-Mamba computes cosine similarity against trajectory-linked tokens from previous frames,

$I^{LR}_t$3

selects the top-$I^{LR}_t$4 previous-frame indices, and assembles the selected token set $I^{LR}_t$5 (Zhu et al., 14 Aug 2025).

The aggregated low-resolution representation is produced by the Trajectory-aware Shifted Mamba Aggregation (TSMA) module: $I^{LR}_t$6 followed by reconstruction

$I^{LR}_t$7

where $I^{LR}_t$8 comprises two convolutions, $I^{LR}_t$9 residual blocks, and a pixelshuffle layer, and $T$0 is bicubic upsampling (Zhu et al., 14 Aug 2025).

A concise summary of the main components is as follows.

Component	Function	Reported configuration
$T$1	Feature extraction and tokenization	$T$2 residual blocks
Trajectory construction	Connect current tokens to previous-frame counterparts	FastFlowNet for trajectory updates
Token selection	Top-$T$3 cosine-similar tokens along trajectories	$T$4
TSMA	Spatio-temporal aggregation with shifted SSMs	Token size $T$5, window size $T$6
$T$7	SR reconstruction	$T$8 residual blocks

The architectural significance lies in the granularity of aggregation. Rather than aligning or attending over complete previous frames, TS-Mamba selects only the most similar trajectory-consistent tokens. A plausible implication is that this reduces irrelevant temporal mixing while preserving the linear-complexity advantages of SSM-based sequence processing.

3. Shifted SSMs, Hilbert scanning, and TSMA

The distinctive “shifted” component of TS-Mamba is implemented through shifted SSM blocks designed around Hilbert scanning and corresponding shift operations (Zhu et al., 14 Aug 2025). To feed 2D image tokens into SSMs, the model uses Hilbert curves to linearize 2D windows into 1D sequences. Compared with raster scans, Hilbert curves are described as space-filling and locality-preserving, making adjacent 2D pixels more likely to remain near each other in the resulting 1D order.

TS-Mamba further analyzes discontinuities induced by Hilbert scanning through a discontinuity degree $T$9. Discontinuities are divided into intra-window and inter-window types. To compensate for these scanning losses, the model introduces a Scan–Shift–Scan strategy: $\{I^{LR}_k, k \in [t-T,t-1]\}$0 after which the SSM consumes $\{I^{LR}_k, k \in [t-T,t-1]\}$1 (Zhu et al., 14 Aug 2025).

The shift operator acts on the grid of local windows: $\{I^{LR}_k, k \in [t-T,t-1]\}$2 Specific shifts include $\{I^{LR}_k, k \in [t-T,t-1]\}$3, $\{I^{LR}_k, k \in [t-T,t-1]\}$4, $\{I^{LR}_k, k \in [t-T,t-1]\}$5, $\{I^{LR}_k, k \in [t-T,t-1]\}$6, and diagonal shifts such as $\{I^{LR}_k, k \in [t-T,t-1]\}$7 and $\{I^{LR}_k, k \in [t-T,t-1]\}$8 (Zhu et al., 14 Aug 2025).

TS-Mamba instantiates two parallel shifted-SSM branches. The Intra-window Compensation Branch (IntraWCB) uses Scan-1 $\{I^{LR}_k, k \in [t-T,t-1]\}$9 Scan-3, while the Inter-window Compensation Branch (InterWCB) uses Scan-2 $G(\cdot)$0 Scan-4 (Zhu et al., 14 Aug 2025). These choices follow the paper’s analysis of eliminated discontinuity degree $G(\cdot)$1, where different scan-shift-scan combinations optimize either intra-window or inter-window compensation.

Temporal aggregation is performed by SS3D, a selective scanning procedure along the temporal dimension. For each spatial window, SS3D constructs a sequence that alternates current tokens with the selected trajectory tokens in Hilbert order per window and across time. TSMA then concatenates current and selected tokens along the channel dimension and processes them in two parallel paths, each containing a standard SSM block for the first scan and two parallel shifted SSM blocks for intra- and inter-window compensation, followed by convolution and a deformable attention block. Each SSM, shifted SSM, and deformable attention block is preceded by layer normalization and followed by a residual connection (Zhu et al., 14 Aug 2025).

A recurrent SSM backbone underlies these blocks: $G(\cdot)$2 with Mamba-style selectivity making the effective parameters input-dependent (Zhu et al., 14 Aug 2025, Gu et al., 2023). In this construction, the shift operations do not replace the SSM recurrence; they modify the spatial ordering seen by the SSM so that local neighborhoods remain more continuous after 2D-to-1D conversion.

4. Objective functions, training protocol, and online inference

TS-Mamba is trained with a spatial reconstruction loss and a trajectory-aware loss (Zhu et al., 14 Aug 2025). The spatial term is a Charbonnier loss: $G(\cdot)$3 with $G(\cdot)$4.

The trajectory-aware term aligns low-resolution trajectories with downsampled high-resolution trajectories: $G(\cdot)$5 where the scale factor is $G(\cdot)$6. The total objective is

$G(\cdot)$7

with $G(\cdot)$8 (Zhu et al., 14 Aug 2025).

The online inference pipeline is explicitly causal. For frame $G(\cdot)$9, the model reads the current low-resolution frame, maintains a buffer of $N_1$0 previous frames, features, and tokens, extracts $N_1$1 and $N_1$2, updates trajectories with FastFlowNet, computes cosine similarities for token selection, applies TSMA to obtain $N_1$3, reconstructs $N_1$4, and updates the buffers for frame $N_1$5 (Zhu et al., 14 Aug 2025). This separation between buffer management, trajectory update, token selection, and TSMA aggregation is central to the method’s real-time framing.

The reported training setup uses REDS and Vimeo-90K for training; evaluation is on REDS4, Vimeo-90K-T, and Vid4. Degradations include BI and BD, with Gaussian blur $N_1$6 for BD. Implementation details include token size $N_1$7, window size $N_1$8, selected tokens $N_1$9, Adam optimization, Cosine Annealing, HR patch size $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$0, batch size 8, 600K iterations, PyTorch, and $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$1 RTX 3090 GPUs. Temporal window size is $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$2 for REDS, and Vimeo sequences are temporally flipped to 14 frames (Zhu et al., 14 Aug 2025).

5. Empirical performance and ablation evidence

On low-resolution $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$3 input, TS-Mamba reports 112 G MACs, 29 ms runtime, approximately 33.5 FPS, and 3.0 M parameters (Zhu et al., 14 Aug 2025). The paper states that this qualifies as real-time by the $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$4 FPS criterion and that TS-Mamba is the second-fastest runtime among the online methods tested; TMP is faster via CUDA acceleration but has higher MACs and parameters.

The main quantitative results in the online setting are summarized below.

Dataset and setting	TS-Mamba result	Comparison noted in the paper
REDS4 (BI, RGB)	30.73 / 0.8727	FDAN 30.71 / 0.8723; KSNet 30.69 / 0.8724
Vid4 (BI, Y)	27.17 / 0.8209	FDAN 27.14 / 0.8206; KSNet 27.14 / 0.8208
Vimeo-90K-T (BD, Y)	37.36 / 0.9482	FDAN 37.36 / 0.9483; KSNet 37.34 / 0.9490
Vid4 (BD, Y)	27.70 / 0.8473	FDAN 27.76 / 0.8471; RRN 27.69 / 0.8488

The paper characterizes these results as state-of-the-art performance in most cases with substantially reduced MACs (Zhu et al., 14 Aug 2025). The numerical pattern is mixed rather than uniform: TS-Mamba leads on REDS4 (BI, RGB) and Vid4 (BI, Y), ties FDAN in PSNR on Vimeo-90K-T (BD, Y), and is competitive rather than dominant on Vid4 (BD, Y). That profile is consistent with an efficiency-oriented architecture whose gains are strongest when long-term causal aggregation matters.

The ablation studies isolate three effects. First, removing trajectories degrades REDS4 (BI, RGB) performance from $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$5 to $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$6, while removing $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$7 yields $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$8 (Zhu et al., 14 Aug 2025). Second, removing either IntraWCB or InterWCB reduces performance to $$Q = G(I^{LR}_t) = \{q_i^t\}, \quad V = G(\{I^{LR}_k\}) = \{v_i^k\},$$9 and $N$0, respectively, and removing both yields $N$1. Third, removing shifts also lowers performance: without $N$2, $N$3; without $N$4, $N$5; without all shifts, $N$6 (Zhu et al., 14 Aug 2025).

The number of selected tokens $N$7 also shows a reported trade-off. Performance rises from $N$8 at $N$9 to $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$0 at $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$1, while $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$2 reaches $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$3 at higher cost. The paper therefore identifies $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$4 as a good trade-off between gains and complexity (Zhu et al., 14 Aug 2025).

6. Conceptual scope, relation to Mamba, and related VSR formulations

TS-Mamba inherits its sequence-modeling backbone from Mamba, which introduced selective state spaces with input-dependent parameters and a hardware-aware parallel scan, yielding linear scaling in sequence length (Gu et al., 2023). In the online VSR setting, TS-Mamba uses SSM and Mamba blocks to process scanned token sequences rather than dense temporal attention, with the stated benefit of linear-time sequence modeling and a global receptive field (Zhu et al., 14 Aug 2025).

A common source of ambiguity is the meaning of “shifted” and “trajectory-aware” across papers. In TS-Mamba for online VSR, “trajectory-aware” refers to constructing trajectories and selecting the most similar tokens from previous frames, while “shifted” refers to Hilbert-based Scan–Shift–Scan operations that compensate for scanning losses and strengthen spatial continuity (Zhu et al., 14 Aug 2025). In Gather-Scatter Mamba, by contrast, the paper explicitly states that its hybrid pipeline realizes “Trajectory-aware Shifted SSMs (TS-Mamba)” via shifted-window self-attention for spatial context aggregation, Mamba-based selective scanning for temporal propagation, and Gather–Scatter Mamba for alignment-aware warping to a center anchor frame (Ko et al., 1 Oct 2025).

That later formulation makes the abstraction explicit. Let $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$5 denote a trajectory-aware alignment operator that maps per-frame features $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$6 to anchor-aligned tokens $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$7. A shifted SSM then acts on $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$8: $\mathcal{T}^t = \{\tau_i^k = (x_i^k, y_i^k)\},$9 followed by a scatter operator $(x_i^k, y_i^k) \in [1,H] \times [1,W]$0 that disperses outputs back to the original frames (Ko et al., 1 Oct 2025). This broader view suggests that TS-Mamba is best understood as a family of designs rather than a single implementation detail: trajectory awareness defines how tokens are matched or aligned across time, shifting defines how locality is preserved under sequence conversion or windowing, and the SSM defines how those ordered tokens are propagated.

The principal misconception is therefore to treat TS-Mamba as synonymous with either Mamba alone or shifted attention alone. In the online formulation, it is a causal token-selection and shifted-SSM architecture for VSR (Zhu et al., 14 Aug 2025). In the broader formalization, it is a compositional recipe that can include explicit alignment operators, shifted local aggregation, and selective SSM propagation (Ko et al., 1 Oct 2025). Across both uses, the unifying principle is efficient long-range temporal modeling without quadratic attention cost.