Self-expressive Sequence Regularization (SSR)

• Self-expressive Sequence Regularization (SSR) is a plug-and-play, training-free method that reduces geometric drift by regularizing a persistent latent state in streaming 3D reconstruction.
• It leverages a Grassmannian manifold interpretation and computes an analytic self-expressive affinity matrix on a sliding temporal window to ensure local temporal consistency.
• Empirical evaluations show SSR enhances depth, pose, and 3D reconstruction tasks with minimal overhead by stabilizing recurrent state updates without retraining.

Self-expressive Sequence Regularization (SSR) is a training-free, plug-and-play regularizer for streaming 3D reconstruction models with a persistent latent state. It treats the latent persistent state as a subspace representation evolving on a Grassmannian manifold and enforces local temporal regularity during inference by reconstructing the current state from a short window of historical states through a self-expressive affinity matrix. In the formulation introduced in "SSR: A Training-Free Approach for Streaming 3D Reconstruction" (Deng et al., 16 Mar 2026), the objective is to reduce geometric drift and improve temporal stability in long-horizon recurrent reconstruction without introducing new learnable parameters, retraining, or test-time optimization.

1. Definition and problem setting

In the formulation of SSR introduced for streaming 3D reconstruction, the base setting is a stateful model such as CUT3R that processes long video streams frame by frame and maintains a persistent state across time (Deng et al., 16 Mar 2026). At time $t$, the model receives an image $\mathbf I_t$, computes visual tokens, updates the persistent state, and predicts per-frame geometry and pose: $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$

$$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$

$$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$

Within this pipeline, $\mathbf S_t$ is the persistent latent state, $\mathbf Y_t$ are per-frame tokens, $\hat{\mathbf X}_t$ is a pointmap, $\hat{\mathbf P}_t$ is the $6$-DoF camera pose, and $\mathbf I_t$0 are confidences (Deng et al., 16 Mar 2026). The central failure mode addressed by SSR is geometric drift: because the persistent state is updated over potentially thousands of frames, small state errors caused by occlusion, low texture, or difficult motion can accumulate, leading to misaligned camera trajectories and distorted or inconsistent pointmaps (Deng et al., 16 Mar 2026).

SSR is described as training-free in two precise senses. It introduces no new learnable parameters, and it is applied only at inference time, with no backpropagation and no test-time training (Deng et al., 16 Mar 2026). Its inputs are the already-computed latent states and a short historical window, and its operations are analytic: dot products, normalization, and linear combinations (Deng et al., 16 Mar 2026). This makes SSR a regularizer on latent-state evolution rather than a retrained model component.

2. Grassmannian interpretation and self-expressive formulation

The paper interprets the latent state trajectory through a Grassmannian manifold perspective. The Grassmannian manifold $\mathbf I_t$1 is the set of all $\mathbf I_t$2-dimensional linear subspaces of $\mathbf I_t$3, where a point is represented by an orthonormal basis matrix $\mathbf I_t$4 with $\mathbf I_t$5, and the corresponding subspace is $\mathbf I_t$6 (Deng et al., 16 Mar 2026). To compare two such subspaces, the paper uses the projection metric

$\mathbf I_t$7

The modeling idea is that the persistent state $\mathbf I_t$8 can be viewed abstractly as a compact subspace descriptor of the scene at time $\mathbf I_t$9, so the sequence $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$0 forms a trajectory on the Grassmannian (Deng et al., 16 Mar 2026). Temporal coherence then becomes a geometric constraint: for a physically coherent scene and smooth camera motion, consecutive latent states should remain close on the Grassmannian, which the paper expresses as $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$1 being small (Deng et al., 16 Mar 2026). Geometric drift is therefore interpreted as the latent state leaving the coherent manifold-consistent evolution.

SSR operationalizes this viewpoint through the self-expressive property. For a sequence of vectors assembled into a matrix

$\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$2

self-expressiveness posits an affinity matrix $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$3 such that

$\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$4

or, equivalently, each state can be written as a linear combination of other states in the sequence: $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$5

The paper explicitly relates this construction to self-expressive models from non-rigid structure from motion and subspace clustering, including classical low-rank formulations and later deep variants (Deng et al., 16 Mar 2026). In SSR, however, the full optimization is replaced by a local, analytic affinity construction on a sliding temporal window.

3. Affinity matrix, update rule, and inference-time mechanics

At time $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$6, SSR forms a sliding window

$\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$7

with default window size $\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$8, and stacks the states in that window as

$\mathbf F_t = \mathrm{Enc}(\mathbf I_t),$9

according to the paper’s notation (Deng et al., 16 Mar 2026). The ideal local self-expressive relation is

$$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$0

where $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$1 is the affinity matrix for the current window.

Instead of solving a nuclear-norm or low-rank objective, SSR computes $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$2 analytically from pairwise similarity. The similarity function is the non-normalized dot product $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$3, and the affinity coefficients are obtained by row-wise normalization: $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$4 Equivalently, if the states in the window are indexed as $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$5,

$$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$6

This normalization ensures that each row of $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$7 sums to $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$8, and large $$[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),$$9 indicates that state $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$0 is a good basis to reconstruct state $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$1 (Deng et al., 16 Mar 2026). The corrected state sequence is then computed by

$$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$2

and the corrected current state is extracted from the last row: $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$3

The intended effect is explicit in the paper’s description: if the raw recurrent update $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$4 is noisy or drifting, while earlier states in the window remain consistent, the affinity weights pull the corrected state back toward a locally coherent subspace inferred from its neighbors (Deng et al., 16 Mar 2026). The regularization is therefore projection-like rather than predictive. It does not modify the encoder or head directly, but the downstream predictions become more stable because they depend on the corrected persistent state (Deng et al., 16 Mar 2026).

4. Integration into streaming 3D reconstruction systems

SSR is integrated into a streaming reconstruction model by wrapping the recurrent state update. In the paper’s formulation, the forward pass proceeds normally to obtain a raw $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$5, after which SSR computes the affinity matrix over the window $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$6, reconstructs the corrected sequence, and replaces the current persistent state with $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$7 for use at the next time step (Deng et al., 16 Mar 2026). The correction is applied at every time step, and the window slides forward by one frame each time.

The method operates only on the latent persistent state. In the CUT3R integration studied in the paper, the persistent state is a collection of transformer tokens maintained across time in the interaction module, and SSR neither changes the encoder nor the task-specific heads (Deng et al., 16 Mar 2026). This design is central to its plug-and-play character.

The computational overhead is also defined explicitly. For each time step, computing the pairwise similarities requires $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$8 operations, reconstructing $$\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\} = \mathrm{Head}(\mathbf Y_t).$$9 adds another $\mathbf S_t$0, and storing the window requires $\mathbf S_t$1 memory, where $\mathbf S_t$2 is the window size and $\mathbf S_t$3 is the latent-state dimension (Deng et al., 16 Mar 2026). Because $\mathbf S_t$4 is small and fixed, and because the operation uses only simple linear algebra without backpropagation, the paper characterizes the overhead as minimal relative to the transformer backbone (Deng et al., 16 Mar 2026).

A useful interpretive consequence is that SSR functions as an inference-time state corrector rather than a learned latent transition model. This suggests that its main value lies in stabilizing recurrent dynamics that are already competent but vulnerable to long-horizon accumulation of small errors.

5. Empirical behavior, ablations, and operating regime

The empirical evaluation in the paper is organized around three tasks: video depth estimation, pose estimation, and 3D reconstruction, all using CUT3R as the base model (Deng et al., 16 Mar 2026). Depth experiments are reported on KITTI, Sintel, and Bonn with Abs Rel and $\mathbf S_t$5 under both per-sequence scale and metric scale. Pose experiments are reported on Sintel, TUM-Dynamics, and ScanNet using ATE, RPE translation, and RPE rotation. Reconstruction experiments are reported on 7-Scenes and NRGBD using Accuracy, Completeness, and Normal Consistency (Deng et al., 16 Mar 2026).

Across video depth estimation, SSR is reported to consistently improve over CUT3R and training-free TTT3R, with particularly large gains on Bonn, a long-sequence benchmark, and similar or better performance on KITTI and Sintel (Deng et al., 16 Mar 2026). For pose estimation, SSR achieves the best ATE on TUM-Dynamics and reduces ATE on ScanNet while remaining competitive in RPE; the qualitative effect highlighted in the paper is improved loop closure and less trajectory drift (Deng et al., 16 Mar 2026). For 3D reconstruction, the behavior is more conditional. On sparse short sequences, SSR can slightly underperform CUT3R on some metrics, whereas on dense long sequences with more continuous input it significantly outperforms the baseline in both Accuracy and Completeness (Deng et al., 16 Mar 2026).

The ablation on window length is central to understanding the method’s effective regime. The paper studies $\mathbf S_t$6 on Bonn and KITTI depth and reports that gains plateau after moderate window sizes, with diminishing returns beyond approximately $\mathbf S_t$7 to $\mathbf S_t$8 frames (Deng et al., 16 Mar 2026). This directly justifies the default use of a small window. Another ablation compares SSR to a naïve temporal fusion rule,

$\mathbf S_t$9

which gives modest improvements but remains weaker than the structured affinity-based correction used by SSR (Deng et al., 16 Mar 2026). The paper interprets this difference as evidence that self-expressive reconstruction is more effective than uniform blending for combating context forgetting and length degradation.

The observed failure mode on sparse short sequences is also important. The paper attributes this degradation to a mismatch between the method’s assumptions and the input structure: when only a few sparse views are available, the ideal affinity matrix should be close to the identity, but similarity between distant frames and the lack of explicit time encoding can produce non-ideal mixing (Deng et al., 16 Mar 2026). This is not presented as a contradiction of the method, but as a boundary condition on when local self-expressiveness is a reliable prior.

6. Relation to prior work, interpretation, and acronym ambiguity

SSR is explicitly grounded in the self-expressive property used in non-rigid structure from motion and subspace clustering, including low-rank representation and deep subspace clustering formulations (Deng et al., 16 Mar 2026). In that literature, $\mathbf Y_t$0 encodes which points lie in the same subspace, often under low-rank or sparsity priors. SSR adopts the same conceptual structure but replaces optimization with an analytic, similarity-based coefficient matrix computed on a local temporal window (Deng et al., 16 Mar 2026). The windowed states function as a dictionary, and the coefficients specify how each state is reconstructed from the others. A plausible implication is that SSR can be understood as a form of dictionary coding without learning, specialized to recurrent latent-state trajectories.

The paper further argues that the same principle is generic beyond 3D reconstruction: any recurrent or streaming model with a latent state $\mathbf Y_t$1 could maintain a recent-state buffer, compute similarity-based affinities, and replace or blend the current state with $\mathbf Y_t$2 (Deng et al., 16 Mar 2026). The listed potential applications include video models, LLMs and sequence models, time-series forecasting, and SLAM or tracking. These are presented as conceptual extensions rather than validated applications (Deng et al., 16 Mar 2026).

A recurring source of confusion is the acronym itself. In arXiv usage, “SSR” is not unique. In "Sparse-firing regularization methods for spiking neural networks with time-to-first spike coding" (Sakemi et al., 2023), SSR stands for spike-timing-based sparse-firing regularization, with concrete variants M-SSR and F-SSR for TTFS-coded spiking neural networks. In "Sentence Semantic Regression for Text Generation" (Wang et al., 2021), SSR stands for Sentence Semantic Regression, a sentence-level language-modeling framework for text generation. Neither of those works defines SSR as Self-expressive Sequence Regularization. The self-expressive, Grassmannian, inference-time regularizer discussed here is specifically the construction introduced in (Deng et al., 16 Mar 2026).

The main limitations are correspondingly specific. SSR assumes enough contextual redundancy within the local window for the affinity structure to be meaningful; it does not explicitly encode time; it may over-smooth or mix non-local contexts in short sparse sequences; and the paper provides no explicit stability proofs or convergence guarantees (Deng et al., 16 Mar 2026). These constraints place SSR in a precise methodological niche: it is an analytic latent-state regularizer that is most effective when long-range temporal context is available and the latent trajectory is expected to remain near a slowly varying local subspace.