---
title: Self-expressive Sequence Regularization (SSR)
url: https://www.emergentmind.com/topics/self-expressive-sequence-regularization-ssr
type: topic
---

# Self-expressive Sequence Regularization (SSR)

Self-expressive Sequence Regularization (SSR) is a training-free, plug-and-play regularizer for streaming 3D reconstruction models with a persistent latent state. It treats the latent persistent state as a subspace representation evolving on a Grassmannian manifold and enforces local temporal regularity during inference by reconstructing the current state from a short window of historical states through a self-expressive affinity matrix. In the formulation introduced in "SSR: A Training-Free Approach for Streaming 3D Reconstruction" [2603.14765], the objective is to reduce geometric drift and improve temporal stability in long-horizon recurrent reconstruction without introducing new learnable parameters, retraining, or test-time optimization.

## 1. Definition and problem setting

In the formulation of SSR introduced for streaming 3D reconstruction, the base setting is a stateful model such as CUT3R that processes long video streams frame by frame and maintains a persistent state across time [2603.14765]. At time \(t\), the model receives an image \(\mathbf I_t\), computes visual tokens, updates the persistent state, and predicts per-frame geometry and pose:
\[
\mathbf F_t = \mathrm{Enc}(\mathbf I_t),
\]
\[
[\mathbf S_t,\mathbf Y_t] = \mathrm{Interaction}(\mathbf S_{t-1},\mathbf F_t),
\]
\[
\{\hat{\mathbf X}_t,\hat{\mathbf P}_t,\hat{\mathbf N}_t\}
= \mathrm{Head}(\mathbf Y_t).
\]

Within this pipeline, \(\mathbf S_t\) is the persistent latent state, \(\mathbf Y_t\) are per-frame tokens, \(\hat{\mathbf X}_t\) is a pointmap, \(\hat{\mathbf P}_t\) is the \(6\)-DoF camera pose, and \(\hat{\mathbf N}_t\) are confidences [2603.14765]. The central failure mode addressed by SSR is geometric drift: because the persistent state is updated over potentially thousands of frames, small state errors caused by occlusion, low texture, or difficult motion can accumulate, leading to misaligned camera trajectories and distorted or inconsistent pointmaps [2603.14765].

SSR is described as training-free in two precise senses. It introduces no new learnable parameters, and it is applied only at inference time, with no backpropagation and no test-time training [2603.14765]. Its inputs are the already-computed latent states and a short historical window, and its operations are analytic: dot products, normalization, and linear combinations [2603.14765]. This makes SSR a regularizer on latent-state evolution rather than a retrained model component.

## 2. Grassmannian interpretation and self-expressive formulation

The paper interprets the latent state trajectory through a Grassmannian manifold perspective. The Grassmannian manifold \(\mathcal G(n,r)\) is the set of all \(r\)-dimensional linear subspaces of \(\mathbb R^n\), where a point is represented by an orthonormal basis matrix \(\mathbf U \in \mathbb R^{n \times r}\) with \(\mathbf U^\top \mathbf U = \mathbf I_r\), and the corresponding subspace is \([\mathbf U] = \mathrm{span}(\mathbf U)\) [2603.14765]. To compare two such subspaces, the paper uses the projection metric
\[
d_{\mathrm{proj}}\big([\mathbf U_1],[\mathbf U_2]\big)
= \frac{1}{\sqrt{2}}\left\| \mathbf U_1\mathbf U_1^\top - \mathbf U_2\mathbf U_2^\top \right\|_F.
\]

The modeling idea is that the persistent state \(\mathbf S_t\) can be viewed abstractly as a compact subspace descriptor of the scene at time \(t\), so the sequence \(\{\mathbf S_t\}_{t=1}^T\) forms a trajectory on the Grassmannian [2603.14765]. Temporal coherence then becomes a geometric constraint: for a physically coherent scene and smooth camera motion, consecutive latent states should remain close on the Grassmannian, which the paper expresses as \(d_{\mathrm{proj}}([\mathbf U_t],[\mathbf U_{t+1}])\) being small [2603.14765]. Geometric drift is therefore interpreted as the latent state leaving the coherent manifold-consistent evolution.

SSR operationalizes this viewpoint through the self-expressive property. For a sequence of vectors assembled into a matrix
\[
\mathbf S =
\begin{bmatrix}
\mathbf S_1^\top\\
\vdots\\
\mathbf S_t^\top
\end{bmatrix}
\in \mathbb R^{t \times d},
\]
self-expressiveness posits an affinity matrix \(\mathbf C \in \mathbb R^{t \times t}\) such that
\[
\mathbf S = \mathbf C\mathbf S,
\]
or, equivalently, each state can be written as a linear combination of other states in the sequence:
\[
\mathbf S_i = \sum_j C_{ij}\mathbf S_j.
\]

The paper explicitly relates this construction to self-expressive models from non-rigid structure from motion and subspace clustering, including classical low-rank formulations and later deep variants [2603.14765]. In SSR, however, the full optimization is replaced by a local, analytic affinity construction on a sliding temporal window.

## 3. Affinity matrix, update rule, and inference-time mechanics

At time \(t\), SSR forms a sliding window
\[
\mathcal W_t = [t-k,\dots,t],
\]
with default window size \(k=8\), and stacks the states in that window as
\[
\mathcal S_t =
\begin{bmatrix}
\mathbf S_{t-k}^\top\\
\vdots\\
\mathbf S_t^\top
\end{bmatrix}
\in \mathbb R^{k \times d}
\]
according to the paper’s notation [2603.14765]. The ideal local self-expressive relation is
\[
\mathcal S_t = \mathbf C^{(t)}\mathcal S_t,
\]
where \(\mathbf C^{(t)} \in \mathbb R^{k \times k}\) is the affinity matrix for the current window.

Instead of solving a nuclear-norm or low-rank objective, SSR computes \(\mathbf C^{(t)}\) analytically from pairwise similarity. The similarity function is the non-normalized dot product \(\phi(\mathbf a,\mathbf b) = \mathbf a \cdot \mathbf b\), and the affinity coefficients are obtained by row-wise normalization:
\[
C_{ij}^{(t)}
= \frac{\phi(\mathbf S_{\mathcal W_i}, \mathbf S_{\mathcal W_j})}
{\sum_{k'} \phi(\mathbf S_{\mathcal W_i}, \mathbf S_{\mathcal W_{k'}})}.
\]
Equivalently, if the states in the window are indexed as \(\mathbf S_{t-k},\dots,\mathbf S_t\),
\[
C_{ij}^{(t)}
= \frac{\phi(\mathbf S_{t-k+i}, \mathbf S_{t-k+j})}
{\sum_{m=0}^{k} \phi(\mathbf S_{t-k+i}, \mathbf S_{t-k+m})}.
\]

This normalization ensures that each row of \(\mathbf C^{(t)}\) sums to \(1\), and large \(C_{ij}\) indicates that state \(j\) is a good basis to reconstruct state \(i\) [2603.14765]. The corrected state sequence is then computed by
\[
\hat{\mathcal S}_t = \mathbf C^{(t)}\mathcal S_t,
\]
and the corrected current state is extracted from the last row:
\[
\hat{\mathbf S}_t = \sum_{j=0}^{k} C_{kj}^{(t)} \cdot \mathbf S_j.
\]

The intended effect is explicit in the paper’s description: if the raw recurrent update \(\mathbf S_t\) is noisy or drifting, while earlier states in the window remain consistent, the affinity weights pull the corrected state back toward a locally coherent subspace inferred from its neighbors [2603.14765]. The regularization is therefore projection-like rather than predictive. It does not modify the encoder or head directly, but the downstream predictions become more stable because they depend on the corrected persistent state [2603.14765].

## 4. Integration into streaming 3D reconstruction systems

SSR is integrated into a streaming reconstruction model by wrapping the recurrent state update. In the paper’s formulation, the forward pass proceeds normally to obtain a raw \(\mathbf S_t\), after which SSR computes the affinity matrix over the window \(\{\mathbf S_{t-k},\dots,\mathbf S_t\}\), reconstructs the corrected sequence, and replaces the current persistent state with \(\hat{\mathbf S}_t\) for use at the next time step [2603.14765]. The correction is applied at every time step, and the window slides forward by one frame each time.

The method operates only on the latent persistent state. In the CUT3R integration studied in the paper, the persistent state is a collection of transformer tokens maintained across time in the interaction module, and SSR neither changes the encoder nor the task-specific heads [2603.14765]. This design is central to its plug-and-play character.

The computational overhead is also defined explicitly. For each time step, computing the pairwise similarities requires \(O(k^2 d)\) operations, reconstructing \(\hat{\mathcal S}_t = \mathbf C^{(t)}\mathcal S_t\) adds another \(O(k^2 d)\), and storing the window requires \(O(kd)\) memory, where \(k\) is the window size and \(d\) is the latent-state dimension [2603.14765]. Because \(k\) is small and fixed, and because the operation uses only simple linear algebra without backpropagation, the paper characterizes the overhead as minimal relative to the transformer backbone [2603.14765].

A useful interpretive consequence is that SSR functions as an inference-time state corrector rather than a learned latent transition model. This suggests that its main value lies in stabilizing recurrent dynamics that are already competent but vulnerable to long-horizon accumulation of small errors.

## 5. Empirical behavior, ablations, and operating regime

The empirical evaluation in the paper is organized around three tasks: video depth estimation, pose estimation, and 3D reconstruction, all using CUT3R as the base model [2603.14765]. Depth experiments are reported on KITTI, Sintel, and Bonn with Abs Rel and \(\delta < 1.25\) under both per-sequence scale and metric scale. Pose experiments are reported on Sintel, TUM-Dynamics, and ScanNet using ATE, RPE translation, and RPE rotation. Reconstruction experiments are reported on 7-Scenes and NRGBD using Accuracy, Completeness, and Normal Consistency [2603.14765].

Across video depth estimation, SSR is reported to consistently improve over CUT3R and training-free TTT3R, with particularly large gains on Bonn, a long-sequence benchmark, and similar or better performance on KITTI and Sintel [2603.14765]. For pose estimation, SSR achieves the best ATE on TUM-Dynamics and reduces ATE on ScanNet while remaining competitive in RPE; the qualitative effect highlighted in the paper is improved loop closure and less trajectory drift [2603.14765]. For 3D reconstruction, the behavior is more conditional. On sparse short sequences, SSR can slightly underperform CUT3R on some metrics, whereas on dense long sequences with more continuous input it significantly outperforms the baseline in both Accuracy and Completeness [2603.14765].

The ablation on window length is central to understanding the method’s effective regime. The paper studies \(k = 2, 4, 8, 16, 32, 64\) on Bonn and KITTI depth and reports that gains plateau after moderate window sizes, with diminishing returns beyond approximately \(4\) to \(8\) frames [2603.14765]. This directly justifies the default use of a small window. Another ablation compares SSR to a naïve temporal fusion rule,
\[
\hat{\mathbf S}_t = \alpha \mathbf S_t + (1-\alpha)\mathbf S_{t-1},
\]
which gives modest improvements but remains weaker than the structured affinity-based correction used by SSR [2603.14765]. The paper interprets this difference as evidence that self-expressive reconstruction is more effective than uniform blending for combating context forgetting and length degradation.

The observed failure mode on sparse short sequences is also important. The paper attributes this degradation to a mismatch between the method’s assumptions and the input structure: when only a few sparse views are available, the ideal affinity matrix should be close to the identity, but similarity between distant frames and the lack of explicit time encoding can produce non-ideal mixing [2603.14765]. This is not presented as a contradiction of the method, but as a boundary condition on when local self-expressiveness is a reliable prior.

## 6. Relation to prior work, interpretation, and acronym ambiguity

SSR is explicitly grounded in the self-expressive property used in non-rigid structure from motion and subspace clustering, including low-rank representation and deep subspace clustering formulations [2603.14765]. In that literature, \(\mathbf C\) encodes which points lie in the same subspace, often under low-rank or sparsity priors. SSR adopts the same conceptual structure but replaces optimization with an analytic, similarity-based coefficient matrix computed on a local temporal window [2603.14765]. The windowed states function as a dictionary, and the coefficients specify how each state is reconstructed from the others. A plausible implication is that SSR can be understood as a form of dictionary coding without learning, specialized to recurrent latent-state trajectories.

The paper further argues that the same principle is generic beyond 3D reconstruction: any recurrent or streaming model with a latent state \(\mathbf S_t\) could maintain a recent-state buffer, compute similarity-based affinities, and replace or blend the current state with \(\hat{\mathbf S}_t = \mathbf C^{(t)}\mathcal S_t\) [2603.14765]. The listed potential applications include video models, language models and sequence models, time-series forecasting, and SLAM or tracking. These are presented as conceptual extensions rather than validated applications [2603.14765].

A recurring source of confusion is the acronym itself. In arXiv usage, “SSR” is not unique. In "Sparse-firing regularization methods for spiking neural networks with time-to-first spike coding" [2307.13007], SSR stands for spike-timing-based sparse-firing regularization, with concrete variants M-SSR and F-SSR for TTFS-coded spiking neural networks. In "Sentence Semantic Regression for Text Generation" [2108.02984], SSR stands for Sentence Semantic Regression, a sentence-level language-modeling framework for text generation. Neither of those works defines SSR as Self-expressive Sequence Regularization. The self-expressive, Grassmannian, inference-time regularizer discussed here is specifically the construction introduced in [2603.14765].

The main limitations are correspondingly specific. SSR assumes enough contextual redundancy within the local window for the affinity structure to be meaningful; it does not explicitly encode time; it may over-smooth or mix non-local contexts in short sparse sequences; and the paper provides no explicit stability proofs or convergence guarantees [2603.14765]. These constraints place SSR in a precise methodological niche: it is an analytic latent-state regularizer that is most effective when long-range temporal context is available and the latent trajectory is expected to remain near a slowly varying local subspace.

Source: https://www.emergentmind.com/topics/self-expressive-sequence-regularization-ssr