Spherical Spatio-Temporal Consistency Learning

• Spherical Spatio-Temporal Consistency Learning (SSTC) is a depth representation regularizer for panoramic videos that stabilizes fluctuating monocular depth cues.
• It combines a Temporal Mixer to smooth frame-wise depth noise with spherical geometry-aware attention that aligns features with the true equirectangular model.
• By filtering short-term noise and encoding geometric priors, SSTC significantly improves object tracking accuracy and identity preservation in 360° scenarios.

Searching arXiv for the specified paper to ground the article in the cited source. Spherical Spatio-Temporal Consistency Learning (SSTC) is a module introduced within the depth-aware cylindrical tracking-by-detection framework CylindTrack for panoramic multi-object tracking. It is designed for equirectangular panoramic videos, where monocular depth estimates fluctuate over time, especially near the $0^\circ/360^\circ$ seam or under occlusion, and where large-FoV scenes exhibit spherical geometric distortions that standard planar networks do not respect. SSTC addresses these issues by enforcing local temporal alignment of per-object depth representations so that short-term scale noise is suppressed, and by explicitly injecting a deterministic spherical geometry prior so that depth query features remain consistent with the equirectangular camera model (Deng et al., 29 Jun 2026).

1. Problem setting and conceptual role

SSTC arises from the observation that panoramic MOT is not a straightforward extension of perspective MOT. In equirectangular panoramic videos, the horizontal image domain is periodic rather than Euclidean, which breaks planar motion assumptions and makes IoU-based association unreliable near the $0^\circ/360^\circ$ seam. At the same time, large-FoV scenes often contain more objects, stronger scale variation, and more frequent interactions, making online association particularly sensitive to unstable frame-wise depth cues.

Within this setting, SSTC operates as the representation-learning component that stabilizes depth-aware object features before motion modeling and association. Its stated function is twofold. First, it promotes temporal coherence in depth-aware representations, reducing frame-to-frame fluctuations that would otherwise corrupt online data association. Second, it encodes spherical geometric structure directly into the representation pathway, so that feature alignment respects the panoramic imaging model rather than an implicit planar approximation.

This design places SSTC between raw depth estimation and downstream association logic. A plausible implication is that SSTC serves as the feature-level counterpart to the trajectory-level mechanisms elsewhere in CylindTrack: it regularizes query representations locally in time, while the larger framework leverages those stabilized cues for identity preservation and trajectory continuity.

2. Placement in the detector and overall architecture

SSTC is inserted into the DETR-style depth-enhanced detector after the depth-query update. It consists of two submodules: a Temporal Mixer and Spherical Geometry-aware Attention (SGA).

The input to SSTC is a batch of depth queries

$X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$

where $B$ is the batch size, $T$ is the time window length, $Q$ is the number of object queries, and $C$ is the feature dimension. The module first reshapes and normalizes these queries for temporal processing, then applies spherical geometry-aware cross-attention using a fixed geometry context derived from feature-map pixel centers.

The architectural decomposition is significant because it separates two distinct failure modes of panoramic depth-aware tracking. The Temporal Mixer targets temporal instability in monocular depth cues, while SGA targets geometric inconsistency induced by the equirectangular camera model. Their combination is therefore not merely additive in a structural sense; it is intended to jointly model temporal coherence and panoramic geometric alignment.

3. Temporal Mixer

The Temporal Mixer ties the $Q\times C$ feature vectors for each object query across $T$ frames via a depthwise convolution along the temporal axis. The computation is given as

$Y = \mathrm{LN}(\mathrm{reshape}(X)),$

$0^\circ/360^\circ$0

$0^\circ/360^\circ$1

Here, $0^\circ/360^\circ$2 is a 1-D convolution along the temporal axis with kernel size $0^\circ/360^\circ$3, stride $0^\circ/360^\circ$4, and groups $0^\circ/360^\circ$5, so each channel is convolved independently over its $0^\circ/360^\circ$6 frames. $0^\circ/360^\circ$7 is a $0^\circ/360^\circ$8 convolution across channels. $0^\circ/360^\circ$9 is a learned scalar initialized to $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$0, $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$1 is LayerNorm, and $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$2 returns the shape $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$3.

The module is explicitly characterized as implementing a learnable finite-impulse-response filter that smooths out frame-to-frame depth noise. Because $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$4 is learned and initialized to zero, the network can gradually turn on temporal mixing during training. No recurrence or external memory is required; SSTC operates on small query tensors of shape $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$5.

This formulation is lightweight in the sense intended by the source: temporal coupling occurs only on object-query tensors rather than dense image grids. That choice is central to SSTC’s role in online tracking, where low-latency query refinement is more relevant than heavy spatio-temporal feature aggregation.

4. Spherical Geometry-aware Attention

The second submodule, Spherical Geometry-aware Attention, injects explicit knowledge of the equirectangular camera geometry. It begins by precomputing a fixed geometry context

$X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$6

where

$X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$7

across $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$8 feature scales.

Each feature-map pixel center $X \in \mathbb{R}^{(B\cdot T)\times Q\times C},$9 is mapped to spherical angles

$B$0

and then embedded via sinusoidal/Fourier features:

$B$1

The geometry context is obtained as

$B$2

Query, key, and value projections are then formed as

$B$3

$B$4

SGA uses width-height separated attention. The geometry tokens are split into two “directions”: horizontal tokens

$B$5

and vertical tokens

$B$6

that is, grouped by rows or columns. Attention is computed as

$B$7

The outputs are fused and added residually:

$B$8

The fuse operation is typically a learned linear projection or simple element-wise sum. The stated rationale is that width/height separation aligns query tokens to geometry tokens along great-circle directions, approximating spherical alignment. The source further notes that $B$9 inherently measures angular similarity between object queries and spherical coordinates, encouraging depth estimates to respect the true $T$0 geometry (Deng et al., 29 Jun 2026).

5. Implementation characteristics and computational profile

SSTC is specified with practical implementation ranges. The number of depth queries is approximately $T$1–$T$2, the feature dimension is $T$3, and the time window is $T$4–$T$5 frames. For the Temporal Mixer, the kernel size is $T$6, groups $T$7, and $T$8 is a $T$9 convolution.

For SGA, one multi-head or single-head cross-attention is used, and fusion is performed by a $Q$0 convolution. Example feature scales are $Q$1, giving

$Q$2

The complexity figures reported for SSTC are:

• Temporal Mixer:

$Q$3

described as negligible since $Q$4.

• SGA:

$Q$5

per scale instead of

$Q$6

Training is joint with detection and depth losses, and no extra supervision is required. The scalar $Q$7 is initialized to zero; learning rate and other hyperparameters follow the base detector, specifically AdamW with $Q$8.

These details clarify that SSTC is meant to be inserted into an existing depth-enhanced detector with minimal training protocol changes. This suggests that its contribution is primarily representational and architectural rather than dependent on specialized supervision or auxiliary objectives.

6. Empirical behavior and ablation evidence

The ablation evidence reported for QuadTrack isolates the effects of the two SSTC submodules. Starting from the depth-enhanced baseline (ID 1), adding only the Temporal Mixer (ID 2) yields a small gain in HOTA $Q$9 and IDF1 $C$0. Adding only SGA (ID 3) yields $C$1 HOTA and $C$2 IDF1. Combining both (ID 4) yields $C$3 HOTA and $C$4 IDF1 (Deng et al., 29 Jun 2026).

Variant	HOTA gain	IDF1 gain
Temporal Mixer only	+0.361	+0.916
SGA only	+0.983	+0.532
Temporal Mixer + SGA	+2.648	+3.850

In the full CylindTrack system, with cylindrical motion modeling as well, SSTC together with depth-temporal Kalman lifts HOTA from $C$5 to $C$6 and IDF1 from $C$7 to $C$8 on QuadTrack—an absolute gain of $C$9 and $Q\times C$0 points, respectively.

The pattern of these results is notable. The individual gains of the two submodules are modest, but the joint configuration produces a larger improvement. This suggests that temporal stabilization and spherical geometric alignment are complementary rather than redundant: the Temporal Mixer suppresses short-term depth noise, while SGA constrains the representation to remain compatible with panoramic geometry. The source summarizes the effect as markedly better identity association and trajectory continuity in panoramic multi-object tracking.

7. Interpretation within panoramic multi-object tracking

SSTC is best understood as a depth-representation regularizer specialized for equirectangular video. Its temporal component addresses the instability of monocular depth when used as a frame-wise cue for online data association. Its geometric component addresses the mismatch between standard planar feature processing and the true topology of $Q\times C$1 imagery.

A common misconception in this area is that better frame-wise depth alone is sufficient for robust panoramic association. The formulation of SSTC argues against that view: the objective is not only to estimate depth, but to make depth-aware query features temporally coherent and geometrically aligned with the spherical camera model. Likewise, SSTC does not replace motion modeling; in CylindTrack it operates alongside trajectory-level depth consistency and topology-aware cylindrical motion prediction.

Within that broader framework, SSTC’s contribution is specific and delimited. It does not introduce recurrence, external memory, or extra supervision. Instead, it uses a depthwise temporal filter on object queries and a fixed spherical geometry context encoded by Fourier features. The resulting representation is intended to remain stable across short temporal windows and consistent with the periodic, spherical structure of equirectangular panoramas. In the terminology of the source, these temporally stable, geometry-consistent depth cues translate into better identity preservation and trajectory continuity in challenging panoramic scenes (Deng et al., 29 Jun 2026).