OmniTrack++: Panoramic MOT Framework

• OmniTrack++ is a panoramic multi-object tracking framework that addresses distortion, large search space, and identity ambiguity using trajectory feedback.
• It integrates DynamicSSM for feature stabilization, FlexiTrack Instances for short-term association, and ExpertTrack Memory for long-term identity preservation.
• The system shows improved performance on specialized benchmarks, highlighting advantages over conventional narrow-FoV methods in real-world 360° applications.

OmniTrack++ is a framework for multi-object tracking (MOT) on 360° panoramic video that addresses panoramic distortion, large search space, and identity ambiguity under a 360° field of view (FoV) through a feedback-driven design that progressively refines perception with trajectory cues (Luo et al., 1 Nov 2025). It operates on panoramic frames, typically in equirectangular format, and outputs a set of trajectories $\mathcal{T}=\{\tau_1,\tau_2,\dots\}$, where each trajectory is a time-ordered list of detections $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$ with a consistent identity label. The method combines a DynamicSSM block for feature stabilization, FlexiTrack Instances for short-term trajectory-informed association, an ExpertTrack Memory for long-term identity preservation, and a Tracklet Management module that adaptively switches between end-to-end and tracking-by-detection modes (Luo et al., 1 Nov 2025).

1. Problem setting and panoramic MOT constraints

OmniTrack++ is formulated for MOT in panoramic imagery, where the input is a sequence of panoramic frames $f_1,f_2,\dots,f_T$ of resolution $H\times W$ and the output is a trajectory set with consistent identities over time (Luo et al., 1 Nov 2025). The target domain is specifically omnidirectional video captured in equirectangular projection, rather than narrow-FoV pinhole-camera imagery.

The motivating difficulty is that 360° FoV introduces resolution dilution, heavy geometric distortion, non-uniform illumination, and a search space in which every object can potentially appear anywhere around the camera (Luo et al., 1 Nov 2025). The paper further identifies frequent occlusions and rapid ego-motion—especially on legged robots—as major causes of identity switches and lost tracklets. In the abstract, these issues are summarized as unique challenges arising from 360° FoV, resolution dilution, and severe view-dependent distortions, and conventional MOT methods designed for narrow-FoV pinhole cameras are reported to generalize unsatisfactorily under these conditions (Luo et al., 1 Nov 2025).

This framing places OmniTrack++ in a setting where geometric and temporal instability are coupled. A plausible implication is that appearance-only or motion-only association heuristics are insufficient in panoramic video because both the representation itself and the temporal matching process must be made distortion-tolerant. OmniTrack++ addresses this by feeding trajectory cues back into perception, rather than treating detection and association as strictly sequential stages.

2. DynamicSSM and feature stabilization

The DynamicSSM block is a plug-in to the Transformer encoder that stabilizes panoramic features by implicitly correcting distortion and improving photometric consistency (Luo et al., 1 Nov 2025). It operates on the mid-level backbone feature $\mathbf{S}_4$ of size $B\times C\times H\times W$ and proceeds in four stages.

First, the model estimates distortion and scale:

$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$

where $$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$ (Luo et al., 1 Nov 2025). Second, it applies distortion-aware refinement via dynamic convolution:

$$\mathbf{Z} = \mathcal{D}_{conv}(\mathbf{D}\odot\mathbf{S},\,\mathbf{S}_4),$$

with $\odot$ denoting elemental modulation (Luo et al., 1 Nov 2025).

Third, DynamicSSM imposes long-range consistency through a multi-directional state-space model:

$\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$0

where $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$1 scans in direction $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$2 and $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$3 is the S6 state-space transform (Luo et al., 1 Nov 2025). Fourth, it fuses features through

$\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$4

where $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$5 denotes concatenation or summation and $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$6 is a small CNN (Luo et al., 1 Nov 2025).

The stated role of this block is to correct distortion cues and impose long-range photometric smoothing, thereby providing a more stable feature $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$7 for the subsequent detection and tracking decoder (Luo et al., 1 Nov 2025). In the discussion section of the paper, this is characterized as geometry-aware feature stabilization that reduces distortion artifacts without requiring explicit spherical convolutions. That characterization is technically significant because it situates DynamicSSM as an implicit alternative to explicitly projection-aware convolutional operators, while retaining compatibility with a Transformer-based encoder-decoder pipeline.

3. FlexiTrack Instances and trajectory-informed decoding

FlexiTrack Instances carry trajectory feedback into the decoder and serve as query tokens for short-term temporal association (Luo et al., 1 Nov 2025). For each tracked object $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$8 at time $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$9, the model maintains

$f_1,f_2,\dots,f_T$0

where $f_1,f_2,\dots,f_T$1 is a learned feature vector and $f_1,f_2,\dots,f_T$2 is an anchor, such as a positional embedding (Luo et al., 1 Nov 2025).

During training, both components are perturbed by noise for regularization:

$f_1,f_2,\dots,f_T$3

At inference time, the set of feedback instances $f_1,f_2,\dots,f_T$4 from frame $f_1,f_2,\dots,f_T$5 is concatenated with the standard $f_1,f_2,\dots,f_T$6 learnable detection queries $f_1,f_2,\dots,f_T$7 and passed into the Transformer decoder (Luo et al., 1 Nov 2025). The decoder then outputs two sets of proposals: $f_1,f_2,\dots,f_T$8 for feedback-driven proposals and $f_1,f_2,\dots,f_T$9 for detection-driven proposals (Luo et al., 1 Nov 2025).

The paper states that this mechanism effectively narrows the search to regions where tracked objects were, enabling reliable short-term association even in a 360° FoV (Luo et al., 1 Nov 2025). In the discussion section, trajectory feedback is described as dramatically narrowing search and stabilizing short-term association. This suggests that the central innovation is not merely query augmentation, but a decoder-level coupling between historical track state and current-frame proposal generation.

An ablation result further clarifies the importance of this component: FlexiTrack Instance versus ground-truth denoised instances performs best when both are used, yielding 28.47 HOTA (Luo et al., 1 Nov 2025). The data block does not specify the exact ablation protocol beyond that statement, but it indicates that learned feedback instances and denoised supervisory signals are complementary.

4. ExpertTrack Memory and long-term identity preservation

For long-term robustness, OmniTrack++ maintains a per-track memory $H\times W$0 of past appearance embeddings, organized into two partitions (Luo et al., 1 Nov 2025). The Stable Identity Memory (SIM) uses the first $H\times W$1 slots for high-confidence keyframe features, while the Dynamic Interaction Memory (DIM) uses the latter $H\times W$2 slots for the most recent frames (Luo et al., 1 Nov 2025). The design is intended to combine long-term stable anchors with short-term adaptability.

A common Mixture-of-Experts (MoE) network compensates for varied panoramic distortions:

$H\times W$3

where each expert $H\times W$4 specializes, for example, in illumination or stretching, and $H\times W$5 is the query embedding for track $H\times W$6 (Luo et al., 1 Nov 2025). A gated selector then retrieves a personalized embedding $H\times W$7 from $H\times W$8, and the final enriched embedding is

$H\times W$9

with learnable balance $\mathbf{S}_4$0 (Luo et al., 1 Nov 2025).

The update rule is score-dependent. If the current detection score $\mathbf{S}_4$1, the appearance embedding is pushed into DIM, discarding the oldest entry if DIM is full. Periodically, or when $\mathbf{S}_4$2, SIM is refreshed with a keyframe embedding (Luo et al., 1 Nov 2025). The paper states that this reduces drift by combining long-term stable anchors and short-term adaptability.

The significance of ExpertTrack Memory lies in its explicit treatment of panoramic appearance variation as a heterogeneous phenomenon. Rather than storing a single feature history, the method separates stable identity evidence from recent interaction cues and mediates both through MoE routing. The discussion section attributes to this module the ability to handle long occlusions and severe appearance shifts (Luo et al., 1 Nov 2025). A plausible implication is that identity preservation is being distributed across both memory structure and expert specialization, rather than delegated to a single re-identification embedding.

5. Dual-branch inference, training objectives, and operating modes

Inference is organized by a high-level Dual-Branch Adapter that decides per track whether to apply end-to-end (E2E) updates, tracking-by-detection (TBD), or both as an ensemble (Luo et al., 1 Nov 2025). The procedure described in Algorithm 1 can be summarized as follows. For each frame $\mathbf{S}_4$3, the system extracts backbone features $\mathbf{S}_4$4–$\mathbf{S}_4$5, applies DynamicSSM to obtain instance features $\mathbf{S}_4$6, retrieves FlexiTrack Instances $\mathbf{S}_4$7 from memory, and decodes proposals $\mathbf{S}_4$8 and $\mathbf{S}_4$9 (Luo et al., 1 Nov 2025). The branch controller then examines tracklet confidences and motion statistics and activates TBD, E2E, or both. In TBD mode, it computes a cost matrix $B\times C\times H\times W$0 using hybrid distance based on appearance and location, solves assignment, and updates, initializes, or deletes tracklets. In E2E mode, it thresholds detections using $B\times C\times H\times W$1 for initialization and update. If both modes are active, their outputs are fused, for example by taking the union of high-confidence assignments, and the final tracks $B\times C\times H\times W$2 are written into ExpertTrack Memory (Luo et al., 1 Nov 2025).

Training follows a DETR-style end-to-end set-prediction formulation with added tracking terms (Luo et al., 1 Nov 2025). The detection loss $B\times C\times H\times W$3 uses Hungarian-matched classification and bounding-box regression with L1 and GIoU. The tracking loss is

$B\times C\times H\times W$4

where $B\times C\times H\times W$5 is predicted track confidence and $B\times C\times H\times W$6 is ground truth (Luo et al., 1 Nov 2025). The overall objective is

$B\times C\times H\times W$7

with $B\times C\times H\times W$8 encouraging consistent retrieval from memory, for example through contrastive loss between $B\times C\times H\times W$9 and updated SIM slots (Luo et al., 1 Nov 2025).

The inference pipeline uses two thresholds: the initialize threshold $$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$0 and the update threshold $$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$1 (Luo et al., 1 Nov 2025). The threshold study reports best performance near $$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$2, and training performance peaks around epoch 16 (Luo et al., 1 Nov 2025). These are implementation-level operating points rather than universal constants, but they indicate the regime used in the reported system.

6. Benchmarks, reported results, and ablations

To support evaluation, the work establishes the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot (Luo et al., 1 Nov 2025). The benchmark contains 44 sequences and 26,400 frames in total, and all inputs are 360° equirectangular imagery (Luo et al., 1 Nov 2025). JRDB is also used, with 5,783 training frames and 11,672 test frames from a wheeled robot exhibiting narrow oscillation (Luo et al., 1 Nov 2025).

The reported evaluation metrics are HOTA$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$3, IDF1$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$4, MOTA$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$5, OSPA$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$6, DetA/AssA (CLEAR-MOT), and FPS (Luo et al., 1 Nov 2025).

Dataset / split	Reported setting	Reported result
QuadTrack	OmniTrack++$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$7	34.90 HOTA, +15.03 over OmniTrack$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$8
QuadTrack	OmniTrack++$$\mathbf{D},\,\mathbf{S} = \mathcal{D}_{est}(\mathbf{S}_4),\quad \mathbf{S} = \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),$$9	36.08 HOTA
QuadTrack	Best pinhole-MOT baseline (OC-SORT/TBD)	$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$0–$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$1 HOTA
BipTrack	OmniTrack++$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$2	44.63 HOTA
BipTrack	OmniTrack++$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$3	44.96 HOTA
JRDB	OmniTrack++$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$4	25.50 HOTA
JRDB	OmniTrack++$$\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$$5	27.03 HOTA
JRDB	MeMOTR	29.51 HOTA

The abstract additionally reports substantial HOTA improvements of +25.5% on JRDB and +43.07% on QuadTrack over the original OmniTrack (Luo et al., 1 Nov 2025). Qualitatively, OmniTrack++ is reported to hold identity under partial occlusion and viewpoint swings where ByteTrack and SORT fail, as illustrated in Fig. 5 of the paper (Luo et al., 1 Nov 2025).

The ablation studies isolate the contribution of major modules. DynamicSSM only yields +1.04 HOTA and +1.10 IDF1; ExpertTrack Memory only yields +0.31 HOTA and +0.02 IDF1; both together yield +1.17 HOTA and +1.49 IDF1 (Luo et al., 1 Nov 2025). For DynamicSSM internals, the combination of Dconv, SSM, and Fusion achieves 28.47 HOTA versus a 27.61 baseline (Luo et al., 1 Nov 2025). These results indicate that feature stabilization contributes more strongly than memory in the isolated setting reported, while the combined configuration is best overall.

7. Interpretation, limitations, and proposed extensions

The paper identifies several strengths. Geometry-aware feature stabilization reduces distortion artifacts without explicit spherical convolutions; trajectory feedback narrows search and stabilizes short-term association; ExpertTrack Memory with MoE handles long occlusions and severe appearance shifts; adaptive E2E versus TBD switching leverages the strengths of both paradigms under varying motion; and the EmboTrack benchmark targets realistic legged-robot oscillations (Luo et al., 1 Nov 2025). Taken together, these points describe OmniTrack++ as a system-level architecture in which representation learning, temporal feedback, and inference policy are jointly tuned for panoramic robotics video.

Several limitations are also stated. The model has increased size, at 70 M parameters, and incurs added latency from DynamicSSM and memory operations (Luo et al., 1 Nov 2025). Under extremely crowded scenes, occlusion reasoning still breaks in prolonged overlaps, and memory gating plus expert routing introduce hyper-parameters that may require tuning per platform (Luo et al., 1 Nov 2025). These caveats indicate that the framework addresses—but does not eliminate—the core combinatorial difficulty of dense identity maintenance in panoramic scenes.

The paper outlines possible extensions: incorporating explicit occlusion models or depth cues from the same 360° camera, extending MoE experts to include motion-pattern specialists such as wheeled versus gait swings, developing lightweight or quantized variants of DynamicSSM for onboard use, and performing unsupervised domain adaptation of memory to new embodiments such as aerial panoramic drones (Luo et al., 1 Nov 2025). These directions remain prospective. A plausible implication is that OmniTrack++ is best understood as a modular baseline for panoramic MOT rather than a closed design, with each module exposing a distinct axis for future work.

The datasets and code are stated to be publicly available through the project repository, and the release includes EmboTrack with QuadTrack and BipTrack (Luo et al., 1 Nov 2025).