---
title: 'OmniTrack++: Panoramic MOT Framework'
url: https://www.emergentmind.com/topics/omnitrack
type: topic
---

# OmniTrack++: Panoramic MOT Framework

OmniTrack++ is a framework for multi-object tracking (MOT) on 360° panoramic video that addresses panoramic distortion, large search space, and identity ambiguity under a 360° field of view (FoV) through a feedback-driven design that progressively refines perception with trajectory cues [2511.00510]. It operates on panoramic frames, typically in equirectangular format, and outputs a set of trajectories $\mathcal{T}=\{\tau_1,\tau_2,\dots\}$, where each trajectory is a time-ordered list of detections $\{(t,x_t^i,y_t^i,w_t^i,h_t^i)\}$ with a consistent identity label. The method combines a DynamicSSM block for feature stabilization, FlexiTrack Instances for short-term trajectory-informed association, an ExpertTrack Memory for long-term identity preservation, and a Tracklet Management module that adaptively switches between end-to-end and tracking-by-detection modes [2511.00510].

## 1. Problem setting and panoramic MOT constraints

OmniTrack++ is formulated for MOT in panoramic imagery, where the input is a sequence of panoramic frames $f_1,f_2,\dots,f_T$ of resolution $H\times W$ and the output is a trajectory set with consistent identities over time [2511.00510]. The target domain is specifically omnidirectional video captured in equirectangular projection, rather than narrow-FoV pinhole-camera imagery.

The motivating difficulty is that 360° FoV introduces resolution dilution, heavy geometric distortion, non-uniform illumination, and a search space in which every object can potentially appear anywhere around the camera [2511.00510]. The paper further identifies frequent occlusions and rapid ego-motion—especially on legged robots—as major causes of identity switches and lost tracklets. In the abstract, these issues are summarized as unique challenges arising from 360° FoV, resolution dilution, and severe view-dependent distortions, and conventional MOT methods designed for narrow-FoV pinhole cameras are reported to generalize unsatisfactorily under these conditions [2511.00510].

This framing places OmniTrack++ in a setting where geometric and temporal instability are coupled. A plausible implication is that appearance-only or motion-only association heuristics are insufficient in panoramic video because both the representation itself and the temporal matching process must be made distortion-tolerant. OmniTrack++ addresses this by feeding trajectory cues back into perception, rather than treating detection and association as strictly sequential stages.

## 2. DynamicSSM and feature stabilization

The DynamicSSM block is a plug-in to the Transformer encoder that stabilizes panoramic features by implicitly correcting distortion and improving photometric consistency [2511.00510]. It operates on the mid-level backbone feature $\mathbf{S}_4$ of size $B\times C\times H\times W$ and proceeds in four stages.

First, the model estimates distortion and scale:
$$
\mathbf{D},\,\mathbf{S}
= \mathcal{D}_{est}(\mathbf{S}_4),\quad
\mathbf{S}
= \sigma\bigl(\mathcal{S}_{est}(\mathbf{S}_4)\bigr),
$$
where $\mathbf{D},\mathbf{S}\in\mathbb{R}^{B\times C\times H\times W}$ [2511.00510]. Second, it applies distortion-aware refinement via dynamic convolution:
$$
\mathbf{Z}
= \mathcal{D}_{conv}(\mathbf{D}\odot\mathbf{S},\,\mathbf{S}_4),
$$
with $\odot$ denoting elemental modulation [2511.00510].

Third, DynamicSSM imposes long-range consistency through a multi-directional state-space model:
$$
\mathbf{Z}^\star
= \frac1L\sum_{d=1}^L F_{S6}\bigl(\mathcal{S}_d(\mathbf{Z})\bigr),
$$
where $\mathcal{S}_d$ scans in direction $d$ and $F_{S6}$ is the S6 state-space transform [2511.00510]. Fourth, it fuses features through
$$
\mathbf{F}
= \mathcal{F}\bigl(\mathrm{Conv}(\mathbf{S}_4)\oplus \mathbf{Z}^\star\bigr),
$$
where $\oplus$ denotes concatenation or summation and $\mathcal{F}$ is a small CNN [2511.00510].

The stated role of this block is to correct distortion cues and impose long-range photometric smoothing, thereby providing a more stable feature $\mathbf{F}$ for the subsequent detection and tracking decoder [2511.00510]. In the discussion section of the paper, this is characterized as geometry-aware feature stabilization that reduces distortion artifacts without requiring explicit spherical convolutions. That characterization is technically significant because it situates DynamicSSM as an implicit alternative to explicitly projection-aware convolutional operators, while retaining compatibility with a Transformer-based encoder-decoder pipeline.

## 3. FlexiTrack Instances and trajectory-informed decoding

FlexiTrack Instances carry trajectory feedback into the decoder and serve as query tokens for short-term temporal association [2511.00510]. For each tracked object $i$ at time $t-1$, the model maintains
$$
\mathcal{I}_F^i = (\mathcal{X}_i,\mathcal{Y}_i)\in\mathbb{R}^{c_s}\times\mathbb{R}^{c_s},
$$
where $\mathcal{X}_i$ is a learned feature vector and $\mathcal{Y}_i$ is an anchor, such as a positional embedding [2511.00510].

During training, both components are perturbed by noise for regularization:
$$
\mathcal{X}' = \mathcal{X} + \mathcal{N}_X,\quad
\mathcal{Y}' = \mathcal{Y} + \mathcal{N}_Y.
$$
At inference time, the set of feedback instances $\{\mathcal{I}_F^i\}$ from frame $t-1$ is concatenated with the standard $N_q$ learnable detection queries $\{\mathcal{I}_L^j\}$ and passed into the Transformer decoder [2511.00510]. The decoder then outputs two sets of proposals: $\mathcal{D}_t^F$ for feedback-driven proposals and $\mathcal{D}_t^L$ for detection-driven proposals [2511.00510].

The paper states that this mechanism effectively narrows the search to regions where tracked objects were, enabling reliable short-term association even in a 360° FoV [2511.00510]. In the discussion section, trajectory feedback is described as dramatically narrowing search and stabilizing short-term association. This suggests that the central innovation is not merely query augmentation, but a decoder-level coupling between historical track state and current-frame proposal generation.

An ablation result further clarifies the importance of this component: FlexiTrack Instance versus ground-truth denoised instances performs best when both are used, yielding 28.47 HOTA [2511.00510]. The data block does not specify the exact ablation protocol beyond that statement, but it indicates that learned feedback instances and denoised supervisory signals are complementary.

## 4. ExpertTrack Memory and long-term identity preservation

For long-term robustness, OmniTrack++ maintains a per-track memory $\mathcal{M}_i$ of past appearance embeddings, organized into two partitions [2511.00510]. The Stable Identity Memory (SIM) uses the first $n_m/2$ slots for high-confidence keyframe features, while the Dynamic Interaction Memory (DIM) uses the latter $n_m/2$ slots for the most recent frames [2511.00510]. The design is intended to combine long-term stable anchors with short-term adaptability.

A common Mixture-of-Experts (MoE) network compensates for varied panoramic distortions:
$$
\mathbf{f}_i^{\mathrm{sh}}
= \sum_{k=1}^{n_e}\alpha_{i,k}\,E_k(\mathbf{q}_i),\quad
\alpha_{i,k}
= \frac{\exp(\mathrm{score}(\mathbf{q}_i,E_k))}{\sum_j\exp(\mathrm{score}(\mathbf{q}_i,E_j))},
$$
where each expert $E_k$ specializes, for example, in illumination or stretching, and $\mathbf{q}_i$ is the query embedding for track $i$ [2511.00510]. A gated selector then retrieves a personalized embedding $\mathbf{f}_i^{\mathrm{pl}}$ from $\mathcal{M}_i$, and the final enriched embedding is
$$
\hat{\mathbf{f}}_i
= \lambda_i\,\mathbf{f}_i^{\mathrm{sh}}
+ (1-\lambda_i)\,\mathbf{f}_i^{\mathrm{pl}},
$$
with learnable balance $\lambda_i$ [2511.00510].

The update rule is score-dependent. If the current detection score $s_t^i>\tau_\mathrm{update}$, the appearance embedding is pushed into DIM, discarding the oldest entry if DIM is full. Periodically, or when $s_t^i>\tau_\mathrm{key}$, SIM is refreshed with a keyframe embedding [2511.00510]. The paper states that this reduces drift by combining long-term stable anchors and short-term adaptability.

The significance of ExpertTrack Memory lies in its explicit treatment of panoramic appearance variation as a heterogeneous phenomenon. Rather than storing a single feature history, the method separates stable identity evidence from recent interaction cues and mediates both through MoE routing. The discussion section attributes to this module the ability to handle long occlusions and severe appearance shifts [2511.00510]. A plausible implication is that identity preservation is being distributed across both memory structure and expert specialization, rather than delegated to a single re-identification embedding.

## 5. Dual-branch inference, training objectives, and operating modes

Inference is organized by a high-level Dual-Branch Adapter that decides per track whether to apply end-to-end (E2E) updates, tracking-by-detection (TBD), or both as an ensemble [2511.00510]. The procedure described in Algorithm 1 can be summarized as follows. For each frame $t$, the system extracts backbone features $\mathbf{S}_3$–$\mathbf{S}_5$, applies DynamicSSM to obtain instance features $\mathcal{I}_L$, retrieves FlexiTrack Instances $\mathcal{I}_F$ from memory, and decodes proposals $\mathcal{D}_t^F$ and $\mathcal{D}_t^L$ [2511.00510]. The branch controller then examines tracklet confidences and motion statistics and activates TBD, E2E, or both. In TBD mode, it computes a cost matrix $C_{ij}$ using hybrid distance based on appearance and location, solves assignment, and updates, initializes, or deletes tracklets. In E2E mode, it thresholds detections using $d.\mathrm{score}>\{\tau_I,\tau_U\}$ for initialization and update. If both modes are active, their outputs are fused, for example by taking the union of high-confidence assignments, and the final tracks $\mathcal{T}_t$ are written into ExpertTrack Memory [2511.00510].

Training follows a DETR-style end-to-end set-prediction formulation with added tracking terms [2511.00510]. The detection loss $\mathcal{L}_\mathrm{det}$ uses Hungarian-matched classification and bounding-box regression with L1 and GIoU. The tracking loss is
$$
\mathcal{L}_\mathrm{track}
= \lambda_\mathrm{score}\,\|\hat s - s^\star\|_2^2
+ \lambda_\mathrm{ID}\,\mathrm{CE}(p_\mathrm{ID},y_\mathrm{ID}),
$$
where $\hat s$ is predicted track confidence and $s^\star$ is ground truth [2511.00510]. The overall objective is
$$
\mathcal{L}
= \mathcal{L}_\mathrm{det}
+ \mathcal{L}_\mathrm{track}
+ \lambda_\mathrm{memory}\,\mathcal{L}_\mathrm{mem},
$$
with $\mathcal{L}_\mathrm{mem}$ encouraging consistent retrieval from memory, for example through contrastive loss between $\mathbf{f}_i^{\mathrm{pl}}$ and updated SIM slots [2511.00510].

The inference pipeline uses two thresholds: the initialize threshold $\tau_I$ and the update threshold $\tau_U$ [2511.00510]. The threshold study reports best performance near $\tau_I\approx0.5,\tau_U\approx0.5$, and training performance peaks around epoch 16 [2511.00510]. These are implementation-level operating points rather than universal constants, but they indicate the regime used in the reported system.

## 6. Benchmarks, reported results, and ablations

To support evaluation, the work establishes the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot [2511.00510]. The benchmark contains 44 sequences and 26,400 frames in total, and all inputs are 360° equirectangular imagery [2511.00510]. JRDB is also used, with 5,783 training frames and 11,672 test frames from a wheeled robot exhibiting narrow oscillation [2511.00510].

The reported evaluation metrics are HOTA$\uparrow$, IDF1$\uparrow$, MOTA$\uparrow$, OSPA$\downarrow$, DetA/AssA (CLEAR-MOT), and FPS [2511.00510].

| Dataset / split | Reported setting | Reported result |
|---|---|---|
| QuadTrack | OmniTrack++$_{(\mathrm{E2E})}$ | 34.90 HOTA, +15.03 over OmniTrack$_{(\mathrm{E2E})}$ |
| QuadTrack | OmniTrack++$_{(\mathrm{DA})}$ | 36.08 HOTA |
| QuadTrack | Best pinhole-MOT baseline (OC-SORT/TBD) | $\sim 20$–$22$ HOTA |
| BipTrack | OmniTrack++$_{(\mathrm{E2E})}$ | 44.63 HOTA |
| BipTrack | OmniTrack++$_{(\mathrm{DA})}$ | 44.96 HOTA |
| JRDB | OmniTrack++$_{(\mathrm{E2E})}$ | 25.50 HOTA |
| JRDB | OmniTrack++$_{(\mathrm{DA})}$ | 27.03 HOTA |
| JRDB | MeMOTR | 29.51 HOTA |

The abstract additionally reports substantial HOTA improvements of +25.5% on JRDB and +43.07% on QuadTrack over the original OmniTrack [2511.00510]. Qualitatively, OmniTrack++ is reported to hold identity under partial occlusion and viewpoint swings where ByteTrack and SORT fail, as illustrated in Fig. 5 of the paper [2511.00510].

The ablation studies isolate the contribution of major modules. DynamicSSM only yields +1.04 HOTA and +1.10 IDF1; ExpertTrack Memory only yields +0.31 HOTA and +0.02 IDF1; both together yield +1.17 HOTA and +1.49 IDF1 [2511.00510]. For DynamicSSM internals, the combination of Dconv, SSM, and Fusion achieves 28.47 HOTA versus a 27.61 baseline [2511.00510]. These results indicate that feature stabilization contributes more strongly than memory in the isolated setting reported, while the combined configuration is best overall.

## 7. Interpretation, limitations, and proposed extensions

The paper identifies several strengths. Geometry-aware feature stabilization reduces distortion artifacts without explicit spherical convolutions; trajectory feedback narrows search and stabilizes short-term association; ExpertTrack Memory with MoE handles long occlusions and severe appearance shifts; adaptive E2E versus TBD switching leverages the strengths of both paradigms under varying motion; and the EmboTrack benchmark targets realistic legged-robot oscillations [2511.00510]. Taken together, these points describe OmniTrack++ as a system-level architecture in which representation learning, temporal feedback, and inference policy are jointly tuned for panoramic robotics video.

Several limitations are also stated. The model has increased size, at 70 M parameters, and incurs added latency from DynamicSSM and memory operations [2511.00510]. Under extremely crowded scenes, occlusion reasoning still breaks in prolonged overlaps, and memory gating plus expert routing introduce hyper-parameters that may require tuning per platform [2511.00510]. These caveats indicate that the framework addresses—but does not eliminate—the core combinatorial difficulty of dense identity maintenance in panoramic scenes.

The paper outlines possible extensions: incorporating explicit occlusion models or depth cues from the same 360° camera, extending MoE experts to include motion-pattern specialists such as wheeled versus gait swings, developing lightweight or quantized variants of DynamicSSM for onboard use, and performing unsupervised domain adaptation of memory to new embodiments such as aerial panoramic drones [2511.00510]. These directions remain prospective. A plausible implication is that OmniTrack++ is best understood as a modular baseline for panoramic MOT rather than a closed design, with each module exposing a distinct axis for future work.

The datasets and code are stated to be publicly available through the project repository, and the release includes EmboTrack with QuadTrack and BipTrack [2511.00510].

Source: https://www.emergentmind.com/topics/omnitrack