OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback

Abstract: This paper investigates Multi-Object Tracking (MOT) in panoramic imagery, which introduces unique challenges including a 360{\deg} Field of View (FoV), resolution dilution, and severe view-dependent distortions. Conventional MOT methods designed for narrow-FoV pinhole cameras generalize unsatisfactorily under these conditions. To address panoramic distortion, large search space, and identity ambiguity under a 360{\deg} FoV, OmniTrack++ adopts a feedback-driven framework that progressively refines perception with trajectory cues. A DynamicSSM block first stabilizes panoramic features, implicitly alleviating geometric distortion. On top of normalized representations, FlexiTrack Instances use trajectory-informed feedback for flexible localization and reliable short-term association. To ensure long-term robustness, an ExpertTrack Memory consolidates appearance cues via a Mixture-of-Experts design, enabling recovery from fragmented tracks and reducing identity drift. Finally, a Tracklet Management module adaptively switches between end-to-end and tracking-by-detection modes according to scene dynamics, offering a balanced and scalable solution for panoramic MOT. To support rigorous evaluation, we establish the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot. Together, these datasets span wide-angle environments and diverse motion patterns, providing a challenging testbed for real-world panoramic perception. Extensive experiments on JRDB and EmboTrack demonstrate that OmniTrack++ achieves state-of-the-art performance, yielding substantial HOTA improvements of +25.5% on JRDB and +43.07% on QuadTrack over the original OmniTrack. Datasets and code will be made publicly available at https://github.com/xifen523/OmniTrack.

• The paper introduces a feedback-driven framework that unifies E2E and TBD paradigms for robust multi-object tracking in panoramic imagery.
• It leverages modules like DynamicSSM, FlexiTrack, and ExpertTrack Memory to mitigate distortions and maintain identity consistency despite dynamic scene changes.
• Experimental results on the EmboTrack benchmark show significant improvements in HOTA and IDF1 metrics, confirming the framework's scalability and effectiveness.

OmniTrack++: A Unified Feedback-Driven Framework for Panoramic Multi-Object Tracking

Introduction and Motivation

OmniTrack++ addresses the unique challenges of multi-object tracking (MOT) in panoramic imagery, characterized by $360^\circ$ field-of-view (FoV), severe geometric distortions, resolution dilution, and dynamic scene perturbations. Conventional MOT algorithms, designed for narrow-FoV pinhole cameras, exhibit significant performance degradation when applied to panoramic data due to increased identity switches, poor localization, and unstable associations. OmniTrack++ introduces a feedback-driven architecture that leverages trajectory-informed cues, geometric correction, and adaptive paradigm switching to achieve robust, identity-consistent tracking in wide-FoV environments.

Figure 1: Comparison of mainstream tracking paradigms. OmniTrack++ adaptively switches between E2E and TBD paradigms, integrating trajectory-feedback for improved panoramic MOT.

Framework Overview

OmniTrack++ is composed of four synergistic modules:

• DynamicSSM Block: Mitigates panoramic distortions and photometric inconsistencies, providing stabilized spatial features.
• FlexiTrack Instance: Encodes trajectory-informed feedback for precise short-term localization and association.
• ExpertTrack Memory: Maintains long-term identity consistency via a hierarchical memory bank and a Mixture-of-Experts (MoE) design.
• Tracklet Management: Dynamically switches between End-to-End (E2E) and Tracking-by-Detection (TBD) paradigms, balancing efficiency and robustness.

The feedback mechanism reinjects historical trajectory information into the perception pipeline, reducing uncertainty and improving temporal coherence.

Figure 2: Pipeline overview of OmniTrack++. The feedback loop integrates DynamicSSM, FlexiTrack, ExpertTrack Memory, and adaptive paradigm switching for robust panoramic MOT.

DynamicSSM Block

The DynamicSSM Block is a plug-in enhancement to the DAB Transformer encoder, implicitly calibrating spatial and photometric feature distributions. It operates in four stages:

1. Distortion and Scale Estimation: Predicts geometric deformation and scale priors from backbone features.
2. Distortion-Aware Refinement: Applies dynamic convolution modulated by predicted cues.
3. Long-Range Consistency: Utilizes multi-directional State Space Models (SSMs) for photometric stability.
4. Feature Fusion: Integrates refined features with a residual CNN branch.

This block yields robust representations, essential for decoding and tracking in panoramic scenes.

Figure 3: DynamicSSM Block implicitly calibrates features to mitigate geometric and photometric distortions in panoramic imagery.

FlexiTrack Instance and Feedback Mechanism

FlexiTrack Instances encode trajectory-informed feedback, guiding the decoder’s attention to relevant spatial regions and supporting precise temporal association. Each instance comprises a feature vector and anchor, regularized with stochastic perturbations to enhance generalization. During inference, FlexiTrack Instances from previous frames are concatenated with current learnable instances, enabling the decoder to exploit both detection-driven and trajectory-informed cues.

ExpertTrack Memory

ExpertTrack Memory is a hierarchical memory module integrating:

• Stable Identity Memory (SIM): Stores high-confidence, long-term identity features.
• Dynamic Interaction Memory (DIM): Captures short-term appearance and motion variations.

A Hierarchical Memory Controller assigns features to SIM or DIM, and a Router selects top-$K_r$ features for processing by a Shared MoE. Each expert specializes in compensating for specific appearance variations (illumination, distortion, etc.), and outputs are fused to form robust FlexiTrack Instances.

Figure 4: ExpertTrack Memory combines long-term and short-term memories with a Mixture-of-Experts for adaptive identity association.

Tracklet Management and Adaptive Paradigm Switching

Tracklet Management employs a Dual-Branch Adapter to select between E2E and TBD paradigms based on scene dynamics. E2E is preferred for stable motion and high detection quality, while TBD is activated under occlusion, re-entry, or appearance ambiguity. Ensemble mode fuses outputs from both branches, mitigating weaknesses and enhancing robustness in complex panoramic sequences.

EmboTrack Benchmark

To support rigorous evaluation, the EmboTrack benchmark is introduced, comprising:

• QuadTrack: Panoramic sequences from a quadrupedal robot, featuring gait-induced oscillations and non-linear motion.
• BipTrack: Sequences from a wheeled-legged robot, introducing pitch variations and hybrid locomotion dynamics.

EmboTrack spans 44 sequences, 26,400 annotated frames, and diverse motion patterns, providing a challenging testbed for panoramic MOT.

Figure 5: EmboTrack benchmark overview and MOT results. OmniTrack++ achieves highest HOTA and IDF1 on QuadTrack.

Figure 6: Data collection platforms for QuadTrack and BipTrack, illustrating motion-induced noise characteristics.

Figure 7: Instance motion trajectories over time for JRDB, QuadTrack, and BipTrack, highlighting platform-induced motion perturbations.

Figure 8: Distribution of bounding box sizes in EmboTrack, showing balanced scale representation across splits.

Experimental Results

OmniTrack++ demonstrates substantial improvements over prior methods:

• QuadTrack (E2E): HOTA 34.90 (+43%), IDF1 41.21 (+52%) over original OmniTrack.
• QuadTrack (TBD): HOTA 36.08 (+35%), IDF1 42.76 (+38%).
• BipTrack (E2E/TBD): HOTA 44.63/44.96, outperforming all baselines.
• JRDB (E2E): HOTA 25.50, IDF1 28.00, narrowing the gap to TBD methods.

Ablation studies confirm the complementary contributions of DynamicSSM and ExpertTrack Memory, with joint integration yielding the highest performance. The feedback mechanism consistently improves tracking metrics under both E2E and TBD paradigms.

Figure 9: Query localization visualization. ExpertTrack Memory maintains consistent focus on targets; absence leads to unstable localization.

Figure 10: Effects of trajectory initialization and update thresholds on HOTA in OmniTrack++$_{E2E}$.

Figure 11: Visual comparison of MOT methods on JRDB. OmniTrack++ maintains consistent associations under occlusion and motion dynamics.

Limitations and Failure Analysis

Despite strong performance, OmniTrack++ encounters challenges in dense crowds and severe occlusions, occasionally producing fragmented trajectories or identity switches. These failure cases highlight the need for more explicit occlusion-aware modeling and advanced temporal reasoning.

Figure 12: Failure case analysis on JRDB, illustrating scenarios where OmniTrack++ struggles compared to ByteTrack.

Implications and Future Directions

OmniTrack++ establishes a robust, scalable framework for panoramic multi-object tracking, integrating geometric correction, trajectory-informed feedback, and adaptive paradigm switching. The EmboTrack benchmark sets a new standard for evaluating MOT algorithms under realistic robotic locomotion and panoramic conditions. Future research should focus on:

• Enhanced occlusion-aware modeling and temporal reasoning.
• Extension to long-term tracking in highly dynamic, real-world environments.
• Domain adaptation and transfer learning for diverse robotic platforms.

Conclusion

OmniTrack++ advances the state-of-the-art in panoramic multi-object tracking by unifying E2E and TBD paradigms within a feedback-driven architecture. The integration of DynamicSSM, FlexiTrack Instances, ExpertTrack Memory, and adaptive Tracklet Management yields substantial improvements in tracking accuracy, identity preservation, and robustness under challenging wide-FoV conditions. The EmboTrack benchmark provides a comprehensive evaluation platform, facilitating further research in embodied panoramic perception. While challenges remain in crowded and highly dynamic scenes, OmniTrack++ lays a solid foundation for future developments in panoramic MOT and embodied AI.