SAM-Deep-EIoU: Selective Mask Propagation for Multi-Object Tracking

Abstract: Multi-object tracking has a heavy-tailed difficulty distribution: most frames are easy for a lightweight base tracker, while a small fraction are intrinsically hard. Video object segmentation (VOS) models can often preserve identity through the hard frames where the base tracker fails, but they are much more expensive in compute and memory. We propose selective mask propagation, a tracking algorithm that dispatches from a base tracker to a VOS model only on windows where an assignment-uncertainty signal fires. The base tracker's output is modified only when the VOS model makes a confident prediction that contradicts the base tracker's identity assignment; weak or inconclusive predictions preserve the base output. The method is training-free, treats both the base tracker and the VOS model as black boxes, and can benefit from replacing the VOS component with a more capable model. On DanceTrack, selective mask propagation improves three different base trackers. On SportsMOT, where identity preservation is central to sports analytics, SAM3-Deep-EIoU with global track association achieves state-of-the-art performance on the benchmark with 86.8 HOTA.

• The paper proposes a novel method that uses assignment margin cues to trigger a VOS module only for ambiguous frames, reducing computational overhead.
• It employs a training-free, modular design that integrates Deep-EIoU tracking with SAM-3 to correct identity assignments during occlusions.
• Empirical results on SportsMOT and DanceTrack show significant HOTA improvements and robust handling of complex multi-agent interactions.

Selective Mask Propagation for Multi-Object Tracking: A Technical Analysis of SAM-Deep-EIoU

Introduction

The paper "SAM-Deep-EIoU: Selective Mask Propagation for Multi-Object Tracking" (2606.13033) addresses the computational and analytic challenges inherent in Multi-Object Tracking (MOT). The authors note the heavy-tailed distribution of frame difficulty in MOT tasks, observing that a lightweight tracking-by-detection (“base”) tracker suffices for the majority of frames, but fails in a small, critical subset where occlusions and near-tied assignments induce identity switches. The proposed method—selective mask propagation—deploys a Video Object Segmentation (VOS) model only in these ambiguous temporal windows, guided by a dispatch signal derived from the assignment margins of the base tracker. This modular and training-free design is instantiated as SAM-Deep-EIoU, a system that leverages Deep-EIoU tracking and the third-generation Segment Anything Model (SAM-3) for selective mask-based association.

Frame Difficulty and the Dispatch Mechanism

The authors characterize the frame-level assignment ambiguity by computing the assignment margin from the Hungarian cost matrix. The heavy-tailed distribution of frame difficulty is empirically validated on SportsMOT: few frames are ambiguous, but these induce downstream error propagation if not handled with more powerful cues.

Figure 1: The distribution of frame difficulty on SportsMOT confirms that most frames are easily solved by lightweight association, whereas a small fraction present ambiguity that challenges assignment consistency.

Selective mask propagation is triggered by low assignment margins—specifically, when the difference between the best and second-best tracker assignments for a detection is below a set threshold. The window-based design includes backward seeding to guarantee that mask propagation commences from an unambiguous frame. Additional signals (gap and witness cues) cover cases not captured by simple margin analysis, ensuring coverage even when tracker gaps or multiple interacting agents are present.

Qualitative and Window-Based Outcomes

An explicit demonstration of the method’s efficacy is provided in qualitative MOT scenarios. For instance, during complex occlusions, Deep-EIoU alone loses identity assignment, whereas selective mask propagation with SAM preserves and recovers correct player identities.

Figure 2: Qualitative instance from SportsMOT, showing Deep-EIoU versus SAM-Deep-EIoU in an occlusion context. Mask seeding and propagation correctly resolve player identities after occlusion.

A window’s lifetime includes propagation, exit checks based on mask-to-tracker association via intersection-over-mask-area (IoMA), and multiple possible outcomes—SWAPs (identity corrections), CLEAN (no change needed), STALE, DEGRADED, EDGE, and END. The asymmetric update rule ensures the base tracker's output alters only if the VOS model is confident and contradictory.

Figure 3: Lifecycle of a window, displaying seeding, propagation through ambiguity, and resolution via exit conditions.

Figure 4: Final outcome distribution of windows with selective mask propagation indicates most windows leave the base output unchanged, with SWAP outcomes representing meaningful corrections.

Generalization and State-of-the-Art Results

The method is benchmarked on both DanceTrack and SportsMOT, spanning varied tracking dynamics and domains. Selective mask propagation provides consistent improvements—both in HOTA and association accuracy (AssA)—across different base trackers (SORT, ByteTrack, Deep-EIoU) and outperforms per-frame mask integration methods by applying VOS selectively, thereby controlling computational overhead.

On SportsMOT, SAM-Deep-EIoU, augmented by global track association (GTA) incorporating jersey and team identity cues, achieves SOTA performance (86.8 HOTA), with per-task deltas indicating that the largest gains manifest in high-interaction sports like basketball. The modularity is validated by showing that improved VOS models (e.g., SAM-3 replacing SAM-2) yield better MOT without any change to system logic or hyperparameters.

Theoretical and Practical Implications

This work demonstrates that careful dispatch of expensive computation addresses the characteristic heterogeneity in MOT difficulty. The assignment-margin cue is a strong, model-agnostic heuristic for ambiguity detection, compatible with black-box deployment of both tracker and VOS modules. The asymmetric update mechanism further constrains the space of possible detrimental interventions, reducing the method’s risk profile relative to constant VOS usage.

The modular treatment enables drop-in upgrades as VOS architectures develop, without necessitating re-engineering. Furthermore, the same framework can incorporate domain-specific association modules (as in sports analytics), enabling the separation of on-screen ambiguity resolution from off-screen re-identification.

Future Directions

The main limitation is the cost-benefit tradeoff inherent to VOS dispatch, as most windows eventually leave the base tracker's output unchanged, especially in less challenging scenes. The design’s future improvements will likely focus on enhancing the precision of dispatch signals and integrating even more discriminative signals (e.g., uncertainty-aware assignment, learned temporal priors) to further constrain VOS invocation. Advances in low-latency VOS models and joint formulation with global association modules are natural stages for further gains.

Conclusion

SAM-Deep-EIoU introduces a robust, modular approach to MOT that exploits the sporadic necessity for high-compute mask-based association. The system’s training-free pipeline, black-box compatibility, and empirical effectiveness across multiple trackers and sports scenarios reinforce its practicality. The methodological separation of dispatch logic, outcome integration, and downstream identity association positions selective mask propagation as a foundation for hybrid tracking architectures, adaptable to evolving segmentation and tracking methods.