PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking

Abstract: We introduce Point-supervised Multi-Object Tracking (PS-MOT) as a cost-effective alternative to traditional bounding box supervision, shifting the focus from spatial fitting to topological center-driven representation. However, PS-MOT faces challenges, e.g., spatial ambiguity and identity drift due to the lack of explicit geometric structure and scale constraints. To address these, we propose PS-Track, a hierarchical pipeline transitioning from points to instances across data, model, and loss levels. At the data level, we introduce Temporal-Feedback Prompting (TFP) to evolve points into temporally consistent pseudo-labels using negative spatial cues and motion priors. At the model level, we design the Point-Excited Wavelet Attention (PEWA) module, which leverages semantic correlations to activate high-frequency components, ``hallucinating'' object boundaries. At the loss level, Uncertainty-Guided Gaussian Learning (UGL) models pseudo-labels as probabilistic distributions, dynamically calibrating supervision intensity. Experiments on DanceTrack, EmboTrack, SportsMOT, and JRDB demonstrate that PS-Track provides a feasible and effective point-supervised alternative across diverse tracking scenarios, establishing a new state-of-the-art for point-supervised tracking. The source code is available at https://github.com/xifen523/PS-MOT.

• The paper presents PS-Track, a framework that transforms single-point annotations into temporally consistent and spatially precise object tracks.
• It employs Temporal-Feedback Prompting, Point-Excited Wavelet Attention, and Uncertainty-Guided Gaussian Learning to refine pseudo-labels and recover object boundaries.
• Empirical evaluations show PS-MOT achieves competitive performance and reduces annotation time by approximately 64% on diverse benchmarks.

PS-MOT: A Unified Framework for Scalable Point-Supervised Multi-Object Tracking

Introduction and Motivation

Multi-Object Tracking (MOT) is fundamental to spatio-temporal perception in autonomous systems, robotics, and video understanding. Standard MOT approaches depend heavily on large-scale frame-by-frame bounding box (BBox) annotations, incurring substantial labor costs and impeding scalability—especially in panoramic and embodied contexts where annotations are even more cumbersome due to occlusions, geometric distortions, and rapidly moving crowds. The paper "PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking" (2606.30476) reframes MOT by proposing a point-supervised alternative, where only a single topological point per object per frame is annotated.

This minimalist paradigm dramatically reduces annotation complexity (0.7-0.9s for a point vs 7-10s for a box), and can enable large-scale dataset curation. However, supervision via points introduces challenges: a point lacks scale and explicit geometry, creating spatial ambiguity and exacerbating identity drift in temporal association. The central contribution of PS-MOT is the PS-Track framework, consisting of three synergistic modules that collectively bridge the gap from sparse points to dense, temporally consistent instance representations.

Figure 1: PS-MOT vs. fully supervised MOT—point prompting sharply reduces labeling time and sustains competitive performance through a topological center-driven representation.

Framework Overview

The proposed framework operates hierarchically from coarse (point annotations) to fine (instance-level identification) across three stages:

• Data Level: Temporal-Feedback Prompting (TFP) evolves points into temporally consistent, spatially disambiguated pseudo-labels using negative cues and motion priors.
• Model Level: Point-Excited Wavelet Attention (PEWA) leverages frequency decomposition (via DWT) to "hallucinate" object boundaries from point cues, encouraging the network to focus on high-frequency edge information.
• Loss Level: Uncertainty-Guided Gaussian Learning (UGL) models pseudo-labels as noisy observations, dynamically re-weighting supervision based on a learned confidence estimate.

  Figure 2: PS-Track operates on data, model, and loss levels to evolve sparse points into robust, temporally consistent instance tracks.

Data Level: Temporal-Feedback Prompting (TFP)

TFP addresses two point supervision issues—identity merging (overlapping masks) and semantic fragmentation (partial segmentation under occlusion). It augments SAM-based pseudo-labeling with:

• Negative cues: Nearby point annotations are provided as negative samples to SAM, constructing a semantic firewall and separating adjacent targets.
• Motion priors: Kalman filter tracklets provide a spatial constraint by restricting mask generation to predicted object locations.
• Joint Quality Score: Each box is assigned a confidence score factoring in both visual confidence and temporal consistency; this moderates downstream supervision.

Qualitative comparison highlights that TFP yields more accurate and robust pseudo-labels than the original, static point-prompted SAM outputs.

Figure 3: TFP refines pseudo-labels from SAM, enhancing spatial correspondence and robustness.

Model Level: Point-Excited Wavelet Attention (PEWA)

PEWA addresses the location-boundary mismatch inherent in point supervision. CNNs struggle to infer high-frequency edge information from points alone. PEWA:

• Decomposes feature maps into low- and high-frequency domains using a single-level Haar DWT.
• Uses spatial Gaussian heatmaps centered at point annotations to generate frequency excitation masks, biasing the network toward edges near annotated centers.
• Suppresses background noise and amplifies target-adjacent edges in the frequency domain.
• Recombines these representations via IDWT to produce features with sharpened boundaries that improve association embeddings.

The PEWA module operates only during training; at inference, the architecture reverts to standard inputs.

Loss Level: Uncertainty-Guided Gaussian Learning (UGL)

UGL mitigates the risk of overfitting to noisy or unreliable pseudo-labels. Rather than regressing to a fixed box, it:

• Treats each pseudo-label as a sample from a Gaussian centered on the true (unknown) target parameters, with variance proportional to the joint quality score.
• Implements a negative log-likelihood loss, dynamically re-weighting contributions from unreliable samples and introducing a probabilistic calibration term.
• This enables robust convergence even when point annotations or pseudo-labels are imperfect.

Empirical Results

Evaluations are presented across diverse benchmarks—DanceTrack, SportsMOT, EmboTrack (panoramic, ego-motion), and JRDB:

• On DanceTrack, PS-Track achieves 52.3 HOTA and 53.4 IDF1 with point-only annotations, surpassing classic fully supervised detectors like CenterTrack (41.8 HOTA) and FairMOT (39.7 HOTA) and narrowing the gap to the state-of-the-art.
• On SportsMOT and JRDB, performance is competitive with fully supervised models, and PS-Track outperforms certain recent baselines, supporting the claim that point supervision is viable—even under severe occlusions and scale variations.
• Annotation time is reduced by approximately 64% compared to conventional protocols on controlled 600-frame settings.

  Figure 4: Qualitative comparison on DanceTrack—PS-Track maintains bounding box quality and identity stability in challenging, dynamic scenarios.

  

Figure 5: Illustration of the DanceTrack dataset, which emphasizes complex, nonlinear motion and cluttered scenes.

Figure 6: Visualization of point annotation evolution, showing spatio-temporal consistency across frames.

Figure 7: Additional qualitative results—PS-Track handles severe occlusion and pose deformation in DanceTrack.

Figure 8: PS-Track on JRDB, sustaining accurate tracking in crowded, ego-centric scenes.

Figure 9: PS-Track on SportsMOT, highlighting robustness to fast motion and complex interactions.

Ablation and Generalization Analysis

Ablation studies emphasize the necessity of integrating TFP, PEWA, and UGL. Removal of any module causes substantial performance degradation, with TFP contributing the largest HOTA gains by preventing identity merging and fragmentation. The pipeline demonstrates robustness to synthetic annotation noise, stable convergence, and insensitivity to moderate loss hyperparameter perturbations.

PS-Track generalizes as a plug-and-play module: it can be integrated into association-based, autoregressive, and query-based tracking pipelines. Across all tested paradigms, the accuracy gap relative to fully supervised baselines shrinks significantly (BYTE+PS-Track: 42.9 vs 47.1 HOTA).

Implications and Future Directions

PS-MOT, via PS-Track, establishes the feasibility of instance-aware, identity-consistent tracking with drastically reduced annotation labor. Relative to the spatially rigid, labor-intensive BBox paradigm, point supervision excels in challenging geometric contexts (e.g., extreme fisheye, panoramic robotics) and domains where scale-precise annotation is ill-defined. Practically, this will enable the expansion of MOT datasets into new domains and reduce curation bottlenecks.

Nevertheless, certain limitations persist: separation of extremely overlapping or occluded instances remains challenging for the negative cue mechanisms; motion blur inherently weakens frequency-domain priors; and the evaluation protocol synthesizes point labels from BBox centers, which could underestimate real-world annotation noise. Temporal sparsity in point annotation and modalities beyond points (language prompts, scribbles) pose natural future research questions. Extending this paradigm to 3D point clouds or video-volume MOT—for autonomous vehicles and spatial computing—represents a compelling direction, given the prohibitive cost of 3D BBox labeling.

Conclusion

PS-MOT and the PS-Track framework make a strong case for the scalability and practicality of point supervision in multi-object tracking. The proposed combination of temporal feedback for pseudo-label refinement, wavelet-domain frequency excitation for boundary reconstruction, and probabilistic loss calibration for supervision noise enables near-state-of-the-art results with a fraction of annotation labor. This approach provides a principled, technically robust foundation for future work in label-efficient video and embodied perception, and opens avenues for large-scale tracking in previously intractable domains.