PS-MOT: Point-Supervised Multi-Object Tracking

• The paper introduces a novel point-supervision paradigm that replaces dense bounding boxes with single point annotations, reducing labeling time by up to 64%.
• PS-MOT addresses the precision–ambiguity paradox by integrating Temporal-Feedback Prompting, Point-Excited Wavelet Attention, and Uncertainty-Guided Gaussian Learning for robust instance recovery.
• Empirical evaluations on datasets like DanceTrack, SportsMOT, JRDB, and EmboTrack demonstrate that the unified PS-MOT framework maintains temporal coherence and improves tracking performance under geometric distortions.

Searching arXiv for the cited PS-MOT and related point-based tracking papers. Point-supervised Multi-Object Tracking (PS-MOT) is a supervision paradigm for multi-object tracking in which classical per-frame bounding box annotation is replaced by a single point annotation per instance, interpreted as a topological center rather than a geometric extent. In the formulation introduced by "PS-MOT: Cultivating Instance Awareness from Point Seeds for Multi-Object Tracking" (Luo et al., 29 Jun 2026), the training signal for each frame $t$ and instance $i$ consists of a point $p_i^+ \in \mathbb{R}^2$ and an identity label across frames, with no ground-truth bounding box or mask used in training. The central problem is how to recover instance geometry and preserve identity under supervision that is precise in location but ambiguous in scale and boundary. The proposed PS-Track framework addresses this by organizing the learning problem across data, model, and loss levels, thereby making point-only supervision a feasible alternative to box-supervised MOT in datasets such as DanceTrack, SportsMOT, JRDB, and EmboTrack (Luo et al., 29 Jun 2026).

1. Problem formulation and motivation

Classical MOT assumes that every object instance in every frame is labeled with a tight box and an identity. PS-MOT replaces those dense geometric annotations with single point annotations per instance per frame. In the formulation under discussion, the point is treated as a topological center, and the absence of explicit geometric structure and scale constraints creates what the authors call a precision–ambiguity paradox: the supervision is precise in location but highly ambiguous in scale and geometry (Luo et al., 29 Jun 2026).

The motivation is simultaneously annotational, geometric, and representational. Bounding boxes are reported as requiring $7$–$10$ s per box, while points require $0.7$–$0.9$ s per point, and the reported experimental setting shows an approximately $64\%$ reduction in measured human annotation time. This cost differential becomes especially consequential for large-scale datasets and for settings in which box annotation is itself ambiguous, including severe perspective distortion, non-rigid bodies, dense crowds, and panoramic or embodied views such as JRDB and EmboTrack. In that setting, points function as topological anchors that are less sensitive to viewpoint distortion than axis-aligned boxes (Luo et al., 29 Jun 2026).

PS-MOT also implies a representational shift. Rather than treating supervision as spatial fitting to a box extent, the method shifts toward topological center-driven representation. A plausible implication is that the learning objective becomes less tied to axis-aligned geometry and more tied to trajectory-consistent instance localization. That implication is explicit in the proposed framework, which seeks to evolve sparse point seeds into temporally coherent instance representations instead of assuming that geometry is supplied a priori (Luo et al., 29 Jun 2026).

2. Position among neighboring paradigms

PS-MOT is presented as distinct from several adjacent lines of work. First, point-to-box methods for static detection, such as pseudo-box generation from points, estimate box extent independently per image. The stated limitation in a tracking context is that frame-wise scale estimation causes temporal jitter and identity instability. PS-MOT instead treats point-to-instance evolution as a trajectory-aware process with motion priors and temporal feedback (Luo et al., 29 Jun 2026).

Second, PS-MOT differs from weakly supervised, semi-supervised, or unsupervised MOT settings in which some box supervision, initialization boxes, or stronger image-level cues often remain available. The stated objective here is point-only supervision without any true box labels. Third, it differs from keypoint-based tracking approaches in the CenterTrack style: such models may treat objects as points at inference, but they are trained on full boxes. In PS-MOT, training supervision is points only, and boxes appear only as noisy pseudo-labels generated on the fly (Luo et al., 29 Jun 2026).

A recurrent misconception is to equate PS-MOT with earlier point-based tracking frameworks whose internal representation is point-centric. "Segment as Points for Efficient Online Multi-Object Tracking and Segmentation" (Xu et al., 2020) and "PointTrack++ for Effective Online Multi-Object Tracking and Segmentation" (Xu et al., 2020) are point-based in representation, not point-supervised in training. In those systems, dense instance masks and semantic labels are still used during training; the point sets arise after segmentation by converting instance masks into unordered 2D point clouds for embedding and association. This distinction matters: PS-MOT concerns the supervision signal itself, whereas PointTrack and PointTrack++ concern the representation used after dense mask supervision has already been provided (Xu et al., 2020).

3. Hierarchical PS-Track framework

PS-Track is the framework proposed to make PS-MOT operational. Its defining structure is a hierarchical coarse-to-fine pipeline spanning data, model, and loss levels, with the explicit objective of transitioning from points to instances (Luo et al., 29 Jun 2026).

Level	Mechanism	Function
Data level	Temporal-Feedback Prompting (TFP)	Evolves points into temporally consistent pseudo masks/boxes
Model level	Point-Excited Wavelet Attention (PEWA)	Excites high-frequency components near points to hallucinate boundaries
Loss level	Uncertainty-Guided Gaussian Learning (UGL)	Treats pseudo boxes as probabilistic supervision with reliability-dependent variance

At the data level, TFP takes raw point annotations and uses SAM-3, motion priors, and negative spatial cues to generate pseudo bounding boxes $B_{\text{pseudo}}$ and a joint quality score $S_{joint}$. At the model level, PEWA converts point annotations into Gaussian heatmaps, applies a single-level Haar Discrete Wavelet Transform to backbone features, and amplifies high-frequency responses near the annotated point. At the loss level, UGL models each pseudo box as a noisy sample from a Gaussian centered at an unknown true box, with variance derived from the reliability score $i$0 (Luo et al., 29 Jun 2026).

The inference regime is deliberately conventional. Only RGB frames are input; SAM and TFP are offline and used only for training label generation; PEWA is bypassed at test time; and the final system behaves as a standard MOT detector–tracker implemented on top of MOTIP and Deformable DETR. This separation is central to the framework’s claim that the supervision paradigm can change without requiring a specialized point-aware test-time interface (Luo et al., 29 Jun 2026).

4. Temporal-Feedback Prompting

Temporal-Feedback Prompting is the data-level mechanism that converts sparse point supervision into pseudo labels with improved separation and temporal stability. Its purpose is to address two specific failure modes induced by point-only supervision: identity merging in crowded scenes and semantic fragmentation under occlusion or partial visibility (Luo et al., 29 Jun 2026).

For spatial disambiguation, TFP augments the positive point $i$1 for target $i$2 with negative prompts drawn from neighboring active tracks: $i$3 SAM is then called with both the positive point and the set of negatives. The stated role of these negative spatial cues is to act as a semantic firewall: regions near $i$4 are encouraged, while regions around nearby track centers are explicitly excluded. This is intended to reduce the tendency of a single prompt to absorb multiple touching persons into one mask (Luo et al., 29 Jun 2026).

For temporal regularization, TFP maintains a Kalman filter per track, which produces a predicted motion prior box $i$5 for frame $i$6. Segmentation is then constrained by both prompts and the motion prior: $i$7 The output mask $i$8 is converted into a pseudo box $i$9. This design constrains segmentation to remain near the predicted trajectory and stabilizes size and position under incomplete visibility (Luo et al., 29 Jun 2026).

Not all pseudo labels are treated equally. TFP assigns a joint quality score

$p_i^+ \in \mathbb{R}^2$0

where $p_i^+ \in \mathbb{R}^2$1 is SAM’s internal confidence and $p_i^+ \in \mathbb{R}^2$2 measures compatibility between the segmentation output and the motion prior. High SAM confidence and high overlap with the motion-predicted box produce high $p_i^+ \in \mathbb{R}^2$3, while visually or temporally inconsistent pseudo labels are down-weighted. This score is then propagated to the loss level, where it controls supervision strength (Luo et al., 29 Jun 2026).

5. Boundary induction and uncertainty calibration

PEWA and UGL together define the model-level and loss-level response to the ambiguity of point supervision. The first addresses the absence of boundary information in the feature space; the second addresses the noise of the pseudo labels used for box regression (Luo et al., 29 Jun 2026).

PEWA is inserted after the ResNet backbone and before the Deformable DETR encoder during training. Given a feature map $p_i^+ \in \mathbb{R}^2$4, it applies a single-level Haar Discrete Wavelet Transform: $p_i^+ \in \mathbb{R}^2$5 where $p_i^+ \in \mathbb{R}^2$6 is the low-frequency approximation and $p_i^+ \in \mathbb{R}^2$7 are the high-frequency components corresponding to LH, HL, and HH. Since boundaries are encoded primarily in high-frequency components, PEWA constructs a Gaussian heatmap centered at the annotated point, downsamples it to wavelet resolution, and passes it through a modulation network: $p_i^+ \in \mathbb{R}^2$8 The high-frequency bands are then modulated as

$p_i^+ \in \mathbb{R}^2$9

and reconstructed via inverse DWT: $7$0 The stated interpretation is that high-frequency responses near the Gaussian spot are amplified while background edges farther away are suppressed, so that the point acts as a top-down spotlight for boundary-sensitive feature refinement (Luo et al., 29 Jun 2026).

UGL formalizes pseudo-label noise as aleatoric, heteroscedastic uncertainty. Each pseudo box is modeled as

$7$1

with variance derived from the TFP quality score: $7$2 Thus high-quality pseudo labels induce small $7$3 and strong supervision, while unreliable pseudo labels induce large $7$4 and weak supervision. The regression loss is the Gaussian negative log-likelihood

$7$5

The complete training objective is

$7$6

Within the proposed framework, this is not merely heuristic weighting: the data-level estimate of pseudo-label reliability is used directly to calibrate the regression term probabilistically (Luo et al., 29 Jun 2026).

6. Training regime, base architecture, and association formulation

PS-Track is built on MOTIP, using a Deformable DETR backbone with ResNet-50 pre-trained on ImageNet, a 6-layer transformer encoder, a 6-layer decoder, and 300 object queries per frame. The architectural additions are training-time only: PEWA is inserted after the backbone during training, and there are no architectural changes at inference (Luo et al., 29 Jun 2026).

For controlled experiments, point annotations are synthesized as centers of ground-truth boxes without extra manual clean-up. The authors explicitly retain difficult cases in which the box center lies outside the visible object or on a neighboring instance, with the stated purpose of simulating imperfect human clicks. TFP then uses SAM-v3 to generate a mask $7$7, converts that mask into a pseudo box $7$8, and computes $7$9. Only these pseudo boxes and scores are used for supervision; no true boxes are used during training (Luo et al., 29 Jun 2026).

The reported training details are specific: AdamW optimizer; base learning rate $10$0; backbone learning rate $10$1; video clips of 30 frames; typically 10 training epochs with learning-rate decay after epochs 6 and 9; ablations often run for 2 epochs; random flip, crop, multiscale resize with $10$2–$10$3 minimum side and up to $10$4 maximum, and color jitter. The reported hardware is a single RTX 5090 GPU with approximately 8 hours per epoch on DanceTrack (Luo et al., 29 Jun 2026).

The tracking formulation itself remains inherited from MOTIP. Detection is handled by a Deformable DETR-like transformer that predicts boxes and class scores for each query. Identity prediction is formulated as ID classification, yielding an identity loss $10$5 that encourages stable identity embeddings. Association is therefore largely end-to-end inside the transformer, while the Kalman filter is used only in TFP for label generation rather than for test-time association. At inference, the model uses frames only and follows the standard MOT setting with no points, with test-time association based on learned ID embeddings and MOTIP’s internal association mechanism (Luo et al., 29 Jun 2026).

7. Empirical results, significance, and limitations

The empirical evidence is reported on DanceTrack, SportsMOT, JRDB, and EmboTrack under point supervision with synthetic center points. On DanceTrack test, fully supervised MOTIP reaches HOTA $10$6 and IDF1 $10$7, while PS-Track reaches HOTA $10$8 and IDF1 $10$9. The same result exceeds classical box-supervised baselines CenterTrack and FairMOT by $0.7$0 and $0.7$1 HOTA, respectively. On SportsMOT test, fully supervised MOTIP reaches HOTA $0.7$2 and PS-Track reaches HOTA $0.7$3. On JRDB test, PS-Track reaches HOTA $0.7$4, exceeding TrackFormer at $0.7$5 and DiffMOT at $0.7$6. On EmboTrack, QuadTrack results are ByteTrack $0.7$7 HOTA, OC-SORT $0.7$8, OmniTrack++ with domain adaptation $0.7$9, and PS-Track $0.9$0 HOTA with IDF1 $0.9$1; on BipTrack, strong fully supervised methods such as OmniTrack++ with domain adaptation reach HOTA $0.9$2, while PS-Track reaches $0.9$3 (Luo et al., 29 Jun 2026).

Ablations show where the gains arise. On DanceTrack validation, the baseline without TFP, PEWA, or UGL reaches HOTA $0.9$4; adding TFP yields $0.9$5; adding UGL yields $0.9$6; and the full PS-Track reaches $0.9$7. Point-noise robustness remains strong under synthetic Gaussian noise on point coordinates, with HOTA approximately $0.9$8 at $0.9$9 px, $64\%$0 at $64\%$1 px, and $64\%$2 at $64\%$3 px. HOTA remains stable around $64\%$4–$64\%$5 for $64\%$6, peaking at $64\%$7, and rises steadily from epoch 2 at $64\%$8 to epoch 10 at $64\%$9, with no reported late-stage overfitting despite noisy labels (Luo et al., 29 Jun 2026).

The framework is also reported as general across tracker families. On DanceTrack validation, BYTE under box supervision reaches HOTA $B_{\text{pseudo}}$0, while BYTE + PS-Track under point supervision reaches $B_{\text{pseudo}}$1; AR-MOT under box supervision reaches $B_{\text{pseudo}}$2, while AR-MOT + PS-Track reaches $B_{\text{pseudo}}$3; MOTIP under box supervision reaches $B_{\text{pseudo}}$4, while MOTIP + PS-Track reaches $B_{\text{pseudo}}$5. By contrast, a naive decoupled pipeline based on Point2RBox-v3 with generic tracking-by-detection associators under point supervision yields HOTA approximately $B_{\text{pseudo}}$6 and MOTA below $B_{\text{pseudo}}$7, whereas a YOLOX detector trained with PS-Track and BYTE association reaches HOTA $B_{\text{pseudo}}$8 and IDF1 $B_{\text{pseudo}}$9. This suggests that point supervision is not adequately handled by independent point-to-box estimation followed by off-the-shelf association; the paper’s claim is that a unified pipeline is required to learn stable instance scale and identity (Luo et al., 29 Jun 2026).

The advantages identified in the work are annotation efficiency, robustness in geometric distortions, minimal changes to inference-time architectures, explicit handling of label noise, and boundary-aware features induced from points. The limitations explicitly discussed are extreme occlusions and inter-person entanglement, severe motion blur, the gap between synthetic box-center points and real human clicks, the assumption of point labels in every frame, and the offline cost of pseudo-label generation with SAM. The proposed extensions include 3D MOT, point-supervised segmentation or video object segmentation, and temporally sparse PS-MOT with occasional point annotations and unsupervised motion propagation. Taken together, these results support the paper’s broader claim that instance awareness and identity-consistent trajectories can be cultivated from sparse, ambiguous point seeds when temporal feedback, frequency-domain boundary induction, and uncertainty-aware supervision are coupled in a single training framework (Luo et al., 29 Jun 2026).