Feature-Assisted IoU Tracker

• The literature shows feature-assisted IoU tracking integrates auxiliary cues to overcome limitations caused by occlusion, rapid motion, and contour deformations.
• It combines techniques such as bounding box prediction, appearance embeddings, optical flow warping, and pose-guided features to boost association reliability.
• Reported improvements in metrics like MOTA, IDF1, and HOTA underscore the practical impact of these methods in real-time and edge-based tracking applications.

In the literature summarized here, a feature-assisted IoU tracker denotes a family of tracking methods in which Intersection over Union (IoU) remains a primary signal for localization, mask overlap, or track-to-detection association, while auxiliary cues are introduced to compensate for IoU’s failure modes under occlusion, fast motion, contour deformation, small objects, or long temporal gaps. The auxiliary cues include tracked object bounding boxes and overlap prediction, appearance embeddings, clothing color and style, target direction, optical-flow-warped masks, pose-guided part features, continual feature adaptation, and memory-assisted motion representations. Across multi-object tracking, visual object tracking, video panoptic segmentation, and application-specific edge systems, the common design pattern is not to discard IoU, but to make IoU-conditioned decisions more reliable (Yang et al., 2021, Ye et al., 2022, Hashempoor et al., 2024, Wang et al., 16 Sep 2025).

1. Conceptual scope and problem setting

Feature-assisted IoU tracking emerged as a response to a recurrent limitation in overlap-based tracking: IoU is highly informative when frame-to-frame localization is accurate, but its reliability degrades when detections are shifted by occlusion, motion, camera movement, small-object scale effects, or segmentation deformation. One line of work therefore augments purely spatial tracking with richer geometric predictions. In the modification of CenterTrack described in "Multi-object Tracking with Tracked Object Bounding Box Association" (Yang et al., 2021), the original center-displacement association is replaced by tracked object bounding box prediction and an IoU distance cost matrix, specifically to reduce identity switches caused by an inferior association method. By contrast, "Hierarchical IoU Tracking based on Interval" (Du et al., 2024) deliberately uses only IoU for association and discards heavy appearance models, tricky auxiliary cues, and learning-based association modules, thereby establishing a strong pure-IoU baseline and clarifying what extra features must justify.

A second line of work uses IoU as a training objective rather than only an association score. "IG-TRACK: IOU Guided Siamese Networks for visual object tracking" (Dasari et al., 2020) adds an IoU-guided loss to Siamese region-proposal tracking so that the network directly maximizes overlap with ground truth during training. "Accurate Bounding-box Regression with Distance-IoU Loss for Visual Tracking" (Yuan et al., 2020) replaces standard IoU loss with DIoU loss to preserve overlap optimization while also minimizing center distance between boxes. "COMET: Context-Aware IoU-Guided Network for Small Object Tracking" (Marvasti-Zadeh et al., 2020) combines IoU guidance with center location error regression, attention, and context-aware proposal generation for aerial small-object tracking.

A third line of work makes IoU one component in a broader association metric. "Hybrid Tracker with Pixel and Instance for Video Panoptic Segmentation" (Ye et al., 2022) fuses an instance-feature association matrix with a pixel tracker that computes a Dice-coefficient-based IoU matrix after optical-flow warping. "OccluTrack: Rethinking Awareness of Occlusion for Enhancing Multiple Pedestrian Tracking" (Gao et al., 2023), "FeatureSORT: Essential Features for Effective Tracking" (Hashempoor et al., 2024), and "FACT: Feature Adaptive Continual-learning Tracker for Multiple Object Tracking" (Song et al., 2024) all integrate IoU with appearance or adaptive feature learning, but they differ in the granularity of feature design and in how association thresholds are adapted. In an application-specific form, "Real-Time Detection and Tracking of Foreign Object Intrusions in Power Systems via Feature-Based Edge Intelligence" (Wang et al., 16 Sep 2025) explicitly names a feature-assisted IoU tracker that combines spatial overlap with ConvNeXt embeddings for edge deployment.

2. Association formulations built around IoU

A canonical formulation appears in the CenterTrack extension of (Yang et al., 2021). For each detected object at position $\hat{p}$, the network predicts the bounding box it would have had in the previous frame, and association is performed with the IoU distance

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$

where $\hat{b}^{(t-1)}$ is the detected bounding box at the previous frame and $\hat{tb}^{(t)}$ is the tracked bounding box predicted from the current frame. The network also predicts the expected overlap between same-object bounding boxes in adjacent frames; if the actual predicted IoU falls below the predicted adjacent overlap, the association cost is set to infinity, after which greedy matching is applied. The paper reports that the IoU-only variant gives the best results among the explored association variants.

Hybrid formulations retain IoU but do not rely on it exclusively. In HybridTracker, the pixel tracker warps instance masks by optical flow and computes a Dice coefficient between warped and current masks, while the instance tracker computes embedding similarities and a differentiable matching layer; the two association matrices are fused into a final matching matrix, followed by row-wise argmax, mutual check, and thresholding (Ye et al., 2022). In FeatureSORT, IoU is used as a pre-filter rather than as the sole distance: a detection-track pair must satisfy $\text{IoU} > \text{IoU}_{\min}$ and a direction constraint before a weighted sum of motion, ReID, color, and style distances is accepted,

$$\mathbf{D}_{combined} = \begin{cases} \lambda_{motion} \mathbf{D}_{motion} + \lambda_{edge} \mathbf{D}_{edge} + \lambda_{color} \mathbf{D}_{color} + \lambda_{style} \mathbf{D}_{style}, & \text{if } \text{IoU} > \text{IoU}_{min} \text{ and } \mathbf{D}_{direction} < \text{Dir}_{max} \ d_{max} + \epsilon, & \text{otherwise}. \end{cases}$$

The reported weights are $0.1$ for motion, $0.4$ for edge, $0.25$ for color, and $0.25$ for style (Hashempoor et al., 2024).

Occlusion-aware association modifies the role of IoU more explicitly. In OccluTrack, the IoU threshold is made track-dependent:

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$0

and the final distance is

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$1

Matching is then solved by the Hungarian algorithm (Gao et al., 2023). FACT replaces direct pairwise scoring in its first stage with affinity estimation from its online continual learner and converts affinity to distance by $$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$2, while preserving a second-stage fallback based on cosine distance and IoU distance for ambiguous or new cases (Song et al., 2024).

A further generalization appears in MeMoSORT, whose Motion-adaptive IoU constructs a dynamic association metric by multiplying Expansion IoU and Height IoU. The expansion term enlarges both boxes to tolerate localization error, while the height term measures vertical overlap and is weighted adaptively according to motion statistics. The paper states that this is intended to reduce the influence of detection errors and association failures while remaining lightweight (Wang et al., 13 Aug 2025).

3. Types of assisting features

The most common assistance mechanism is appearance embedding. The power-system FOI framework of (Wang et al., 16 Sep 2025) extracts 1024-dimensional ConvNeXt embeddings trained with triplet loss and uses cosine similarity when IoU-based matching is insufficient or ambiguous. FeatureSORT replaces the original DeepSORT CNN with Enhanced BoT ReID embeddings and smooths them by an exponential moving average,

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$3

with $$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$4 (Hashempoor et al., 2024). FACT takes detector/ReID features and applies an embedding transformation layer followed by an online analytic classifier that uses all past tracking information, thereby turning appearance features into continually updated affinity scores rather than static descriptors (Song et al., 2024).

Several trackers enrich appearance with semantically structured cues. OccluTrack extracts both local body-part features and global features using a pose-guided Re-ID module, and fuses them according to the number of confidently detected body parts:

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$5

This design makes local features dominant when pose is reliable and global features dominant otherwise (Gao et al., 2023). FeatureSORT adds clothing color, clothing style, and target direction. Color and style are predicted by separate YOLOX heads and matched through minimum cross-entropy loss over feature banks, while direction is estimated over 72 bins and used as a gating condition in association (Hashempoor et al., 2024).

Other systems use non-appearance features that still function as assistance to IoU. HybridTracker’s pixel tracker computes an overlap-like score from optical-flow-warped masks via the Dice coefficient,

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$6

which the paper describes as functionally equivalent to a soft version of IoU (Ye et al., 2022). CenterTrack’s extension adds tracked bounding box prediction and overlap prediction, so that association uses object extent rather than only center displacement (Yang et al., 2021). COMET uses multi-scale feature learning, channel and spatial attention, precise RoI pooling, and a modulation mechanism that injects reference target and context information into the test branch, with a multitask loss

$$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$7

to jointly maximize overlap and reduce normalized center error (Marvasti-Zadeh et al., 2020).

Motion itself can be treated as a learned feature. MeMoSORT’s Memory-assisted Kalman Filter maintains a memory vector with an LSTM and uses gated modules for state prediction and state update, thereby compensating for the mismatch between assumed and actual object motion. Its assistance is not appearance-based, but it still augments IoU-centered association with historical motion representation and height similarity (Wang et al., 13 Aug 2025).

4. Failure modes and the mechanisms designed to address them

Occlusion is the dominant failure mode in this literature. OccluTrack attributes errors under partial occlusion to abnormal detections that corrupt motion estimation, appearance features, and association fairness; it introduces abnormal motion suppression inside the Kalman Filter, a pose-guided Re-ID module, and occlusion-aware association to mitigate these effects (Gao et al., 2023). HybridTracker addresses occlusion and contour deformation at inference time with mutual check and temporal consistency constraints, including a cycle-consistency relation across $$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$8, $$\text{IoU distance} = 1 - IoU(\hat{b}^{(t-1)},\, \hat{tb}^{(t)}),$$9, and $\hat{b}^{(t-1)}$0 that can preserve identity through a temporary disappearance (Ye et al., 2022). COMET attacks similar problems in aerial small-object tracking through offline reference proposal generation, context-aware learning, and proposals that encode not only the target as a whole but also its parts and surrounding context (Marvasti-Zadeh et al., 2020).

Fast motion and camera motion expose a different weakness: frame-to-frame overlap may be too small for plain IoU to remain discriminative. HIT identifies three inconsistency issues—target size, camera movement, and hierarchical cues—and introduces Consistent-IoU, Consistent-camera, and Consistent-motion to address them (Du et al., 2024). In MeMoSORT, Motion-adaptive IoU expands the matching space for rapid movement and discounts height similarity when height is changing rapidly, while MeKF uses memory to preserve plausible motion prediction after abrupt maneuvers or occlusion (Wang et al., 13 Aug 2025). The FOI tracker provides case studies in which a crane vehicle exhibits IoU as low as $\hat{b}^{(t-1)}$1 while cosine similarity remains at least $\hat{b}^{(t-1)}$2, and a dust-proof net yields very low or zero IoU in some frames while cosine similarity remains approximately $\hat{b}^{(t-1)}$3 to $\hat{b}^{(t-1)}$4; in both cases the paper attributes identity continuity to feature-assisted disambiguation (Wang et al., 16 Sep 2025).

Small objects and contour variability create scale-sensitive overlap errors. COMET is explicitly designed for small targets from aerial videos of medium to high altitudes, using multi-scale aggregation and fusion, attention, and center-location supervision (Marvasti-Zadeh et al., 2020). HIT’s Consistent-IoU enlarges small boxes before computing IoU because small boxes are penalized more heavily for the same localization error (Du et al., 2024). HybridTracker’s use of optical flow and soft overlap is likewise motivated by small objects and strong instance contour deformations in video panoptic segmentation (Ye et al., 2022).

These mechanisms indicate a recurrent structural point. Pure IoU is not uniformly abandoned; instead, it is stabilized by learned or hand-designed compensators targeted at specific sources of inconsistency. This suggests that feature assistance is best understood as a method for conditioning when and how IoU should be trusted, rather than as a complete replacement for overlap-based reasoning.

5. Representative systems and reported results

System	Setting	Reported result
CenterTrack with tracked object bounding box association	MOT17 test	IDF1 from 64.7% to 66.2%, IDs from 3039 to 2352, MOTA from 67.8% to 68.1% (Yang et al., 2021)
HIT	MOT17, KITTI, DanceTrack, VisDrone	MOT17: HOTA 63.5, IDF1 77.4, MOTA 79.3; KITTI: HOTA 77.7, IDF1 78.3, MOTA 90.9; DanceTrack: HOTA 56.6, MOTA 92.1, IDF1 55.4; VisDrone: highest MOTA 53.5 and IDF1 65.6 (Du et al., 2024)
IG-TRACK	VOT2018, GOT-10k	VOT2018 EAO 0.290 to 0.327 and robustness 0.347 to 0.309; GOT-10k AO 0.453 to 0.459 and SR$\hat{b}^{(t-1)}$5 0.195 to 0.220 (Dasari et al., 2020)
COMET	UAVDT, VisDrone-2019, Small-90	Outperforms ATOM by an average margin of 6.2% in precision and 7% in success (Marvasti-Zadeh et al., 2020)
OccluTrack	MOT17, MOT20	MOT17: IDF1 81.1%, ID Switches 1038; MOT20: IDF1 78.6%, ID Switches 1124 (Gao et al., 2023)
Feature-assisted IoU tracker for FOI	Surveillance and drone video; Jetson Orin Nano	[email protected] from 0.52 to 0.83, F1-score from 0.51 to 0.85; 14–24 FPS, 42–73 ms, ~2.2 GB, 9–15 W (Wang et al., 16 Sep 2025)

Additional results reinforce that IoU assistance is not confined to one tracking regime. HybridTracker reports VPQ values of 58.4 on Cityscapes-VPS validation and 52.1 on VIPER, with temporal consistency further improving these to 58.5 and 52.4; the paper states that fusion consistently outperforms either tracker alone (Ye et al., 2022). DIoUTrack reports that DIoU loss brings approximately 11% improvement in success score on LaSOT over IoU loss, and the paper also reports real-time speed above 40 FPS on RTX 2080Ti (Yuan et al., 2020). FACT reports gains of +0.7 to +1.2 HOTA and +0.8 to +2.0 IDF1 across integrated trackers, with more obvious advantages on MOT20, and states that it sustains more than 7 FPS for large models and over 20 FPS for lighter trackers on a 2080Ti GPU (Song et al., 2024).

The quantitative pattern is consistent across domains. When IoU is enhanced by bounding-box prediction, structured appearance cues, temporal consistency, online continual learning, or motion-adaptive filtering, the metrics that improve most often are IDF1, ID switches, HOTA, VPQ, and success/overlap measures. This pattern is explicit in the CenterTrack modification, OccluTrack, COMET, IG-TRACK, and the FOI framework (Yang et al., 2021, Gao et al., 2023, Marvasti-Zadeh et al., 2020, Dasari et al., 2020, Wang et al., 16 Sep 2025).

6. Methodological debates, limitations, and likely directions

A central debate concerns how much of tracking accuracy truly requires appearance features. HIT argues that a unified hierarchical tracking framework using only IoU, tracklet intervals as priors, and consistency corrections can achieve promising performance on MOT17, KITTI, DanceTrack, and VisDrone, and can be seamlessly integrated with motion-based, appearance-based, or learning-based trackers (Du et al., 2024). This directly contrasts with methods such as FeatureSORT, OccluTrack, FACT, and the FOI tracker, all of which treat appearance information as indispensable for longer occlusions or ambiguous associations (Hashempoor et al., 2024, Gao et al., 2023, Song et al., 2024, Wang et al., 16 Sep 2025). The literature therefore does not support a single universal prescription; rather, it delineates conditions under which carefully normalized IoU is sufficient and conditions under which auxiliary descriptors materially improve identity preservation.

Another recurring issue is computational budget. DIoUTrack emphasizes a real-time tracker that combines DIoU-based estimation with a Conjugate-Gradient-based online classifier (Yuan et al., 2020). COMET remains real-time capable at 24 FPS on a V100 GPU despite its context, attention, and multitask refinement (Marvasti-Zadeh et al., 2020). FACT is designed to use all past tracking information without explicit storage of all history through recursive analytic updates (Song et al., 2024). The FOI system pushes this concern to deployment, using mixed-precision inference, Triton Inference Server, gRPC, and Redis to keep all computation on-device on NVIDIA Jetson hardware (Wang et al., 16 Sep 2025). A plausible implication is that feature assistance is increasingly evaluated not only by association quality but by whether it can be inserted into real-time and edge pipelines without prohibitive overhead.

Limitations also recur. The FOI tracker states that performance can still degrade under severe mis-segmentation, extreme low-visibility, or highly confusable classes, and that temporary identity switches may occur with overlapping, rapid objects under heavy occlusion (Wang et al., 16 Sep 2025). FeatureSORT reports that style features can improve MOTA and HOTA while also producing more IDs, indicating that over-distinct features can fragment trajectories (Hashempoor et al., 2024). MeMoSORT begins from the premise that conventional tracking-by-detection methods are fundamentally limited by their reliance on Kalman filter and rigid IoU-based association because the motion model often mismatches real-world object dynamics and rigid association struggles under occlusions (Wang et al., 13 Aug 2025). These observations suggest that future feature-assisted IoU trackers will likely continue to move toward adaptive thresholds, motion-conditioned overlap metrics, hierarchical or continual memory, and modular feature banks rather than static, one-shot IoU matching.

Taken together, the literature portrays feature-assisted IoU tracking as a design space rather than a single algorithmic template. IoU remains the organizing primitive because it ties association directly to spatial or mask overlap. The assisting features determine which ambiguities can be resolved: precise box geometry in CenterTrack-style association, optical-flow alignment in HybridTracker, discriminative embeddings in application-specific FOI monitoring, pose-aware part features in OccluTrack, structured color/style/direction cues in FeatureSORT, whole-history adaptation in FACT, and motion memory plus adaptive overlap in MeMoSORT (Yang et al., 2021, Ye et al., 2022, Wang et al., 16 Sep 2025, Gao et al., 2023, Hashempoor et al., 2024, Song et al., 2024, Wang et al., 13 Aug 2025).