DFR-FastMOT: Real-Time Object Tracking

• The paper introduces a detection failure–resistant tracking framework that uses a dual-matrix, algebraic sensor fusion approach to combine 2D and 3D detections.
• It achieves high real-time performance by integrating minimal-state Kalman filtering with a long-term memory mechanism for robust occlusion recovery.
• Experimental evaluations on KITTI demonstrate superior accuracy and speed, with significant improvements in both MOTA and HOTA under challenging conditions.

DFR-FastMOT is a detection failure–resistant multi-object tracking (MOT) framework designed for real-time performance and robust occlusion handling in autonomous vehicle environments. It employs a lightweight, algebraic sensor fusion approach to object association, leveraging both camera and LiDAR detections, and introduces a long-term memory mechanism that allows recovery from extended occlusions without compromising computational efficiency. By unifying measurements from heterogeneous sensors and maintaining track persistence via minimal-state Kalman filtering, DFR-FastMOT achieves superior accuracy and runtime efficiency compared to existing learning-based and non-learning baselines (Nagy et al., 2023).

1. System Architecture and Data Flow

DFR-FastMOT operates on synchronized inputs from a monocular or stereo camera (2D bounding boxes, $D_t^{2d}$) and a LiDAR sensor (3D clusters or bounding boxes, $D_t^{3d}$), together with calibration parameters for cross-modal projection. The tracker supports two operational modes: mono-detector, which projects detections from one sensor into the other's frame to fill in missing measurements, and multi-detector, which performs initial fusion and deduplication across both sensor detection outputs.

For each time step $t$, the core pipeline proceeds as follows:

1. Detection Matching and Fusion: If both sensors are active, initial matching amalgamates 2D and 3D outputs into a unified set $S_t$, ensuring no object is double-counted.
2. Association and Sensor Fusion: $S_t$ is algebraically associated to the set of live tracks in memory.
3. State Update: All tracks undergo a Kalman-filter update/prediction cycle.
4. Track and Memory Management: Tracks are introduced, persisted, or pruned based on visibility and occlusion counters, enabling long-term maintenance of object identities in challenging scenarios.

2. Algebraic Association and Sensor Fusion Mechanisms

Association relies on constructing two $m\times n$ matrices—$M_c$ for camera detections (2D IoU) and $M_l$ for LiDAR detections (3D centroid distances)—where $m$ is the number of detections and $n$ is the number of active tracks. The entries of $D_t^{3d}$0 are given by

$D_t^{3d}$1

while those of $D_t^{3d}$2 are based on thresholded inverse distances,

$D_t^{3d}$3

where $D_t^{3d}$4 is the Euclidean centroid distance and $D_t^{3d}$5 are sensor-specific thresholds.

These matrices are fused into a single association matrix,

$D_t^{3d}$6

where $D_t^{3d}$7 are modality weights, subject to user-tuning. Final detection-to-track assignment is performed via a greedy algorithm: the entry with the largest $D_t^{3d}$8 above threshold $D_t^{3d}$9 is selected, and the corresponding detection and track are removed from further consideration, iterating until no eligible matches remain. This bypasses combinatorial solvers, yielding high performance even for $t$0.

3. Long-Term Memory and Occlusion Recovery

For robust handling of both brief and extended occlusions, DFR-FastMOT incorporates per-track counters $t$1 and $t$2, tracking the number of consecutive frames a given object is unobserved in each modality. A track remains "alive" as long as

$t$3

where $t$4 is the maximum allowable miss count (dozens of frames, configurable). If a track receives no detection assignment, its state is still propagated forward using a constant-acceleration Kalman filter, operating on minimal 2D and 3D bounding-box corner representations (e.g., top-left and bottom-right in 2D). This enables tracks to "drift" but remain recoverable across occlusion and partial observability periods. Disabling this memory structure leads to a 4–6% MOTA reduction under moderate/high distortion, highlighting its significance for occlusion recovery.

4. Real-Time Implementation and Computational Efficiency

DFR-FastMOT models each object with an 8- or 12-dimensional state (positions, velocities, accelerations for the key corners in 2D and 3D). The algebraic association and Kalman updates are optimized for minimal per-object computational cost.

• Each frame's data association (matrix computation and assignment) and state update for typically up to several dozen tracks execute within 200~µs on a single CPU core.
• On the full KITTI MOT dataset (7763 frames), DFR-FastMOT completes tracking in 1.48 seconds (≈ 5250 FPS), outperforming the EagerMOT and DeepFusionMOT baselines by factors of 7 and over 25, respectively.

Tracker	KITTI Runtime (s, 7763 frames)	Relative Speed
DFR-FastMOT	1.48	1× (fastest)
EagerMOT	11.47	7.7× slower
DeepFusionMOT	37.38	25.3× slower

5. Experimental Protocols for Robustness Evaluation

The main evaluation is performed on KITTI MOT (21 sequences, ≈8000 frames), simulating various detection reliability regimes:

• High distortion: 2D YOLOv3, projected LiDAR (poor detection quality)
• Medium distortion: RCC (moderate 2D), projected 3D
• High quality: TrackRCNN/RCC (2D), PointRCNN/PointGNN (3D)

State-of-the-art non-learning methods (EagerMOT: IoU+KF; DeepFusionMOT: fusion with deep association) are rerun under the same detection conditions for controlled benchmarking.

6. Quantitative Results and Performance Analysis

DFR-FastMOT substantially outperforms both non-learning and learning-based competitors, particularly under conditions of detector distortion and object occlusion.

Detector Quality	Tracker	HOTA (%)	MOTA (%)
Poor (YOLOv3)	DFR-FastMOT	39.2	44.5
	EagerMOT	36.5	41.6
	DeepFusionMOT	30.0	31.8
Medium (RCC)	DFR-FastMOT	81.9	91.0
	EagerMOT	70.8	82.2
	DeepFusionMOT	42.6	40.2
High Quality	DFR-FastMOT	82.8	90.7
	Best Baseline	—	85–88

On the official KITTI test server, DFR-FastMOT achieves:

• 2D mode: HOTA 83.4%, MOTA 93.06%, AMOTA 90.79%
• 2D+3D mode: HOTA 84.28%, MOTA 91.96%, AMOTA 85.36%, compared to leading learning-based MOTA results of 89.44%, and non-learning methods in the 74–84% range.

7. Factors Contributing to Occlusion and Distortion Robustness

Key properties underlying DFR-FastMOT's resilience include:

• Algebraic Sensor Association: Weighted matrix fusion of 2D IoU and normalized, inverted 3D centroid distances (with tunable $t$5, $t$6) ensures robustness to single-sensor degradation.
• Minimal-State Kalman Filtering: By updating only key corners, the tracker efficiently maintains plausible object state over extended detection loss.
• Long-Term Invisibility Counters: Extended persistence ($t$7 large) facilitates re-identification across major occlusions and frame drops.
• Lightweight Architecture: The entire pipeline avoids expensive combinatorial association or deep network inference, yielding exceptional real-time throughput on CPUs.

Ablation studies show that reducing the occlusion tolerance window ($t$8) degrades MOTA by 4–6% under distortion, corroborating the importance of memory for occlusion handling.

In summary, DFR-FastMOT introduces a dual-matrix, weight-blended sensor association paradigm with long-term, minimal-state memory and achieves state-of-the-art tracking metrics and frame rates within a real-time, CPU-based operational envelope (Nagy et al., 2023).