Real-Time Object Detection: Speed & Safety

• Real-Time Object Detection is the process of rapidly identifying and localizing objects in images or videos, balancing accuracy with computational speed.
• It incorporates methodologies that merge conventional metrics like mAP with latency-aware scores to ensure timely and reliable detection in diverse settings.
• System-level strategies such as hardware acceleration, pipeline redesign, and edge-cloud partitioning are essential for robust, real-world deployment.

Real-time object detection (OD) is the task of identifying and localizing object instances in images or video quickly enough for live or near-live decision making, but in safety-critical systems its meaning is stricter than high detector throughput alone. In autonomous driving, the operational quantity may be the delay from an object’s physical appearance until its first detection is reported, and the broader deployment problem couples accuracy, latency, throughput, computational resources, and often multi-camera embedded execution rather than isolated benchmark inference (Jang et al., 2020, Sali et al., 4 Sep 2025).

1. Definition, timing semantics, and evaluation criteria

Object detection is commonly formulated as the joint problem of categorization and localization, typically by predicting class labels and bounding boxes for object instances in digital images or videos. Survey treatments in the literature distinguish throughput, usually reported in frames per second, from latency, understood as the elapsed time from frame acquisition until all objects in that frame are detected. In autonomous vehicles, this distinction is amplified by multi-camera operation, limited onboard compute, and the need for low enough reaction time under safety constraints (Sali et al., 4 Sep 2025, Bose et al., 17 Feb 2026).

A central correction to conventional practice is that real-time OD cannot be reduced to neural-network inference speed. R-TOD defines end-to-end delay as the time “from the physical appearance of an object … until its first detection is reported,” and shows that camera buffering, software pipelining, synchronization, and CPU/GPU contention can dominate the control-relevant delay even when the detector’s cycle time appears respectable (Jang et al., 2020). This directly challenges the widespread but incomplete equation of “real time” with detector-side FPS.

A second critique concerns evaluation by accuracy alone. In the autonomous-driving setting, “Real-time Traffic Object Detection for Autonomous Driving” argues that mean average precision is inadequate as a sole KPI because it is independent of inference time, and introduces the latency-aware score

$$\begin{aligned} RTOP_T(p, FPS) = \; & p \times w \ w = \; & b^{\phi - 1} \ \phi = \; & \min(\frac{FPS}{T}, 1), \end{aligned}$$

with $p=mAP$, $T=30$, and $b=2$, so that sub-real-time models are explicitly penalized (Khan et al., 2024). In edge offloading, an analogous timing notion is outage probability, defined as the event that OD results are not received within the inter-frame duration, which shifts the criterion from isolated inference time to full capture–transmit–infer–return completion (Kim et al., 2020).

2. Architectural families and their real-time trade-offs

The standard architectural split remains between two-stage and one-stage detectors. Survey literature describes R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, and Cascade R-CNN as proposal-based systems that typically favor accuracy, while YOLO, SSD, RetinaNet, CenterNet, and EfficientDet represent one-stage designs that are generally faster and therefore more suitable for real-time deployment (Bose et al., 17 Feb 2026). In the autonomous-vehicle review, one-stage detectors are repeatedly favored for practical deployment because they localize and classify in a single pass, whereas two-stage methods retain a latency burden that becomes more problematic under multi-camera embedded operation (Sali et al., 4 Sep 2025).

Within one-stage real-time OD, the YOLO and SSD lineages remain foundational. YOLO variants emphasize unified regression, dense convolutional parallelism, and increasingly refined multi-scale handling; SSD emphasizes direct prediction from multi-scale feature maps; RetinaNet addresses class imbalance via Focal Loss while preserving a single-stage pipeline (Bose et al., 17 Feb 2026). Challenge-oriented work on the Waymo real-time 2D detection track shows how these design pressures manifest in practice: a competitive system built from YOLOR-P6 and YOLOR-W6 used TensorRT, FP16 inference, class-specific NMS, and a crop-and-enlarge branch for small objects, reaching 75.00% L1 mAP, 69.72% L2 mAP, and 45.8 ms/frame on an NVIDIA Tesla V100 GPU under a strict runtime budget (Zhang et al., 2021).

Transformer-based OD changed the speed–accuracy landscape but did not eliminate real-time constraints. On surveillance imagery, OD-VIRAT baselines show that Deformable-DETR with a ResNet50 backbone and pretraining can achieve 75.0 mAP, 64.9 mAPS, and 143.2 FPS, while YOLOX remains the highest-throughput family at 240–281 FPS but materially weaker on tiny-object-heavy surveillance scenes (Ullah et al., 16 Jul 2025). This suggests that “transformer” and “non-real-time” are no longer interchangeable descriptions in deployment literature.

Recent real-time DETR systems make that shift explicit. DEIMv2 extends the DEIM training framework with DINOv3 features, a Spatial Tuning Adapter, simplified decoder components, and an upgraded Dense O2O strategy, yielding a detector family from server-scale to ultra-lightweight regimes. Representative points include DEIMv2-X at 57.8 AP, 50.3M parameters, 151.6 GFLOPs, and 13.75 ms; DEIMv2-S at 50.9 AP with 9.71M parameters and 5.78 ms; and DEIMv2-Pico at 38.5 AP with 1.51M parameters and 2.14 ms (Huang et al., 25 Sep 2025). In open-vocabulary detection, OmDet-Turbo shows that an Efficient Fusion Head, language cache, and TensorRT can lift a transformer OVD system to 100.2 FPS in the base variant while remaining competitive on COCO, LVIS, ODinW, and OVDEval (Zhao et al., 2024). For oriented object detection, RiO-DETR adapts DETR to rotated boxes with content-driven angle estimation, periodic angle refinement, and oriented dense supervision, reporting, for example, 78.4 AP$_{50}$ at 2.7 ms in the n variant and 81.7 AP$_{50}$ at 13.4 ms in the l variant on DOTA-1.0 (Hu et al., 10 Mar 2026).

3. Systems engineering, hardware, and deployment

The literature increasingly treats real-time OD as a systems problem rather than a detector-only problem. R-TOD formalizes best- and worst-case delay bounds and demonstrates that on Jetson AGX Xavier with Darknet YOLOv3, an object detection cycle time of about 163 ms coexisted with an end-to-end delay of about 1070 ms before architectural changes; on-demand capture, zero-slack pipeline, and contention-free pipeline then reduced the end-to-end delay of Darknet YOLOv3 from 1070 ms to 261 ms without changing the neural network architecture itself (Jang et al., 2020). This is a canonical example of latency reduction through pipeline redesign rather than model redesign.

Embedded co-scheduling introduces an additional layer of difficulty. A mobile-robot demonstration using a DB21M Duckiebot with Jetson Nano 2GB, PREEMPT_RT, ROS, and Docker showed YOLOv7-tiny at 160×160 running with 163.4 ms ACET in isolation, but response times degraded substantially when object detection, lane following, and a VAE-based OOD monitor were run concurrently; the paper explicitly states that object detection suffered the most degradation, likely because CPU utilization approached 100% and the 2 GB RAM became fully utilized (Yuhas et al., 2022). The result is less a new detector than a caution against reporting OD timing outside realistic co-scheduled workloads.

Edge/cloud partitioning offers another systems response. In the communication-aware EODF framework, an autonomous vehicle transmits the original image when channel quality is sufficient, but under poor channel conditions it extracts regions of interest using spectral residual visual saliency, masks the rest, and offloads the compressed image to an edge cloud running RFBNet. The paper evaluates outage probability under a 30 FPS criterion and reports that EODF can provide over 99% of results to the vehicle at 30 FPS while maintaining over 84% mAP and over 82% AP even at a 32% compression ratio (Kim et al., 2020). Edge YOLO adopts a different partition: cloud retraining and model updates are retained centrally, but latency-critical inference runs on Jetson-class onboard hardware, with the resulting detector reported at 26.6 FPS, 47.3% [email protected], and about 25.27 MB on COCO2017 (Liang et al., 2022).

Power-aware deployment adds yet another objective. AyE-Edge frames edge OD as a three-way conflict among accuracy, real-time latency, and power, and coordinates keyframe selection, CPU–GPU configuration, and DNN pruning with a multi-agent RL controller. On a OnePlus 8T smartphone, it reports a remarkable 96.7% reduction in power consumption compared to state-of-the-art competitors while maintaining strong real-time behavior and competitive accuracy on BDD100K-derived video streams (Wu et al., 2024). At the review level, the autonomous-vehicle hardware survey generalizes this point: throughput, latency, power, memory footprint, thermal limits, and the number of simultaneous cameras must all be considered together, and GPUs, ASICs, FPGAs, NPUs, and heterogeneous SoCs each imply different efficiency–flexibility trade-offs (Sali et al., 4 Sep 2025).

4. Challenging regimes: small objects, adverse conditions, open world, and non-RGB modalities

Small objects are a persistent bottleneck across real-time OD. The Waymo challenge solution explicitly attributes a large fraction of its gains to scale handling, including a crop of the central image band $$(0\times W,\; 0.3\times H,\; 1.0\times W,\; 0.8\times H)$$, resized by 1.5$\times$, with only boxes smaller than $96\times 96$ retained from that branch before NMS (Zhang et al., 2021). OD-VIRAT shows the same issue from a different angle: on elevated long-range surveillance imagery, the strongest separation among detector families appears in small-object AP, where Deformable-DETR and RetinaNet clearly outperform YOLOX, RTMDet, and DETR despite YOLOX’s much higher FPS (Ullah et al., 16 Jul 2025).

Occlusion, clutter, and adverse weather further complicate the real-time regime. “Real-Time Object Detection in Occluded Environment with Background Cluttering Effects Using Deep Learning” compares YOLO v3, YOLO v4, and SSD-MobileNet v2 on a custom two-class car/tank dataset built from about 60,000 images, and reports that after preprocessing and augmentation SSD-MobileNet v2 reached 86.6% mAP and 104 FPS, compared with YOLO v4 at 82.3% mAP and 63 FPS and YOLO v3 at 78% mAP and 58 FPS (Aamir et al., 2024). Under real snow, CF-YOLO replaces the YOLOv5s neck with a Cross Fusion block and evaluates on the RSOD dataset, where CF-YOLO $(K=1)$ preserves real-time behavior at 49 FPS on a Tesla V100 while improving robustness to snowy scenes relative to several baselines (Ding et al., 2022).

Open-world safety introduces a different failure mode: unknown objects may be ignored entirely by closed-set detectors. UNCOVER addresses this by adding an OOD class and an occupancy prediction branch to YOLOX. Its occupancy target is

$p=mAP$0

and inference recovers low-score detections if $p=mAP$1 even when the standard score is below $p=mAP$2 (Schmarje et al., 2024). On Fishyscapes Lost & Found, the Cityscapes-trained UNCOVER variant reports 58.56 R@100 at 26.29 FPS, compared with YOLOWorld at 33.70 R@100 and 30.59 FPS, and the depth-based false-positive reduction stage reduces FPR@100 by 18.4% on average while increasing R@100 by 4.1% on average (Schmarje et al., 2024).

Real-time OD is not limited to RGB closed-set boxes. LiDAR-only dynamic-object detection and tracking can also be formulated as real-time OD in robotics. The mapless system of “LiDAR-based Real-Time Object Detection and Tracking in Dynamic Environments” projects point clouds with intensity $p=mAP$3 into a 2D intensity image, extracts sparse foreground-feature points via Gaussian smoothing and thresholding, clusters them, estimates ego-motion from LiDAR intensity, and performs sliding-window motion analysis to identify dynamic objects, achieving an average detection accuracy of 88.7% and recall of 89.1% (Du et al., 2024). Oriented detection extends the output space rather than the sensing modality: RiO-DETR shows that once angle $p=mAP$4 is added to the box tuple $p=mAP$5, periodicity and semantics-dependent orientation require task-native designs such as shortest-path periodic loss

$p=mAP$6

without abandoning real-time efficiency (Hu et al., 10 Mar 2026).

5. Benchmarks, datasets, and application regimes

Real-time OD research is strongly shaped by benchmark geometry, annotation policy, and deployment target. General-purpose datasets such as COCO, PASCAL VOC, and ImageNet remain central in survey treatments, but many real-time conclusions in the literature arise from domain-specific benchmarks: Waymo for real-time autonomous driving, KITTI for traffic scenes and pedestrians, BDD100K and nuImages for diverse road conditions, Shift for synthetic domain shift, OD-VIRAT for elevated surveillance, RSOD for snowy scenes, Duckietown for resource-constrained robotics, and Fishyscapes or Segment Me If You Can for unknown-object evaluation (Bose et al., 17 Feb 2026, Khan et al., 2024).

Benchmark or dataset	Capture regime	Real-time relevance
Waymo Open Dataset real-time 2D track	5 synchronized camera views, $p=mAP$7 or $p=mAP$8	Hard 70 ms/frame runtime constraint (Zhang et al., 2021)
OD-VIRAT Tiny / Large	Stationary elevated surveillance cameras, 10 scenes	Tiny, occluded, long-range objects in realistic surveillance (Ullah et al., 16 Jul 2025)
RSOD	2100 real snowy traffic/town images	Real-snow adverse-weather OD with four difficulty levels (Ding et al., 2022)
KITTI / BDD100K / nuImages / Shift / TJU-DHD-Traffic	Driving scenes with day/night and weather variation	Traffic OD, pedestrian robustness, and real-time KPI analysis (Khan et al., 2024)
Duckietown object detection dataset	Small mobile robot environment	Embedded co-scheduling of YOLO, control, and OOD monitor (Yuhas et al., 2022)

The benchmark-specificity of detector design is a recurring theme. In Waymo, the central crop strategy was motivated by the empirical concentration of small-object centers in predictable image regions due to fixed camera geometry (Zhang et al., 2021). In OD-VIRAT, long-range surveillance geometry shifts the decisive metric from overall mAP to $p=mAP$9, because large-object AP is already high for many families (Ullah et al., 16 Jul 2025). In RSOD, severity grading by snow coverage rate reveals monotonic degradation with increasingly adverse conditions (Ding et al., 2022). A plausible implication is that “real-time object detection” is not a single benchmark task but a family of deployment regimes whose geometry and latency budgets materially change both architecture choice and metric interpretation.

6. Limitations, recurring debates, and future directions

Several recurrent misconceptions are directly contradicted by the literature. First, a detector can have respectable throughput and still be dangerously late in the control sense because end-to-end delay includes camera buffering, synchronization, and contention (Jang et al., 2020). Second, mAP alone can rank detectors in ways that are misaligned with autonomous-driving deployment, which is precisely why RTOP was proposed (Khan et al., 2024). Third, the fastest detector family on a conventional benchmark is not necessarily the strongest real-time detector in surveillance or other small-object-heavy regimes, as OD-VIRAT demonstrates for YOLOX versus Deformable-DETR and RetinaNet (Ullah et al., 16 Jul 2025). Fourth, transformers are not uniformly too slow for real-time OD; OmDet-Turbo, DEIMv2, and RiO-DETR all show counterexamples within their respective problem settings (Zhao et al., 2024, Huang et al., 25 Sep 2025, Hu et al., 10 Mar 2026).

The open research agenda is correspondingly multi-dimensional. Surveillance work explicitly identifies corruption robustness, tiny-object recall, occlusion reasoning, and duplicate suppression under clutter as unresolved problems (Ullah et al., 16 Jul 2025). The autonomous-vehicle hardware review calls for evaluation beyond average accuracy, including worst-case miss rates, timeliness under constrained compute, uncertainty estimation, anomaly or OOD detection, and tighter coupling of algorithms with hardware accelerators across multiple simultaneous cameras (Sali et al., 4 Sep 2025). Edge systems work suggests that future progress will come from coordinated optimization across data selection, detector sparsity, and hardware control rather than any single knob (Wu et al., 2024).

Another frontier is adaptation after deployment. ODIP is not a high-FPS detector paper, but it makes a different argument: in industrial systems, the dominant obstacle may be the cost of re-collecting and relabeling data when object sets change. Its interactive perception loop uses grasping, observation, release, pseudo-labeling, and few-shot adaptation to reduce human annotation requirements, implying that “practical real time” in some deployments includes adaptation latency across product or domain changes rather than frame-level inference alone (Chen et al., 2021). In autonomous driving, UNCOVER similarly suggests that unknown-object detection and post-hoc geometric filtering can be incorporated into a real-time one-stage detector with small architectural changes rather than only through heavyweight open-world pipelines (Schmarje et al., 2024).

Taken together, the literature supports a broad but precise view of real-time OD. It is not merely the pursuit of higher FPS, nor the pursuit of peak benchmark AP under unconstrained hardware. It is the design, evaluation, and deployment of detectors whose accuracy, end-to-end delay, throughput, robustness, power behavior, and adaptability are all compatible with the timing and resource structure of the target system—whether that system is an autonomous vehicle, an edge device, a surveillance installation, or a mobile robot (Jang et al., 2020, Sali et al., 4 Sep 2025).