Object Detection and Localization (ODAL)

• Object Detection and Localization (ODAL) is the integrated task of estimating both object identity and spatial position using methods such as 2D bounding boxes, 3D boxes, and semantic descriptors.
• Advanced methodologies like LocNet, SABL, and IoU-Net enhance detector performance by refining localization parameterization, calibration, and ranking under strict IoU criteria.
• Practical ODAL frameworks address challenges in annotation efficiency, computational constraints, and human-centric applications, driving innovations in autonomous driving and image captioning.

Object Detection and Localization (ODAL) denotes the joint estimation of object identity and object position. In the canonical formulation, the output is a class-labeled 2D bounding box; in broader contemporary usage, the localization target may be a 3D box, a world-frame object position, or an application-specific semantic location descriptor. The research literature treats ODAL not only as detector benchmarking under overlap-based criteria such as IoU and mAP, but also as a problem of localization parameterization, score ranking, calibration, human interpretation, annotation efficiency, and deployment under severe computational constraints (Balasubramaniam et al., 2022, Gidaris et al., 2015, Saucedo et al., 2024, Cao et al., 2023, Witte et al., 2024, Mészáros et al., 27 Aug 2025).

1. Scope and representational forms

ODAL is most commonly instantiated as category-level detection in images, where a detector predicts a set of class labels and bounding boxes. That conventional setting underlies localization-focused work such as LocNet, SABL, Double-Head, MLC, IoU-Net, MCCL, and localization distillation, all of which assume that detection quality depends jointly on classification and spatial accuracy (Gidaris et al., 2015, Wang et al., 2019, Wu et al., 2019, Zhang et al., 2021, Jiang et al., 2018, Pathiraja et al., 2023, Zheng et al., 2022). In autonomous-driving surveys, this decomposition is explicit: localization determines where an object is, while classification determines what it is, and both are constrained by real-time latency, robustness, and embedded efficiency (Balasubramaniam et al., 2022).

At the same time, the literature represented here broadens ODAL in three directions. First, some systems use detection as a downstream aid to humans rather than an autonomous endpoint; the counting study on aquatic-creature images evaluates not detector mAP in isolation but how overlaid detections affect human counting accuracy, response time, and trust (Witte et al., 2024). Second, some systems use object detection and localization as an intermediate representation for another task; the captioning model built on Faster R-CNN and masked-image localization vectors uses detected objects and their locations as the attention memory for language generation rather than as a final detection output (Yang et al., 2017). Third, newer works generalize localization beyond 2D image boxes: BOX3D lifts detections into 3D and then into a world frame (Saucedo et al., 2024), CoDA targets open-vocabulary 3D boxes for unseen categories (Cao et al., 2023), and the car-interior ODAL framework formulates localization as semantic position reporting in structured JSON such as "Seat.Row2.Middle" rather than as pixel-space rectangles (Mészáros et al., 27 Aug 2025).

Formulation	Localization output	Representative work
Closed-vocabulary 2D detection	2D bounding box	LocNet (Gidaris et al., 2015), SABL (Wang et al., 2019)
Human-assisted detection	Overlay boxes or dots	Counting/trust study (Witte et al., 2024)
Detection as intermediate representation	Object-centric annotation vectors	Captioning with ODAL (Yang et al., 2017)
Global 3D detection/localization	3D boxes in world frame	BOX3D (Saucedo et al., 2024)
Open-vocabulary 3D detection	3D box plus novel-category label	CoDA (Cao et al., 2023)
Semantic localization	Structured position string in JSON	Car-interior ODAL (Mészáros et al., 27 Aug 2025)

This diversity implies that ODAL is not reducible to one metric or one geometry. A plausible implication is that localization should be analyzed relative to the consuming task: pixel-tight boundaries, globally consistent 3D placement, and human-readable semantic positions are distinct localization targets even when all are framed as “detection and localization.”

2. Localization formulations and architectural design

A major research thread in ODAL concerns whether direct four-parameter box regression is the right localization primitive. LocNet argues that it is not. Given a candidate box $B$, it enlarges it by a factor $\gamma = 1.8$ to form a search region $R$, discretizes $R$ into $M=28$ rows and columns, and predicts row/column probabilities instead of regressing $(x_1,y_1,x_2,y_2)$ directly. It defines both In-Out probabilities and border probabilities, then infers the final box by maximizing likelihoods such as

$$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$

This structured formulation yields large gains at stricter localization thresholds: on VOC2007 with Fast R-CNN + 2k Edge Boxes, bbox regression gives $57.0$ mAP at IoU $0.7$ and $45.3$ COCO-style mAP, while Combined ML gives $\gamma = 1.8$0 and $\gamma = 1.8$1, respectively (Gidaris et al., 2015).

SABL advances a closely related critique from a different angle. Instead of center-and-size regression, it localizes the four sides of the box separately with side-aware features and a two-step bucketing-then-refinement scheme. The method predicts which bucket contains each boundary and then regresses a local offset inside the selected bucket, with the overall loss

$\gamma = 1.8$2

On COCO test-dev with ResNet-101-FPN, replacing standard box regression by SABL improves Faster R-CNN from $\gamma = 1.8$3 AP to $\gamma = 1.8$4 AP and $\gamma = 1.8$5 $\gamma = 1.8$6 to $\gamma = 1.8$7 $\gamma = 1.8$8; on RetinaNet it improves $\gamma = 1.8$9 AP to $R$0 AP (Wang et al., 2019). The consistent pattern is that gains are larger at stricter IoU thresholds, which indicates improvement in precise localization rather than only in recognition.

Architectural specialization for the two ODAL sub-tasks is another recurring theme. Double-Head shows that the dominant head families have opposite preferences: the fc-head is more suitable for classification, while the conv-head is more suitable for localization. The resulting detector assigns classification to a two-fc branch and box regression to a conv branch, yielding gains of $R$1 AP and $R$2 AP over FPN baselines with ResNet-50 and ResNet-101 on COCO, with especially large improvements at $R$3 (Wu et al., 2019). Earlier multiscale work reaches a related conclusion from the perspective of scale reasoning rather than head design: the Multi-Scale Structure detector replaces single-scale local windows with scale volumes,

$R$4

and learns scale-specific detectors over that joint descriptor, reducing localization errors and especially benefiting small objects and classes with large scale variation (Ohn-Bar et al., 2015).

A distinct alternative abandons one-shot regression entirely and treats localization as sequential search. The deep-reinforcement-learning study formulates active object localization as an MDP with IoU-based reward,

$R$5

and reports a best average IoU of $R$6 for the hierarchical method with non-overlapping subregions on VOC2012. The same study emphasizes that this is class-specific active localization rather than full modern multi-class detection (Samiei et al., 2022). This suggests that ODAL admits both feed-forward and search-based formulations, although the latter remains more specialized.

3. Ranking, confidence, calibration, and localization-aware supervision

A second major theme is that classification confidence is not a reliable proxy for localization quality. IoU-Net formalizes this directly by predicting a scalar localization confidence, namely the IoU between a detected box and its matched ground truth. On MS-COCO, it reports a Pearson correlation of 0.217 between IoU and classification confidence, versus 0.617 between IoU and predicted localization confidence. This predicted IoU is then used both for localization-aware NMS and for optimization-based box refinement with Precise RoI Pooling; for ResNet-101-FPN, the full system raises AP from $R$7 to $R$8 and $R$9 from $R$0 to $R$1 (Jiang et al., 2018).

MLC addresses the same divergence from the perspective of joint training and ranking. Its Mutual Labeling module redefines task positives per object using Otsu-thresholded IoU and score sets,

$R$2

and its IoU Rescoring module ranks detections by

$R$3

The method improves FCOS$R$4 from $R$5 AP to $R$6 AP and RetinaNet from $R$7 AP to $R$8 AP on COCO val, with gains larger at $R$9 than at $M=28$0 (Zhang et al., 2021). The underlying claim is that good detection ranking should reflect semantic correctness and geometric accuracy simultaneously.

MCCL extends this logic from ranking to calibration. It defines a detection-specific calibration criterion and uses D-ECE to evaluate the joint calibration of confidence and box geometry. At train time, it estimates class-wise and box-wise uncertainty via MC dropout, introduces a multiclass confidence calibration loss and a localization calibration loss,

$M=28$1

and shows consistent D-ECE reductions on both in-domain and out-of-domain settings for FCOS and Deformable DETR while largely preserving AP (Pathiraja et al., 2023). The practical implication is that ODAL reliability is not exhausted by mAP: confidence should track true detection correctness under localization constraints.

Knowledge transfer for localization follows a similar pattern. “Localization Distillation for Object Detection” argues that earlier logit-based KD underperformed because it did not distill localization. Its Localization Distillation matches teacher and student localization distributions over discretized box variables,

$M=28$2

and combines this with Valuable Localization Region selection. On COCO, ResNet-50 students consistently gain about $M=28$3 AP over baseline GFocal, with larger improvements at $M=28$4 (Zheng et al., 2022). Across these lines of work, a common misconception is explicitly challenged: high class confidence does not imply good localization, and localization knowledge must be modeled, transferred, or calibrated on its own terms.

4. Human-centered ODAL and downstream utility

The human-in-the-loop counting study provides one of the clearest demonstrations that detector benchmarking and downstream utility can diverge. In a Mechanical Turk experiment on 30 aquarium images, participants counted aquatic creatures under conditions including clean images, perfect boxes, shifted boxes with IoU $M=28$5, center dots, false positives, and false negatives. The baseline mean absolute counting error on clean images is $M=28$6; with perfect boxes it falls to $M=28$7. Crucially, shifted boxes at IoU $M=28$8 produce $M=28$9 error and trust $(x_1,y_1,x_2,y_2)$0, versus $(x_1,y_1,x_2,y_2)$1 and $(x_1,y_1,x_2,y_2)$2 for perfect boxes, with no significant difference in accuracy or trust. By contrast, false positives and false negatives sharply reduce trust and worsen performance, and the paper concludes that “it is more important to optimize the F1-score over the IoU” in that human-computer task (Witte et al., 2024). The same study further reports that center dots outperform perfect boxes for counting, with error $(x_1,y_1,x_2,y_2)$3 versus $(x_1,y_1,x_2,y_2)$4, and are more resilient to localization perturbation.

This result does not imply that localization never matters. The paper itself restricts its claim to a visual multi-object counting task on clear aquarium images and explicitly notes that tasks such as lesion measurement, robotic grasping, tracking under occlusion, instance segmentation, or safety-critical auditing may remain localization-sensitive (Witte et al., 2024). The stronger conclusion is narrower and more useful: downstream human performance may depend more on presence/absence errors and interface design than on moderate box misalignment.

A different form of downstream integration appears in image captioning. “Image Captioning with Object Detection and Localization” replaces generic image subregions with detected objects and masked-image localization embeddings, forming annotation vectors

$(x_1,y_1,x_2,y_2)$5

which are consumed by an attention-based LSTM decoder (Yang et al., 2017). This work does not propose a new detector or localization loss, but it shows that ODAL outputs can serve as semantically grounded intermediate representations. Taken together, these papers suggest that ODAL should be analyzed not only as an endpoint task but also as an interface layer between perception, language, and human decision making.

5. Annotation efficiency, active learning, and supervision economy

Because precise localization labels are expensive, several papers study how to prioritize, reduce, or restructure annotation. “Localization-Aware Active Learning for Object Detection” argues that classification uncertainty alone is inadequate because object detection is uncertainty over class-location hypotheses. It introduces localization tightness,

$(x_1,y_1,x_2,y_2)$6

and localization stability,

$(x_1,y_1,x_2,y_2)$7

then combines stability with classification uncertainty as $(x_1,y_1,x_2,y_2)$8. On PASCAL 2012 and 2007, the localization-stability method achieves average relative improvements over the classification-only baseline of 96.5% and 81.9%, respectively, in labeled-image savings (Kao et al., 2018). The key ODAL insight is that annotation value is partly geometric: images where boxes are unstable under perturbation are informative even when class scores are not especially uncertain.

ALWOD pushes the same idea into weakly and semi-supervised detection. It begins with weak image-level labels, creates an auxiliary fully annotated synthetic set from a tiny seed of true boxes, trains a student-teacher detector, and then selects informative weakly labeled images using student-teacher disagreement and teacher entropy:

$(x_1,y_1,x_2,y_2)$9

$$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$0

The product fusion of these signals performs best. With only 5% fully annotated real images on VOC2007 and RealPizza10, and 1% on COCO2014, ALWOD with Swin-T + Sparse DETR reaches $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$1 AP50 on VOC2007, $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$2 AP50 / $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$3 AP on COCO2014, and $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$4 AP50 on RealPizza10 (Wang et al., 2023). Its labeling protocol is also localization-specific: annotators select and correct model-proposed boxes rather than drawing all boxes from scratch, reducing average annotation time to about 2 s with only a 0.6 AP50 drop relative to full drawing on RealPizza10 (Wang et al., 2023).

Ladder addresses the same annotation problem from a software perspective rather than an acquisition-theoretic one. It provides a GUI workflow for manual rectangle drawing, YOLOv3-based prelabeling, editing, retraining, and deployment under a recurrent “Prediction $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$5 Edit $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$6 Update” loop (Tang et al., 2023). The work is explicit that it is a tool and workflow contribution rather than a new detector, but it shows how ODAL pipelines can be made iterative and usable for domain scientists.

6. Resource-aware ODAL and deployment constraints

In autonomous driving and related embedded settings, ODAL is governed by a persistent speed-accuracy-resource trade-off. The AV survey emphasizes that perception systems must meet four simultaneous requirements: accuracy, latency, robustness, and hardware efficiency. It contrasts historically accurate but slow two-stage detectors such as R-CNN, Fast R-CNN, and Faster R-CNN with one-stage families such as YOLO, SSD, and RetinaNet, and further distinguishes 2D detection from 3D detection using camera, LiDAR, and radar (Balasubramaniam et al., 2022). The survey’s examples are illustrative: YOLOv3 is reported at 95.2 fps and $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$7 mAP on COCO, while R-CNN is reported at 0.02 fps and $$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$8 mAP on Pascal VOC (Balasubramaniam et al., 2022). The implication is not that one-stage methods are universally preferable, but that AV deployment turns latency into a first-class design variable.

Compression and pruning work refines this deployment view by treating localization preservation as a pruning objective. Localization-aware Channel Pruning extends discrimination-aware pruning with contextual RoIAlign and a localization-aware auxiliary loss

$$L_{\textrm{borders}}(B)=p_l(B_l)\cdot p_t(B_t)\cdot p_r(B_r)\cdot p_b(B_b).$$9

where the regression term uses GIoU (Xie et al., 2019). On COCO with SSD + ResNet-50 at 70% pruning, LCP achieves $57.0$0 mAP versus $57.0$1 for DCP, with $57.0$2 versus $57.0$3; on VOC with SSD + VGG-16 at 75% pruning, it preserves $57.0$4 mAP from an original $57.0$5 (Xie et al., 2019). The larger gains at stricter IoU thresholds support the paper’s central claim that compression methods designed for classification can damage box quality unless localization-sensitive signals are preserved explicitly.

This line of work suggests a general deployment principle: resource reduction should not be treated as orthogonal to localization. In ODAL systems, especially in robotics and driving, preserving spatial precision under compression is often more consequential than preserving class accuracy alone.

7. 3D, open-vocabulary, and semantic extensions

Modern ODAL increasingly extends beyond closed-vocabulary 2D detection. BOX3D is representative of a lightweight geometric 3D pipeline. It uses YOLOv8 detection and segmentation on RGB, projects synchronized LiDAR points into the image,

$57.0$6

selects point clusters by mask membership, fits local 3D boxes, transforms them into a world frame, merges duplicates across viewpoints, and refines clusters through point-to-voxel comparison on the global map (Saucedo et al., 2024). The three layers take $57.0$7 ms, $57.0$8 ms, and $57.0$9 ms on the reported CPU setup, and the framework is explicitly oriented toward global object localization rather than per-frame detection alone.

CoDA generalizes ODAL along another axis: open-vocabulary 3D detection under limited base-category supervision. It trains a class-agnostic 3D detector, projects predicted 3D boxes to 2D, uses CLIP image-text similarity over a super-category vocabulary, and accepts a proposal as a novel pseudo-box if it passes geometric and semantic thresholds while staying sufficiently far from seen-category boxes:

$0.7$0

It then aligns 3D object features with image and text features through class-agnostic distillation and discovery-driven class-specific alignment (Cao et al., 2023). On SUN-RGBD, full CoDA reaches $0.7$1 $0.7$2 at IoU $0.7$3, improving over the 3D-CLIP baseline’s $0.7$4 (Cao et al., 2023). The important ODAL point is that localization for unseen categories is explicitly bootstrapped rather than assumed.

The car-interior ODAL framework extends the notion of localization yet again, replacing box coordinates with structured semantic positions and introducing ODALbench. Its pipeline runs the vision encoder onboard, uploads embeddings to the cloud, and returns JSON strings that specify detected objects, semantic positions, and visibility (Mészáros et al., 27 Aug 2025). The reported evaluation uses

$0.7$5

alongside $0.7$6. The paper also contains a reported inconsistency: the abstract and conclusion claim an $0.7$7 of $0.7$8 for the fine-tuned ODAL-LLaVA and a near-$0.7$9 advantage over GPT-4o, whereas the detailed results table reports GPT-4o at 89.29% and the best tabulated ODAL-LLaVA variant at 76.79%, though with a much higher $45.3$0 of 7.1428 versus 2.44 for GPT-4o (Mészáros et al., 27 Aug 2025). Even with that discrepancy, the broader contribution is clear: ODAL can be evaluated as structured semantic localization under open-vocabulary reasoning and deployment constraints, not only as bounding-box prediction.

Across 3D, open-vocabulary, and semantic formulations, the shared pattern is that localization is becoming less tied to a single image-plane rectangle. ODAL now includes global consistency, vocabulary expansion, and human-readable spatial semantics alongside classical box fitting, while still retaining the central requirement that object identity and object position be estimated jointly.