ODALbench: In-Car Object Detection Benchmark

• ODALbench is a benchmark for in-car scene understanding that detects arbitrary objects and assigns them semantic cabin regions using JSON outputs.
• It employs a custom BMW dataset with 223 high-res images and evaluates models via ODAL_score and ODAL_SNR, emphasizing correct detections and penalizing hallucinations.
• The evaluation protocol integrates LLM-mediated judging to compare baseline and fine-tuned models, ensuring semantic alignment in object detection and localization.

ODALbench is a task-specific benchmark for evaluating how vision foundation models perform object detection and semantic localization in car interiors, with explicit emphasis on hallucinations, meaning false or invented objects. It was introduced together with the ODAL (Object Detection and Localization) framework in "Scalable Object Detection in the Car Interior With Vision Foundation Models" (Mészáros et al., 27 Aug 2025). The benchmark targets in-car scene understanding of the kind required by assistants that can produce semantically grounded statements such as “Your backpack is on the second-row middle seat,” and it is designed for settings in which models return structured JSON rather than bounding boxes or closed-set labels.

1. Origin and problem formulation

ODALbench was created for interior scene understanding in cars, specifically for detecting arbitrary personal objects such as backpacks, laptops, and wallets, and localizing them to semantic cabin regions such as "Seat.Row2.Middle", "Front passenger seat", or "UNDEFINED" for atypical placements (Mészáros et al., 27 Aug 2025). The benchmark is therefore not a conventional object-detection benchmark in the mAP sense. It assumes open-vocabulary objects rather than a fixed training label set, semantic categories for location rather than bounding boxes, and generative model outputs in structured JSON.

This design reflects the problem setting of the ODAL framework itself. Computation is split between on-board and cloud components so that foundation models need not run entirely inside the vehicle. Within that setting, evaluation must address not only whether a model names the right objects, but also whether it assigns the correct semantic position and whether it hallucinates objects that are not present.

Standard metrics such as mAP, precision/recall, and F1 are treated as insufficient for this purpose because they typically assume fixed categories and bounding-box IoU, do not treat semantic positions as discrete correctness conditions, do not explicitly quantify hallucinations relative to correct detections, and do not operate naturally on generative JSON outputs. ODALbench was therefore introduced to evaluate four interacting properties: whether the right objects are detected, whether the semantic location is correct, how often hallucinations occur, and how those factors jointly affect assistant quality.

2. Dataset and annotation regime

The benchmark is evaluated on a custom in-car dataset recorded with a GoPro Hero7 Black mounted inside a BMW, at 4000 × 3000 resolution so that small objects such as keys or wallets remain visible (Mészáros et al., 27 Aug 2025). The dataset contains 223 images, 13 distinct personal items, and 5 predefined positions in the car interior. The split is 80% training and 20% validation.

Annotations are stored in structured JSON. A representative example is:

$f$5

The key fields are "position" and "is_visible". The former encodes semantic location; the latter indicates whether the object is visible and therefore detectable. Non-standard placements, such as objects on the middle console, are labeled "UNDEFINED".

Three augmentation regimes are described: no augmentation, basic augmentation with rotation, flipping, and brightness adjustment, and extensive augmentation with the basic set plus affine transformations and sharpness adjustments. For ODALbench, the analysis ultimately focuses on basic augmentation, which is reported as giving the best performance.

The dataset design makes explicit what the benchmark measures. It does not evaluate pixel-level localization or geometric reconstruction; it evaluates whether the model can recover object identity and assign the correct cabin-level semantic slot.

3. Metric architecture: ODAL$_{\text{score}}$ and ODAL$_{\text{SNR}}$

ODALbench consists of two metrics: ODAL$_{\text{score}}$ and ODAL$_{\text{SNR}}$ (Mészáros et al., 27 Aug 2025). Together they summarize detection accuracy, localization quality, and hallucination severity.

For a frame $f$, ODAL$_{\text{score}}$ uses two binary indicators for each possible object $i$:

• $d_i \in \{0,1\}$: detection correctness
• $l_i \in \{0,1\}$: localization correctness

The frame score is given as

$$S_f = \sum_{i=1}^{N} \left( d_i \cdot l_i + d_i \cdot \frac{1-l_i}{2} \right) + \delta_{0,D}$$

where $_{\text{SNR}}$0 is the number of ground-truth objects in the frame, $_{\text{SNR}}$1 is the number of correctly detected objects, and

$_{\text{SNR}}$2

The interpretation is discrete and asymmetric. A correctly detected and correctly localized object contributes 1. A correctly detected but incorrectly localized object contributes 0.5. An object that is not correctly detected contributes 0. In addition, if no object is correctly detected in the frame, the metric awards a baseline +1 through $_{\text{SNR}}$3.

The benchmark description also states that hallucinations incur a minus one penalty per false positive. The printed equation does not explicitly contain a $_{\text{SNR}}$4 term, but the narrative states that hallucinated objects are penalized. A consistent re-implementation therefore uses

$_{\text{SNR}}$5

where $_{\text{SNR}}$6 is the number of hallucinated objects in frame $_{\text{SNR}}$7.

Across $_{\text{SNR}}$8 frames, the total score is

$_{\text{SNR}}$9

and the reported percentage score is

$_{\text{score}}$0

with $_{\text{score}}$1 defined as the maximum achievable score over the dataset.

The second metric, ODAL$_{\text{score}}$2, isolates the balance between correct detections and hallucinations:

$_{\text{score}}$3

where $_{\text{score}}$4 is the total number of correct detections and $_{\text{score}}$5 is the total number of hallucinated objects. Localization is not directly encoded in $_{\text{score}}$6; ODAL$_{\text{score}}$7 is intended as a signal-to-noise ratio for detection versus false invention. When $_{\text{score}}$8, the metric formally diverges. The benchmark handles this case by taking the maximum possible value of ODAL$_{\text{score}}$9.

4. Evaluation protocol and LLM-mediated judging

ODALbench is built around the fact that vision-LLMs are generative and may produce outputs that vary lexically or structurally from run to run (Mészáros et al., 27 Aug 2025). Direct string matching is therefore inadequate. The paper addresses this with an LLM-as-a-judge procedure, using GPT-4o as the judge model.

The evaluation pipeline has three stages. First, the prediction model is prompted with a system prompt and a user prompt that ask it to detect and localize objects and return JSON. The user prompt may be either detailed or minimalistic. Second, the prediction JSON is passed to the judge LLM, which returns structured evaluation data: the number of objects detected, a list of hallucinated objects, and, for each ground-truth object, whether it was detected and whether the predicted location matches the annotated semantic position. Third, a dedicated evaluation script computes the per-frame score $_{\text{SNR}}$0, the dataset-level ODAL$_{\text{SNR}}$1, and the counts needed for ODAL$_{\text{SNR}}$2.

This protocol is explicitly semantic rather than geometric. Detection correctness is discrete and judged semantically, so that variants such as “rucksack” and “backpack” can be aligned. Localization correctness is also discrete: either the predicted "position" matches the ground-truth category, or it does not. There is no IoU, distance error, or bounding-box threshold.

A notable design choice is the frame-level baseline term $_{\text{SNR}}$3. The benchmark description characterizes this as a way to avoid zero scores and structure the range of possible values. A related point of attention is the hallucination penalty: because it is stated in the narrative but absent from the printed equation, implementations must decide whether to follow the equation literally or the textual description. The benchmark overview argues for the latter when the goal is a comprehensive measure of generative interior-scene understanding.

5. Use within the ODAL framework

The benchmark is used to compare GPT-4o, LLaVA 1.5 7B, and a fine-tuned ODAL-LLaVA within the ODAL framework (Mészáros et al., 27 Aug 2025). In this system, LLaVA 1.5 7B is described as comprising a CLIP ViT-L/14 visual encoder, a Vicuna 7B language decoder, and a two-layer MLP connector between vision and language.

Fine-tuning is performed with LoRA, Hugging Face tools (Transformers, TRL, PEFT), and 4-bit quantization to reduce memory footprint. Two adaptation strategies are described. In selective fine-tuning, only attention projection layers are adapted. In comprehensive fine-tuning, all linear layers are adapted, in the LLM and optionally the vision encoder. Two prompt variants are also studied: V1, a detailed prompt with explicit instructions about interior context, naming conventions, and JSON format, and V2, a minimalistic prompt with less output-format guidance.

ODALbench is the instrument by which these choices are compared. It measures the extent to which fine-tuning improves detection and semantic localization while suppressing hallucinations, and it also captures a practical systems property through JSON rate, namely whether the model consistently emits correctly formatted JSON.

6. Reported results, interpretation, and limitations

The benchmark reports distinct quantitative outcomes for different settings, including figure-based results, tabled ablations, and the abstract-level summary (Mészáros et al., 27 Aug 2025).

Model or configuration	ODAL$_{\text{SNR}}$4 (%)	ODAL$_{\text{SNR}}$5
GPT-4o	89.29	2.44
Baseline LLaVA 1.5	17.86	0.1
Fine-tuned ODAL-LLaVA, main configuration	69.64	7.14
Best ODAL-LLaVA variant (V2)	76.79	7.1428

The abstract reports another configuration in which fine-tuned ODAL-LLaVA reaches 89% ODAL$_{\text{SNR}}$6, representing a 71% improvement over its baseline performance and outperforming GPT-4o by nearly 20%. The benchmark overview notes that these values come from different experiments and settings, and therefore should not be read as a single internally identical ranking across all configurations.

In substantive terms, ODALbench interprets a high ODAL$_{\text{SNR}}$7 as reliable object finding plus correct semantic attribution, and a high ODAL$_{\text{SNR}}$8 as a favorable ratio of correct detections to hallucinations. On that reading, the fine-tuned ODAL-LLaVA substantially reduces hallucinations relative to GPT-4o, achieving an ODAL$_{\text{SNR}}$9 about 2.9× higher in the figure-based comparison. The best ODAL-LLaVA variant is also reported with 100% JSON rate, which indicates consistent structured output for downstream consumption.

The benchmark’s strengths are explicitly identified as comprehensive evaluation of detection and localization, sensitivity to hallucinations, semantic alignment with conversational assistant use, robustness to open-vocabulary and generative outputs, and interpretability through the scoring rules $f$0, $f$1, and $f$2. Its limitations are equally clear. The dataset covers only 223 images, 13 object types, and 5 main positions, all recorded in a single BMW with one camera setup. The metric evaluates semantic positions rather than geometry, so it is not suited to tasks requiring fine-grained spatial accuracy. It depends on GPT-4o as judge, which introduces proprietary-model dependence and the possibility of judge bias. The source also identifies a broader research question: whether domain-specific fine-tuning for ODAL tasks may reduce a model’s general knowledge.

The stated future directions are to validate on more object types, more complex scenes with clutter, occlusion, and varying lighting, and different vehicle models and interior layouts, and to examine general-knowledge retention under domain specialization. A plausible extension, suggested by the benchmark discussion, is richer treatment of location hierarchies and more nuanced handling of the $f$3 case in ODAL$f$4; however, those refinements are presented as implications rather than as part of the current specification.