ODAL-LLaVA: Multimodal Object Detection in Car Interiors

• The paper presents ODAL-LLaVA’s core contribution: a fine-tuned LLaVA-1.5 model for in-car, structured object detection and semantic localization.
• It employs a distributed architecture that executes vision encoding on-board while performing multimodal reasoning in the cloud under privacy and resource constraints.
• Results show significant improvements over baseline models, achieving 100% JSON compliance and high ODALbench scores compared to GPT-4o.

Searching arXiv for the main paper and closely related backbone papers. arXiv search: "Scalable Object Detection in the Car Interior With Vision Foundation Models" ODAL-LLaVA is the fine-tuned, in-car version of LLaVA-1.5 7B that serves as the multimodal core of the Object Detection and Localization (ODAL) framework for interior scene understanding in cars. It is specialized for detecting arbitrary externally introduced objects in car interiors, assigning them to standardized semantic locations such as Seat.Row2.Middle or UNDEFINED, and emitting structured JSON that can be consumed directly by downstream systems such as a car’s personal assistant (Mészáros et al., 27 Aug 2025). Within the ODAL design, ODAL-LLaVA is deployed in a distributed on-board/cloud architecture: vision encoding is executed locally in the vehicle, while multimodal reasoning and generation are executed in the cloud. The model is therefore positioned at the intersection of open-vocabulary detection, semantic localization, structured multimodal generation, and resource-constrained automotive deployment.

1. ODAL-LLaVA within the ODAL framework

The ODAL framework is designed for interior scene understanding in cars, with a particular emphasis on identifying externally introduced objects such as wallets, backpacks, laptops, and keys, and localizing them semantically within the vehicle (Mészáros et al., 27 Aug 2025). The target output is directly usable by a personal assistant, for example in reminders about forgotten items when leaving the vehicle.

Within this framework, ODAL-LLaVA is a fine-tuned variant of LLaVA-1.5 7B specialized for three coupled tasks: detecting arbitrary objects in car interiors, assigning them to a car-specific semantic ontology of positions, and generating machine-readable JSON responses. The role of the model is therefore not merely descriptive. It functions as a structured detector and localizer that converts generic multimodal reasoning into a domain-constrained representation aligned with automotive assistance workflows.

A central property of ODAL is that it frames localization semantically rather than geometrically. Instead of predicting bounding boxes, the model is trained to output human-understandable location labels such as seat row and seat position, plus UNDEFINED for non-standard placements. This makes ODAL-LLaVA particularly suitable for assistant-facing interaction, where linguistic location descriptions are often more actionable than pixel coordinates. A plausible implication is that the framework prioritizes downstream interpretability over conventional detection-head geometry.

The paper also positions ODAL-LLaVA against both classical closed-set detectors and large general-purpose vision foundation models. In that comparison, the key differentiator is not only model size, but the combination of open-vocabulary recognition, semantic localization, strict output formatting, and operation under automotive compute constraints (Mészáros et al., 27 Aug 2025).

2. Distributed architecture, privacy, and scalability

ODAL uses a distributed pipeline that splits computation between the vehicle and the cloud (Mészáros et al., 27 Aug 2025). On-board, the system performs vision encoding only. The vehicle captures high-resolution interior images at $4000 \times 3000$ resolution in the dataset, and a vision foundation model encoder runs locally. In the LLaVA-based configuration, this encoder is CLIP ViT-L/14. The encoder converts the image into visual embeddings.

This partition is motivated by both privacy and resource constraints. Raw interior images, which may contain passengers and personal items, never leave the car. Only image embeddings are transmitted. At the same time, the paper states that on-board hardware is computationally constrained, making full deployment of a multimodal transformer impractical. Running only the encoder is significantly cheaper than running the complete model stack.

In the cloud, the embeddings are combined with a text prompt and passed to the multimodal model, namely ODAL-LLaVA, which performs object detection and localization and returns a structured JSON response. That response can be consumed by cloud services or sent back to the car for use by onboard applications such as the personal assistant (Mészáros et al., 27 Aug 2025).

The paper explicitly presents this split as the mechanism by which large vision foundation models can be used without requiring heavy on-board compute. Scalability is further supported by three implementation choices: use of a lightweight 7B LLaVA variant, 4-bit quantization, and LoRA-based fine-tuning. The text also states that the cloud side can exploit ample memory and compute, while the on-board side remains limited to encoder-only processing. This suggests that ODAL-LLaVA is less a standalone model than a deployment-specific instantiation of a broader systems architecture.

3. Architecture and task formulation

Architecturally, ODAL-LLaVA uses the LLaVA 1.5 7B backbone (Mészáros et al., 27 Aug 2025). Its components are:

• Vision encoder: CLIP ViT-L/14, with $336 \times 336$ input resolution.
• LLM: Vicuna 7B 1.5, based on LLaMA 2.
• Vision-language connector: a two-layer MLP that maps CLIP features into the LLM embedding space.
• Adaptation mechanism: LoRA adapters on selected or all linear layers.

The visual token embeddings are denoted $V \in \mathbb{R}^{T_v \times d_v}$, and the connector maps them conceptually as

$E_v = \text{MLP}(V),$

where $E_v \in \mathbb{R}^{T_v \times d_{LM}}$. The decoder then performs autoregressive generation conditioned on $E_v$ and prompt tokens.

ODAL-LLaVA does not introduce specialized detection heads such as bounding-box regressors or segmentation heads. Instead, it reuses the existing language generation head to enumerate objects, output location labels, and produce structured JSON. This is a defining technical characteristic of the model. The task is cast as textual detection and semantic localization rather than pixel-level geometry. The paper therefore emphasizes evaluation methodology and fine-tuning alignment rather than detector-head design (Mészáros et al., 27 Aug 2025).

The model accepts multimodal input. In the distributed setting, the image arrives as pre-encoded CLIP embeddings; during training, raw images may be used. The text prompt can take one of two forms. Prompt V1 is detailed: it describes the car interior setting, specifies naming conventions for positions, and includes JSON response examples. Prompt V2 is minimalistic: it states that the environment is a car interior and asks for detection and localization without detailed naming conventions or output examples.

The target output is a JSON string keyed by object names in open vocabulary form. Each object is associated with two attributes, position and is_visible. The paper gives the following structure:

$E_v = \text{MLP}(V),$3

This format makes the model jointly responsible for presence estimation, semantic localization, and adherence to a parseable schema. A common misconception would be to treat ODAL-LLaVA as a conventional detector with a language wrapper; the paper instead presents it as a generative multimodal model aligned to a structured detection-localization protocol.

4. Dataset construction and fine-tuning regime

The ODAL study uses a custom car interior dataset built specifically for the task (Mészáros et al., 27 Aug 2025). Images were captured with a GoPro Hero7 Black mounted inside a BMW. The dataset contains 223 images at $4000 \times 3000$ resolution, split into 80% training and 20% validation. It covers 13 distinct object types and 5 predefined interior locations, with non-standard placements labeled as UNDEFINED. Ground truth is represented as structured JSON per image.

To address data scarcity, the paper evaluates three augmentation levels: no augmentation, basic augmentation, and extensive augmentation. Basic augmentation consists of rotation, flipping, and brightness adjustment. Extensive augmentation adds affine transformations and sharpness adjustments. The reported findings are highly asymmetric. Basic augmentation gives the best ODALbench performance, with ODAL$_{score}$ of 74.10%, ODAL$_{SNR}$ of 1.2205, and JSON format success of 97.72%. Extensive augmentation degrades performance severely, yielding ODAL$_{score}$ 19.64%, ODAL$336 \times 336$0 0.55, and JSON Rate 6.81%. On that basis, basic augmentation is selected as the default setting for later experiments.

Fine-tuning is implemented with the Hugging Face ecosystem, specifically transformers, TRL, and PEFT (Mészáros et al., 27 Aug 2025). The core adaptation mechanism is LoRA, in which low-rank matrices are added to linear layers and only the LoRA parameters are updated. The model is trained with 4-bit quantization via BitsAndBytes to reduce memory usage.

Two main fine-tuning strategies are described. In selective fine-tuning, only attention projection layers are adapted. In comprehensive fine-tuning, all linear layers are adapted. The experiments also vary whether fine-tuning is applied to the LLM only or to both the LLM and the vision encoder, and whether the prompt is V1 or V2. Although exact hyperparameters are not listed, the paper characterizes the setup as standard supervised instruction tuning in TRL/PEFT, with the target behavior being replication of the desired JSON strings from image-plus-prompt input.

The behavioral shift relative to untuned LLaVA is pronounced. Baseline LLaVA 1.5 is reported to produce verbose, unstructured responses, poor JSON adherence, and a high hallucination rate. After fine-tuning, ODAL-LLaVA achieves strict JSON compliance, large gains in detection and localization, and a substantial reduction in hallucinations (Mészáros et al., 27 Aug 2025).

5. ODALbench and reported performance

The paper introduces ODALbench as a task-specific benchmark for evaluating detection accuracy, localization accuracy, hallucinations, and output format compliance in car interior object detection and localization (Mészáros et al., 27 Aug 2025). Because generative models may use different object names for the same item, the evaluation adopts an LLM-as-a-judge scheme. The candidate model produces a JSON-like or textual response, a judge LLM post-processes the response to extract detections and hallucinations and to normalize object naming, and a separate script computes ODAL$336 \times 336$1 and ODAL$336 \times 336$2 against ground truth.

The per-frame ODAL score is defined as

$336 \times 336$3

where $336 \times 336$4 indicates correct detection, $336 \times 336$5 indicates correct localization, $336 \times 336$6 is the total number of correctly detected objects, and $336 \times 336$7 is the Kronecker delta. Correctly detected and correctly localized objects contribute 1 point; correctly detected but incorrectly localized objects contribute 0.5 points; and if no object is correctly detected, the frame receives a base score of 1. The text additionally states that hallucinated objects incur a penalty of $336 \times 336$8, although this term does not appear explicitly in the displayed formula.

The hallucination-sensitive ratio is defined as

$336 \times 336$9

where $V \in \mathbb{R}^{T_v \times d_v}$0 is the total number of correct detections and $V \in \mathbb{R}^{T_v \times d_v}$1 is the total number of hallucinations. If $V \in \mathbb{R}^{T_v \times d_v}$2, the ratio formally diverges and is capped in practice at the maximum possible number of correct detections in the dataset.

The core reported metrics are as follows:

Model or setting	ODAL$V \in \mathbb{R}^{T_v \times d_v}$3	ODAL$V \in \mathbb{R}^{T_v \times d_v}$4
GPT-4o baseline	89.29%	2.44
Baseline LLaVA 1.5	17.86%	0.1
Fine-tuned ODAL-LLaVA, best Table 2 setup	76.79%	7.1428
Final ODAL-LLaVA in conclusion	89%	about 3× GPT-4o

The paper also reports JSON compliance. GPT-4o achieves 100% JSON rate. Baseline LLaVA has poor JSON compliance. Fine-tuned ODAL-LLaVA reaches 100% correct JSON response rate across evaluation (Mészáros et al., 27 Aug 2025).

Two quantitative summaries in the paper require careful reading. In the detailed ODALbench comparison, GPT-4o is reported at 89.29% ODAL$V \in \mathbb{R}^{T_v \times d_v}$5, whereas the conclusion states that the final ODAL-LLaVA achieves an ODAL$V \in \mathbb{R}^{T_v \times d_v}$6 of 89%, representing a 71% improvement over baseline LLaVA and outperforming GPT-4o by nearly 20%. The text elsewhere notes that this comparison is based on a different normalization. The most stable cross-section of the results is therefore the qualitative pattern: untuned LLaVA performs poorly; fine-tuned ODAL-LLaVA sharply improves both structured output and hallucination behavior; and the final model is reported as competitive with or superior to GPT-4o on the task-specific benchmark (Mészáros et al., 27 Aug 2025).

6. Hallucination behavior, comparative positioning, and limitations

In the ODAL setting, a hallucination is an object reported by the model that is not present in the ground-truth label for that frame (Mészáros et al., 27 Aug 2025). The paper treats hallucination control as a central problem rather than a peripheral failure mode. This is reflected in both ODAL$V \in \mathbb{R}^{T_v \times d_v}$7, which imposes a penalty for hallucinated objects, and ODAL$V \in \mathbb{R}^{T_v \times d_v}$8, which directly measures correct detections relative to hallucinations.

The reported ratios distinguish the models sharply. Baseline LLaVA has $V \in \mathbb{R}^{T_v \times d_v}$9, corresponding to roughly ten hallucinated objects per correct detection on average. GPT-4o has $E_v = \text{MLP}(V),$0. ODAL-LLaVA reaches $E_v = \text{MLP}(V),$1 in the best ODALbench setup, and the paper summarizes the final model as having an ODAL$E_v = \text{MLP}(V),$2 three times higher than GPT-4o (Mészáros et al., 27 Aug 2025). The paper attributes this reduction to domain-specific fine-tuning, structured prompting and output specification, LoRA adaptation of both vision encoder and LLM, and model selection guided by ODALbench.

Relative to classical detectors such as YOLO and Faster R-CNN, the ODAL approach is described as addressing three limitations: closed-set label dependence, inability to produce semantic location descriptions such as “second row middle seat,” and weaker direct integration with assistant-facing natural-language systems (Mészáros et al., 27 Aug 2025). ODAL-LLaVA instead provides open-vocabulary detection, language-based localization, and prompt-based customization without retraining. This suggests a shift from geometry-centric detection pipelines toward generative task interfaces in domains where semantic localization is operationally sufficient.

The paper also identifies clear limitations. Full multimodal reasoning currently resides in the cloud, so the system depends on cloud connectivity. The dataset is small, comprising 223 images from a single vehicle type, with limited object and scenario diversity. The authors explicitly raise the question of “potential loss of the model’s general knowledge due to domain-specific fine-tuning.” ODALbench relies on GPT-4o as judge, which introduces possible evaluation bias, and the metrics do not include geometric accuracy because localization is semantic rather than pixel-based (Mészáros et al., 27 Aug 2025).

Future directions named in the paper include broader validation across more objects, more challenging scenes, and multiple vehicle types; extension of ODALbench to capture more nuanced aspects of multimodal detection and localization; systematic hallucination control methods such as conditional generation constraints and confidence-based filtering; and extension of ODAL-style frameworks beyond car interiors to domains such as home robots and warehouses. A plausible implication is that ODAL-LLaVA is best understood not as a finalized detector, but as a domain-specialized demonstration of how a lightweight open multimodal model can be aligned to structured perception tasks under privacy and compute constraints.