DepthVision: LiDAR-Assisted RGB Synthesis

• DepthVision is a multimodal framework that synthesizes RGB images from sparse LiDAR data using a conditional GAN and a refiner network.
• It dynamically fuses LiDAR-generated and real RGB images via a luminance-aware mechanism to enhance vision-language model reliability in adverse conditions.
• Empirical evaluations show significant performance gains in low-light and safety-critical tasks on both simulated and real-world datasets without retraining downstream models.

DepthVision is a framework for multimodal scene understanding designed to address unreliable camera input in robotics. It synthesizes RGB images from sparse LiDAR point clouds using a conditional generative adversarial network with an integrated refiner network, and then combines the synthetic views with real RGB data using a Luminance-Aware Modality Adaptation (LAMA) mechanism that blends the two data sources dynamically based on ambient lighting conditions (Kirchner et al., 9 Sep 2025). Its stated purpose is to compensate for sensor degradation such as darkness or motion blur without requiring any fine-tuning of downstream vision-LLMs, and its evaluation emphasizes low-light and safety-critical tasks on both simulated and real datasets (Kirchner et al., 9 Sep 2025).

1. Definition and problem setting

DepthVision addresses a specific robotics failure mode: Vision-LLMs depend on RGB input, but standard cameras often fail in challenging environments such as nighttime, glare, occlusion, or sensor failure. LiDAR remains geometrically robust under these conditions, yet it is rarely used directly in VLMs because LiDAR lacks color and semantic richness and because large-scale LiDAR-text paired corpora are scarce. DepthVision therefore reformulates multimodal robustness as an image-domain adaptation problem: rather than retraining a VLM on LiDAR, it generates a VLM-compatible RGB-like representation from LiDAR and fuses that representation with the camera stream (Kirchner et al., 9 Sep 2025).

This places DepthVision within a broader class of systems that compensate for failure modes of single-cue perception. Related work shows that monocular depth cues can be enhanced by manipulating shading, contrast, and motion parallax on conventional displays, as in the Joint Shading/Contrast Model and Motion Parallax framework of Ryu, Yoo, and Akhavan (Ryu et al., 17 May 2026). Other work demonstrates that monocular, binocular, and multi-view estimators can all be misled by 3D visual illusions, motivating fusion schemes that use semantic reasoning to decide which depth cue to trust (Yao et al., 19 May 2025). In that context, DepthVision is distinguished by treating LiDAR-guided RGB synthesis as the compatibility layer between robust sensing and frozen language-conditioned perception (Kirchner et al., 9 Sep 2025).

2. LiDAR-to-RGB synthesis pipeline

DepthVision begins with LiDAR preprocessing. Three-dimensional LiDAR points $$\mathbf{X}_{\text{LiDAR}}^i = [x^i, y^i, z^i, 1]^\mathsf{T}$$ are transformed into the camera frame by

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$

with $$\mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} = \begin{bmatrix} \mathbf{R} & \mathbf{t} \ \mathbf{0}^\mathsf{T} & 1 \end{bmatrix}$$, and then projected onto the image plane using the camera intrinsic matrix $\mathbf{K}$. Only points in front of the camera and within image boundaries are retained, and nearest-neighbor interpolation densifies the sparse projection to form the single-channel GAN input (Kirchner et al., 9 Sep 2025).

The image synthesis core is a conditional GAN. The generator is a U-Net encoder-decoder with skip connections, inspired by pix2pix, operating on a $512 \times 512$ LiDAR image and producing an RGB image of the same size. The discriminator is a PatchGAN that judges the realism of local $70 \times 70$ patches, which encourages high-frequency textural plausibility. A separate refiner, implemented as a three-layer convolutional residual module, further improves the generated image. At each of three iterations, the refiner receives the current RGB estimate and the LiDAR projection and predicts a residual: $$\mathbf{I}_{\text{refined}} = \mathbf{I}_{\text{input}} + \text{Refiner}(\mathbf{I}_{\text{input}}, \mathbf{I}_{\text{LiDAR}}).$$ Training uses a combination of adversarial loss and $L_1$ reconstruction loss,

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{GAN}} + \lambda \| \mathbf{I}_{\text{RGB,synth}} - \mathbf{I}_{\text{RGB,real}} \|_1,$$

with $\lambda = 100$ in practice (Kirchner et al., 9 Sep 2025).

The design objective is not metric depth recovery per se, but photorealistic modality translation. That makes DepthVision structurally different from methods such as EM3D, which fuse diffraction- and polarization-based depth cues to reconstruct accurate 3D point clouds for extended scenes (Shen et al., 11 Feb 2025), or ClearDepth, which performs stereo depth recovery of transparent objects with a vision transformer-based architecture and a structural feature post-fusion module (Bai et al., 2024). DepthVision instead converts robust geometry into a representation that existing VLMs can ingest without architectural modification (Kirchner et al., 9 Sep 2025).

3. Luminance-Aware Modality Adaptation

After synthesis, DepthVision applies LAMA to fuse the generated RGB image with the real camera image according to illumination. The grayscale luminance is computed with Rec. 709 weights,

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$0

In full fusion, the mean luminance

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$1

selects between the two modalities. If $$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$2, the system uses the GAN image; if $$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$3, it uses the real RGB image; and otherwise it linearly blends them with

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$4

DepthVision also defines a pixelwise fusion rule with

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$5

and

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$6

The thresholds used in practice are $$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$7 and $$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$8 (Kirchner et al., 9 Sep 2025).

The reported behavior of the two fusion modes is task-dependent. Full fusion is generally more robust, especially for safety-critical tasks, whereas pixelwise fusion excels at fine-grained local detail such as object classification in challenging lighting but can cause minor accuracy drops in well-lit scenes, likely because the fused image statistics are less familiar to frozen VLMs (Kirchner et al., 9 Sep 2025). This luminance-gated fusion strategy is conceptually adjacent to other cue-selection mechanisms in recent literature. In the 3D visual illusion setting, a VLM-based fusion module is used to generate region-wise confidence maps and select between monocular and binocular depth cues when one cue is unreliable (Yao et al., 19 May 2025). DepthVision applies a simpler gating signal—scene luminance—but for the same general purpose of selective trust under sensor degradation (Kirchner et al., 9 Sep 2025).

4. Integration with frozen vision-LLMs

DepthVision is explicitly designed as a preprocessing layer for off-the-shelf VLMs. The fused RGB image is fed as standard visual input to frozen models such as LLaVA and Qwen2-VL, without retraining. The image is patch-tokenized as in a Vision Transformer, producing

$$\mathbf{X}_c^i = \mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} \cdot \mathbf{X}_{\text{LiDAR}}^i,$$9

visual tokens for patch size $$\mathbf{T}_{\text{cam} \leftarrow \text{LiDAR}} = \begin{bmatrix} \mathbf{R} & \mathbf{t} \ \mathbf{0}^\mathsf{T} & 1 \end{bmatrix}$$0, which are then concatenated with tokenized text queries before standard multimodal reasoning proceeds (Kirchner et al., 9 Sep 2025).

The absence of downstream fine-tuning is central to the framework. It avoids the data bottleneck that would arise from directly adapting large VLMs to LiDAR, and it preserves compatibility with existing pretrained systems. This design contrasts with approaches such as Vanishing Depth, which extend pretrained RGB encoders by incorporating metric depth into feature embeddings through Positional Depth Encoding and self-supervised alignment for RGBD tasks (Koch et al., 25 Mar 2025). DepthVision does not attempt to change the internal encoder geometry of the VLM; it instead changes the input image so that the frozen model can operate in degraded environments (Kirchner et al., 9 Sep 2025).

A plausible implication is that DepthVision is best understood as a modality-compatibility framework rather than a depth-estimation algorithm. The input sensor carrying the reliable signal is LiDAR, but the output consumed by the reasoning stack is still image-like. That design choice makes the framework particularly relevant to robotics pipelines built around camera-native foundation models (Kirchner et al., 9 Sep 2025).

5. Empirical evaluation and performance characteristics

DepthVision was evaluated on CARLA simulation and the nuScenes dataset, both with calibrated RGB and LiDAR under varying day and night conditions. The tasks include safety-critical object existence queries, object counting, and object classification, with Top-1 accuracy as the principal metric (Kirchner et al., 9 Sep 2025).

The reported gains are concentrated in low-light conditions. On the existence task at night, Qwen2-VL-7B improves from 53.3% with camera-only input to 66.7% with camera plus LiDAR using full fusion, a gain of 13.4%. Under the same condition, LLaVA-1.6-Mistral-7B improves from 57.8% to 73.3%, a gain of 15.5% (Kirchner et al., 9 Sep 2025). Other tasks, including counting and object classification, also improve during dark scenes, with less or no degradation during the day. The qualitative example emphasized in the paper is a nighttime scene in which a vehicle ahead is invisible in the RGB image but becomes visible after LiDAR-to-RGB synthesis, enabling the VLM to make the correct safety-critical decision (Kirchner et al., 9 Sep 2025).

The main empirical trend is that performance gains scale inversely with luminance: the darker the scene, the greater the benefit of DepthVision. At high luminance, blending is reduced so that standard VLM performance is not hindered (Kirchner et al., 9 Sep 2025). This result is consistent with other perceptual findings in adjacent domains. For instance, work on foveated rendering reports that stereoscopic acuity remains unaffected, or even improves, under high levels of peripheral blur, implying that aggressive image-quality modulation does not necessarily degrade depth-sensitive tasks when the relevant cue remains intact (Kergaßner et al., 28 Jan 2025). DepthVision extends the same general logic from display rendering to robotic reasoning: representation quality can be adapted if the task-relevant information is preserved or restored (Kirchner et al., 9 Sep 2025).

6. Relation to surrounding depth and perception research

The name “DepthVision” is used specifically for the LiDAR-to-RGB synthesis framework described above, but the underlying problem intersects multiple research traditions. One tradition studies how depth is perceived or displayed. Hamilton and Polhemus, for example, argue that affine distance is the natural distance for stereoscopic visualization in curved spacetime and note that binocular vision can fail when wavefronts become ellipsoidal, motivating alternative cue constructions for relativistic visualization (Hamilton et al., 2010). TomoReal targets near-eye tomographic display, supporting quasi-continuous accommodation, omni-directional motion parallax, preserved resolution, and full frame rate through synchronized focus-tunable optics and a fast spatially adjustable backlight (Lee et al., 2018). JSM enhances monocular depth perception on conventional 2D displays by modulating shading, texture contrast, and motion parallax (Ryu et al., 17 May 2026).

A second tradition concerns robust geometric sensing under difficult materials or weak cues. ClearDepth addresses stereo depth recovery for transparent objects through a cascaded Vision Transformer backbone, structural feature post-fusion, and a physically realistic Sim2Real pipeline (Bai et al., 2024). EM3D fuses diffraction-based depth from defocus with polarization-based shape cues to produce a million-pixel and accurate 3D point cloud for extended scenes including low-texture, reflective, and nearly transparent objects (Shen et al., 11 Feb 2025). Hybrid systems combining binocular stereo and monocular structured light improve weak-texture indoor depth by injecting guided disparity hints into the stereo cost volume (Xu et al., 2022). LightDepth exploits illumination decline for single-view self-supervised depth in endoscopic settings where camera and light source are co-located (Rodríguez-Puigvert et al., 2023).

DepthVision differs from these lines of work in objective and output space. It is not primarily a method for reconstructing dense geometry, nor a display technology for human depth perception. Instead, it is a robustness layer for downstream semantic reasoning. Its contribution lies in the claim that synthesizing a familiar modality from a robust sensor can improve performance of frozen VLMs in adverse conditions, especially on safety-critical tasks (Kirchner et al., 9 Sep 2025).

7. Limitations and prospective directions

DepthVision is explicitly presented as complementary rather than universal. The paper notes that some tasks still require real color or fine details, such as reading traffic signs or interpreting subtle human cues, so the framework is not a complete substitute for high-fidelity vision (Kirchner et al., 9 Sep 2025). The adaptive fusion policy also reflects this limitation: at high luminance, the system reduces blending so that the original camera signal dominates (Kirchner et al., 9 Sep 2025).

The broader literature suggests two pressure points for future development. First, semantic cue selection is increasingly important when raw geometric or appearance cues are systematically misleading, as shown by recent results on 3D visual illusions (Yao et al., 19 May 2025). Second, generalized encoders that natively incorporate metric depth, such as Vanishing Depth, indicate another route to robustness that operates inside the feature space rather than in the image space (Koch et al., 25 Mar 2025). This suggests that future systems may combine LiDAR-guided synthesis, confidence-aware cue arbitration, and depth-adapted encoders in a single stack. What DepthVision establishes is the viability of a minimal intervention strategy: robust robot reasoning can be improved substantially in low light by synthesizing RGB from LiDAR, adaptively fusing it with camera data, and leaving the downstream VLM frozen (Kirchner et al., 9 Sep 2025).