Luminance-Aware Modality Adaptation (LAMA)

• The paper introduces LAMA as a zero-parameter, non-learned fusion module that uses fixed luminance thresholds to gate between real and synthesized RGB imagery.
• LAMA employs both full and pixel-wise fusion modes, enabling global and local adaptation based on scene luminance conditions in DepthVision.
• Empirical evaluations show that full fusion improves safety-critical vision-language tasks in low-light environments with up to 15% absolute accuracy gains.

Searching arXiv for the specified paper and closely related background work to ground the article with citations. Luminance-Aware Modality Adaptation (LAMA) is a fixed fusion module introduced within DepthVision to improve vision-language understanding when camera imagery is unreliable because of low-light, glare, motion blur, or occlusion. In DepthVision, a conditional GAN with an integrated refiner synthesizes RGB imagery from sparse LiDAR point clouds, and LAMA then decides whether to pass the real RGB image, the synthesized RGB image, or a blend of both to a frozen Vision-LLM (VLM). Its defining characteristic is that this adaptation is driven by simple luminance cues computed from the incoming real RGB image, without any fine-tuning of the downstream VLM and without learning any parameters inside LAMA itself (Kirchner et al., 9 Sep 2025).

1. Functional role within DepthVision

LAMA is motivated by a specific failure mode of camera-centric VLM pipelines: real camera images can become unreliable under adverse illumination or motion conditions. Within DepthVision, it occupies the interface between the RGB/LiDAR-to-RGB generator and the frozen VLM, and its function is operational rather than representational. It receives the real RGB image and the GAN-produced RGB image, evaluates the luminance of the real RGB input, and then performs one of three actions: pass the real RGB image, pass the GAN output, or blend the two into a fused image for subsequent VLM inference (Kirchner et al., 9 Sep 2025).

This placement is significant because it makes LAMA a modality adaptation layer rather than a joint encoder. The module does not alter the VLM, does not retrain it, and does not introduce a new multimodal backbone. A plausible implication is that the method targets deployment compatibility: the VLM remains frozen while the input pathway is adapted to illumination conditions.

2. Architectural characteristics

The architecture of LAMA is explicitly lightweight. No heavy attention mechanisms or transformers are added. The module contains three elements only: a luminance estimator implemented as a fixed linear grayscale conversion, a small gating logic that compares mean luminance against two thresholds, and a blending operation that can be applied globally or pixel-wise (Kirchner et al., 9 Sep 2025).

The paper characterizes LAMA as a differentiable “gate + mixer” in the data path, but not a learned one. Its thresholds, denoted $L_{\text{low}}$ and $L_{\text{high}}$, are hand-set, and the remaining operations are straightforward arithmetic. This distinguishes LAMA from trainable cross-modal fusion mechanisms. A common misconception would be to treat it as a learned adapter analogous to feature-level fusion modules; in fact, the summary states that LAMA is a zero-parameter arithmetic module whose behavior is fully specified by luminance computation, threshold comparison, and mixing.

3. Luminance estimation and normalization

LAMA begins from a real RGB input image $$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$, with each channel normalized to $[0,1]$. Per-pixel luminance is computed using the Rec. 709 grayscale formula:

$$I_{\text{gray}}(u,v) = 0.2126 \cdot R(u,v) + 0.7152 \cdot G(u,v) + 0.0722 \cdot B(u,v).$$

For full-image fusion, LAMA computes the mean luminance

$$L_{\text{mean}} = \frac{1}{H \cdot W}\sum_{u=1}^{W}\sum_{v=1}^{H} I_{\text{gray}}(u,v).$$

The synthesized image from the generator is denoted $I_{\text{GAN}}$, and it is also represented in $[0,1]$ (Kirchner et al., 9 Sep 2025).

This design gives luminance a direct gating role. Rather than estimating confidence through a learned uncertainty head or a semantic consistency signal, LAMA uses a deterministic photometric proxy. This suggests an intentionally narrow operational hypothesis: when scene luminance falls sufficiently low, the LiDAR-guided synthesized RGB view becomes a better input substrate for the frozen VLM than the original camera image.

4. Fusion modes and decision rules

LAMA supports two fusion modes: full (global) fusion and pixel-wise (local) fusion. In full fusion, $L_{\text{mean}}$ is compared to the fixed thresholds $L_{\text{low}}$ and $L_{\text{high}}$0, which are given in practice as $L_{\text{high}}$1 and $L_{\text{high}}$2. If $L_{\text{high}}$3, the module uses $L_{\text{high}}$4; if $L_{\text{high}}$5, it uses $L_{\text{high}}$6; otherwise it interpolates between them with

$L_{\text{high}}$7

and

$L_{\text{high}}$8

In pixel-wise fusion, the global scalar $L_{\text{high}}$9 is replaced by a per-pixel weight map,

$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$0

and the fused image is computed as

$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$1

These equations define the entire forward behavior of the module (Kirchner et al., 9 Sep 2025).

Fusion mode	Gating variable	Output rule
Full (global) fusion	$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$2	Selects real RGB, GAN RGB, or a global blend
Pixel-wise (local) fusion	$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$3 from $$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$4	Blends RGB and GAN RGB at each pixel

The distinction between the two modes is operationally consequential. Full fusion makes one scene-level decision from average luminance, whereas pixel-wise fusion permits spatially varying substitution. A plausible implication is that full fusion preserves global image consistency more effectively, while pixel-wise fusion increases local adaptivity at the risk of compositional artifacts or inconsistent spatial evidence for the downstream VLM.

5. Relation to training objectives

LAMA itself is not trained. All optimization in DepthVision occurs upstream in the LiDAR-to-RGB generator and refiner. The generator is trained with a standard pix2pix-style objective comprising an adversarial loss with PatchGAN and an $$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$5 reconstruction loss:

$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$6

$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$7

and the total generator loss is

$$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$8

The summary states explicitly that no additional consistency or perceptual losses are introduced for LAMA itself (Kirchner et al., 9 Sep 2025).

This separation is methodologically important. LAMA does not participate in training as a learnable fusion head, nor does it require an auxiliary supervision signal for reliability estimation. The module therefore belongs to the inference-time adaptation layer of the system, while synthesis quality is delegated entirely to the GAN and refiner. A common misunderstanding would be to attribute downstream robustness gains to a jointly optimized fusion mechanism; the paper instead attributes learning to the generative front end and adaptation to a fixed arithmetic gate.

6. Empirical behavior and technical implications

The empirical evaluation uses two frozen VLMs, Qwen2-VL-7B-Instruct and LLaVA-1.6-Mistral-7B, on a nuScenes-QA style benchmark consisting of 100 real-world front-view scenes, split into 50 day and 50 night scenes, with tasks including safety-critical existence, counting, and object classification. On the night safety-critical task, Qwen2 records 53.3 Top-1 accuracy for the camera-only baseline, 66.7 with LAMA full fusion, and 44.4 with LAMA pixelwise. For LLaVA on the same task, the corresponding values are 57.8, 73.3, and 60.0. The summary further states that full fusion consistently outperforms camera-only at night across all tasks, with up to 15 % absolute gains in the safety-critical scenario, while pixelwise fusion can further boost object-level tasks in very dark patches but may incur occasional daytime degradations (Kirchner et al., 9 Sep 2025).

These results establish two technical points. First, LAMA’s gains are concentrated in illumination regimes where real RGB is least reliable, rather than being uniformly distributed across conditions. Second, the relative weakness of pixel-wise fusion on safety-critical night evaluation indicates that greater local flexibility does not necessarily translate into better end-task reasoning for frozen VLMs. Figure 1 is described as showing that the relative improvement of full fusion increases as mean scene luminance drops, which supports the interpretation that LAMA redirects reliance toward the LiDAR-synthesized view precisely when the camera image becomes least dependable. This also addresses a likely misconception that local fusion is inherently more expressive and therefore superior: the reported results show that global fusion is the more robust default under the evaluated conditions.

7. Interpretation, scope, and limitations

LAMA is described as a simple, interpretable, zero-parameter module that uses a Rec. 709 grayscale cue and two hand-set luminance thresholds to adaptively gate between real and synthetic RGB inputs. Its principal scope is robust downstream vision-language reasoning under challenging illumination, and it is explicitly designed to preserve compatibility with frozen VLMs while compensating for sensor degradation such as darkness or motion blur (Kirchner et al., 9 Sep 2025).

Its limitations follow directly from that design. Because the gating policy depends on luminance alone, it is not a general uncertainty estimator for all forms of visual corruption. Because the thresholds are hand-set rather than learned, behavior depends on fixed operating points at $$I_{\text{RGB}} \in \mathbb{R}^{H \times W \times 3}$$9 and $[0,1]$0. Because pixel-wise fusion may produce daytime degradations and can underperform under bad local blending, the more adaptive mode is not uniformly preferable. These observations do not negate the module’s utility; rather, they delimit the conditions under which its arithmetic simplicity is advantageous. In that sense, LAMA exemplifies a class of modality-adaptation mechanisms that favor transparent decision rules and frozen-backbone compatibility over learned fusion complexity.