Lighting-Based Retrieval Advances

• Lighting-based retrieval is a methodology that organizes and matches data (e.g., images, irradiance maps, and text) by leveraging lighting properties and unified feature embeddings.
• Recent systems like UniLight implement modality-specific encoders and contrastive learning, achieving improved Recall@1 and robust cross-modal matching under severe illumination changes.
• Photometric normalization and active lighting strategies optimize scene understanding, delivering enhanced performance in both line-of-sight and non-line-of-sight imaging applications.

Lighting-based retrieval refers to the set of methodologies and systems for retrieving, organizing, or matching data—typically images, environment maps, irradiance distributions, or textual descriptions—according to lighting properties. These systems enable effective cross-modal search, light-aware image retrieval under severe illumination change, and optimized scene understanding in both line-of-sight (LOS) and non-line-of-sight (NLOS) settings. Recent advances unify multiple data modalities within shared embedding spaces, leverage contrastive learning, exploit radiometric priors, and incorporate learnable or physics-informed normalization to enable robustness across lighting conditions.

1. Unified Cross-Modal Lighting Retrieval

Lighting-based retrieval has historically faced the challenge of incompatible modality representations. The UniLight framework (Zhang et al., 3 Dec 2025) addresses this by constructing a unified latent embedding for four key modalities: environment maps, photographs, irradiance maps, and textual lighting descriptions. Each sample in the database comprises aligned versions in all four modalities. Modality-specific encoders—based on either a DINOv2-B ViT backbone for image/irradiance/env-maps or a fine-tuned Qwen3 embedding model for text—produce sequences of tokens that are abstracted into an 8×512-dimensional latent. Retrieval over this joint space employs cosine similarity as the matching metric.

Cross-modal queries (e.g., retrieving text descriptions that match a lighting environment from an image) proceed by computing the latent embedding of the query and ranking all database items in the target modality according to cosine similarity. Recall@K, mean reciprocal rank (MRR), and median rank are principal evaluation metrics, with UniLight achieving Recall@1 ≈ 24.9%, Recall@5 ≈ 49.0%, MRR ≈ 0.367, and median rank ≈ 9.8 across four modalities. This performance far exceeds baseline image-text retrieval models such as CLIP ViT-B/32 (R@1 = 2.6%) or Qwen3-VL 2B (R@1 = 8.9%) under lighting-centric matching (Zhang et al., 3 Dec 2025).

2. Modality-Specific Encoder Designs

All image-based modalities in UniLight are encoded using a DINOv2-B Vision Transformer. For environment maps, three separate channels are concatenated for each pixel: (1) an LDR tone-mapped RGB, (2) a logarithmic HDR encoding transformed as log($I_{hdr}+1$)/log($I_{max}$)), and (3) per-pixel $(x, y, z)$ directional encodings representing latitude-longitude positions. During training, random dropping of the log-HDR channel increases robustness to low-dynamic-range inputs. Perspective photographs and irradiance maps are fed directly as 512×512 RGB images. The text modality relies on the Qwen3 Embedding model, which is fine-tuned specifically for lighting semantics.

A shared multi-modal summary module processes modality-specific features using learnable queries (eight per input), multi-head attention, LayerNorm, and a linear projection to the canonical $8\times512$ latent. An auxiliary SH (spherical harmonics) head, implemented as an MLP, predicts SH coefficients up to degree $l=3$ for each color channel, directly from the embedding.

3. Contrastive and Spherical Harmonics Supervision

The UniLight system is trained end-to-end using an InfoNCE-style contrastive objective. For each modality pair $(m, n)$, the contrastive loss penalizes mismatches between embeddings of matching pairs and encourages separation from batch negatives:

$$L_C^{(m\rightarrow n)} = - \sum_{i=1}^B \log \frac{\exp(\text{sim}(E_i^m, E_i^n)/\tau)}{\sum_{j=1}^B \exp(\text{sim}(E_i^m, E_j^n)/\tau)}$$

with $\text{sim}(x,y)=x^Ty/(||x||\,||y||)$ and $\tau$ a temperature parameter.

A secondary objective, the SH prediction loss, compels the embedding to predict ground-truth degree-3 spherical harmonic coefficients derived from the HDR environment map of each sample:

$$L_{SH} = \sum_{i=1}^B \| SH_{pred,i} - SH_{GT,i} \|_2^2$$

The total loss is a weighted combination: $I_{max}$0, with $I_{max}$1. SH prediction enforces that the embedding is sensitive to both the overall luminance/color temperature and principal directions of illumination—properties essential for disentangling lighting in cross-modal retrieval tasks (Zhang et al., 3 Dec 2025).

4. Photometric Normalization in Image-Based Retrieval

Photometric normalization techniques have been established for robust image retrieval under varying illumination (such as day-night transitions). Methods include both hand-crafted approaches (global histogram equalization, contrast-limited adaptive histogram equalization (CLAHE), and gamma correction) and learnable strategies based on U-Net architectures. All these operate by preprocessing the image to produce an illumination-normalized version that is then fed to a standard or fine-tuned CNN descriptor extractor.

Experimental evaluations show that both CLAHE and a U-Net-based normalization network, especially when pre-trained on paired multi-exposure data and fine-tuned on day-night landmark datasets, improve mean average precision (mAP) by 7–8 percentage points on illumination-challenging retrieval benchmarks such as Tokyo 24/7, while maintaining performance on daylight retrieval (e.g., Oxford/Paris) (Jenicek et al., 2019).

5. Lighting-Aware Approaches in NLOS Imaging and Active Illumination

Adaptive lighting extends the concept of lighting-based retrieval to NLOS scene understanding, where localized illumination and detection are used for inferring object location or class in occluded regions. An adaptive algorithm distributes projector power across segmented LOS patches, maximizing the end-to-end radiometric throughput to the hidden region—computed via radiosity theory. The contribution of each patch, $I_{max}$2, is used to solve an optimization for power allocation under total energy and intensity constraints.

Data-driven CNN architectures downstream from this physics-inspired lighting control ingest the resulting image and predict either the 3D centroid of the hidden object (with a mean-squared error of $I_{max}$3 cm) or its class (87.1% identification accuracy over four object types) (Chandran et al., 2019). Concentrating power on a small number of key LOS patches is empirically shown to produce up to twice the NLOS signal as uniform lighting strategies.

6. Ablations and Sensitivity Analyses

Systematic ablation experiments in UniLight confirm that spherical harmonics supervision is essential: omitting the SH loss drops Recall@1 from 24.9% to 10.2%. Performance varies with the number of learnable query tokens, T, but 8 tokens suffice for near-optimal retrieval accuracy. SH supervision at degree 3 outperforms both lower (degree-1) and higher (degree-5) orders, suggesting optimal expressivity without overfitting (Zhang et al., 3 Dec 2025).

In photometric normalization methods, U-Net-based and CLAHE pre-processing both yield substantial gains under extreme illumination change. CLAHE combined with state-of-the-art embeddings achieves top mAP on Tokyo 24/7 (72.4%), outperforming baseline VGG GeM (62.0%) in the most challenging day-night matching cases (Jenicek et al., 2019).

7. Challenges and Future Directions

Despite advances in modality-unifying embeddings and normalization, challenges persist. For retrieval: (a) computational and memory overheads for normalization networks (U-Net), (b) limited diversity in training data (real-world, dynamic, multi-illumination scenarios), and (c) reliance on approximate overlap criteria in landmark-based datasets constrain generalization and scalability. For active illumination: (d) current models typically assume Lambertian surfaces and neglect higher-order or non-diffuse transport, and (e) learning-based joint optimization of illumination patterns remains to be fully exploited.

Ongoing work aims to extend end-to-end joint training of normalization and descriptor pipelines, adopt lightweight/faster normalization mechanisms, incorporate multi-modal/modal-invariant retrieval (infrared/depth), and integrate photometric invariance into self-supervised CNNs. For active lighting, hybrid optimization schemes that combine physical transport models with neural architectures may further improve NLOS scene inference (Zhang et al., 3 Dec 2025, Chandran et al., 2019, Jenicek et al., 2019).