---
title: 'LiNO-UniPS: Unified Photometric Stereo'
url: https://www.emergentmind.com/topics/lino-unips
type: topic
---

# LiNO-UniPS: Unified Photometric Stereo

LiNO-UniPS denotes "Light of Normals: Unified Feature Representation for Universal Photometric Stereo," a deep learning framework targeting robust surface normal estimation from multi-illumination image sets without lighting, reflectance, or segmentation assumptions. It advances the universal photometric stereo (UPS) paradigm by unifying lighting–invariant feature encoding, detail-rich geometric recovery, and multi-domain learning in a single model architecture, validated on diverse benchmarks and synthetic datasets [2506.18882].

## 1. Universal Photometric Stereo: Formulation and Challenges

Universal photometric stereo seeks to estimate the per-pixel normal map $N \in \mathbb R^{H\times W\times 3}$ of an object given a fixed-view collection $\{I_f \in \mathbb R^{H\times W\times3}\}_{f=1}^F$ of images under unknown and potentially complex lighting, without presuming known light directions or simple reflectance. The underlying image-formation process is modeled as

$$
I_f(p) = \mathcal F\big(n(p), L_f, m(p)\big)
$$

where $p=(x,y)$ is the image pixel, $n(p)$ is the surface normal, $L_f$ aggregates lighting parameters for the $f$-th view (environment, point, area), and $m(p)$ describes material/BRDF. Unlike classical calibrated or Lambertian PS, both $L_f$ and $m(\cdot)$ are unknown and may include non-Lambertian and spatially-varying characteristics.

The practical goal is to learn a feed-forward mapping $\{I_f\}_{f=1}^F \to N$ robust across lighting and reflectance ambiguity. The main technical obstacles are: (1) disentangling illumination from surface normal features where intensity variation is ambiguous; and (2) preserving high-frequency detail in surface geometry when image cues are intricate or degraded by shadow, interreflection, and nonideal lighting [2506.18882].

## 2. Lighting-Invariant, Detail-Rich Feature Encoding

LiNO-UniPS introduces a multi-component encoder achieving both lighting invariance and fine geometric fidelity through three primary innovations:

- **Light-Register Tokens:** Three learnable tokens ($x_{\rm hdri}, x_{\rm point}, x_{\rm area}\in\mathbb R^D$) are prepended to every per-image token stream, acting as attention gatherers for environment (HDRI), point, and area lighting cues across all frames. Self-attention aggregates global illumination context through these tokens.

- **Interleaved Multi-Image Attention:** After ViT-style patch embedding, tokens are processed through four cascaded attention blocks (Frame, LightAxis, Global, LightAxis), alternating between intra-image, inter-image, and global contextualization. This enables information flow that disentangles lighting effects from geometry and fosters global spatial awareness.

- **Wavelet-Aware Down/Up Sampling:** To maintain high-frequency geometry, each $I_f$ is decomposed into a bilinearly downsampled stream ($I^d_f$) and four discrete Haar-type wavelet bands ($I^{ll}_f, I^{lh}_f, I^{hl}_f, I^{hh}_f$), which are separately tokenized and jointly encoded. A WaveUpSampler later performs inverse DWT followed by summation and smoothing to reconstruct the full-detail encoder output $F_{\rm enc}$.

The fused encoder representation $F_{\rm enc}\in\mathbb R^{F\times H\times W\times C}$ is explicitly designed to be invariant to lighting changes and rich in normal-sensitive, high-spatial-frequency content.

## 3. Network Architecture and Processing Pipeline

LiNO-UniPS adopts an encoder–decoder paradigm, integrating specialized stages for illumination-context integration and gradient-preserving detail recovery:

- **Input Preprocessing:** $F$ images are decomposed into low-pass and high-frequency wavelet bands, each receiving three light tokens.

- **Patch Embedding & Transformer Backbone:** Each stream is patch-embedded (patch size $P=8$, $D=384$ embedding), producing tokens processed by a DINOv2-initialized ViT backbone.

- **Cascaded Contextual Attention:** Four interleaved attention blocks reinforce lighting–normal decoupling and spatial detail aggregation.

- **Multi-Scale Feature Fusion:** A DPT-based fusion module forms a four-level spatial pyramid, fused via residual convolution blocks to ensure feature consistency and spatial resolution.

- **Wavelet-Based Detail Synthesis:** The WaveUpSampler upsamples and reconstructs spatial features, using inverse DWT and smoothing for final $F_{\rm enc}$.

- **Pixel-Sampling Decoder:** $m$ pixel locations per scene are randomly sampled. For each pixel, corresponding $F^f_{\rm enc}(x_i)$ and a high-dim projection of $I_f(x_i)$ form the input to pooling-by-multihead-attention and frame/light-axis self-attention, followed by a 2-layer MLP to predict the normal $\tilde n(x_i)$. Full maps are assembled by interpolation.

This architecture allows selective attention to both pixel-level and scene-global information, yielding robust reconstruction in complex lighting and material regimes.

## 4. Supervision: Loss Functions and Training

The training objective combines several loss components targeting both lighting disentanglement and accurate normal regression:

- **Lighting Alignment Losses:** For synthetic data with ground-truth lighting, each lighting parameter (HDRI, point, area) is mapped to a $D$-dim vector via MLPs and encouraged to align with the learned light tokens via cosine similarity. Losses:
  $$
  \mathcal L_{\rm hdri} = 1 - \langle \ell_{\rm hdri}^h, x_{\rm hdri}^h \rangle,\;\; \text{and analogous for point/area}.
  $$

- **Normal Regression and Gradient Loss:** Predictions $\tilde N$ and their gradients $\tilde G = \nabla \tilde N$ are compared to ground truth via a confidence-weighted quadratic loss and explicit gradient loss:
  $$
  \mathcal L_{\rm conf} = \sum_p C(p)\|\tilde N(p) - N(p)\|^2,\;\;
  \mathcal L_{\rm grad} = \sum_p \|\tilde G(p) - G(p)\|^2,
  $$
  where $C = \exp(\|\tilde G\|)$.

- **Total Loss:**
  $$
  \mathcal L_{\rm total} = \lambda_1\mathcal L_{\rm hdri} + \lambda_2\mathcal L_{\rm point} + \lambda_3\mathcal L_{\rm area} + \lambda_4\mathcal L_{\rm conf} + \lambda_5\mathcal L_{\rm grad}.
  $$
  With $\lambda_4=1$ and others adaptively set to keep auxiliary terms at $\sim 0.1\times$ the main loss.

Training uses the PS-Verse synthetic set (100K rendered scenes over five complexity/material/illumination levels), AdamW optimization, and progressive fine-tuning from simple to texture-rich scenes.

## 5. Benchmark Performance and Empirical Analysis

LiNO-UniPS achieves state-of-the-art normal estimation across standard universal PS benchmarks:

**Ablation analyses** on PS-Verse show each model component (light tokens, global attention, loss terms, wavelets, normal-gradient loss) cumulatively reduces mean angular error (MAE, up to $-3.9^\circ$), with steady improvements in encoder feature SSIM/CSIM.

**Quantitative results:**
- **DiLiGenT:** LiNO-UniPS MAE $4.74^\circ$ (vs Uni MS-PS $4.97^\circ$, SDM-UniPS $5.80^\circ$, UniPS $14.70^\circ$), SOTA on most objects.
- **LUCES:** MAE $9.48^\circ$ (vs Uni MS-PS $11.10^\circ$).
- **DiLiGenT$^2$:** Error matrices uniformly lower than prior art.

**Feature–accuracy correlation:** Encoder feature CSIM/SSIM tightly predicts per-pixel normal accuracy, confirming the efficacy of the lighting-invariant, detail-rich representation.

**High-resolution, real-world validation:** 2K–4K, mask-free, complex-object scenes demonstrate superior fidelity to commercial 3D scanner outputs.

## 6. Strengths, Limitations, and Future Development

**Strengths:**
- Decoupling of lighting and geometric features using light tokens and global attention.
- Preservation of high-frequency detail by wavelet-based sampling and gradient-aware supervision.
- Strong cross-domain generalization (materials, lighting, sparse input counts; robust results for as few as $K=3$ lights).
- ECSIM/SSIM-based consistency of encoder features closely tracks output accuracy.

**Limitations and directions for extension:**
- Global attention stages pose high computational cost; sparser mechanisms may lower resource use.
- Occasional normal-flip ambiguity on near-planar regions in absence of explicit lighting cues.
- Current approach is single-view only; extension to multi-view could exploit geometric consistency.
- No explicit BRDF/material recovery; joint estimation with normals could enhance scene understanding.

## 7. Summary and Context

LiNO-UniPS represents a unification of illumination-invariant feature learning, detail-sensitive geometry representation, and scaled synthetic-supervised training for universal photometric stereo. The approach leverages light-register tokenization, interleaved attention, and wavelet-aware processing to overcome prevailing ambiguities in surface normal estimation under arbitrary lighting, reliably outperforming prior models on a broad spectrum of synthetic and real benchmarks [2506.18882]. The architecture and training strategies serve as a reference point for future research in photometric geometric learning and domain-agnostic scene reconstruction.

Source: https://www.emergentmind.com/topics/lino-unips