VALDM: View-Align Latent Diffusion Model

• The paper introduces VALDM, a latent diffusion framework that integrates MVAM to enforce cross-view geometric consistency during the denoising process.
• It employs a DDIM formulation in a learned latent space, combining multi-view encoding, spatial transformation via homographies, and CLIP-based conditioning for robust viewpoint-invariant representations.
• Applications in visual anomaly detection and BEV-to-street synthesis demonstrate significant improvements (e.g., AUROC gains and refined FID scores) over traditional methods.

The View-Align Latent Diffusion Model (VALDM) is a specialized latent diffusion framework engineered to ensure geometric alignment and viewpoint-invariance in multi-view vision tasks. VALDM has been applied prominently in two domains: multi-view visual anomaly detection, where it underpins the VSAD (ViewSense-AD) framework (Chen et al., 24 Nov 2025), and conditional image generation from spatial layouts, primarily in bird’s-eye to street-view synthesis pipelines (Xu et al., 2 Sep 2024). This article provides a comprehensive exposition of VALDM’s theoretical foundations, architectural design, and empirical performance.

1. Foundational Principles and Motivation

VALDM addresses challenges arising when visual systems require consistency across multiple input viewpoints or scene representations. In unsupervised multi-view anomaly detection, differing camera poses introduce appearance variation, causing conventional per-view detectors to yield inconsistent features and high false-positive rates. Similarly, in generative settings—such as transforming a bird’s-eye view (BEV) map into coherent street-view images—ensuring spatial and style alignment across outputs is non-trivial. VALDM enforces cross-view geometric consistency by integrating explicit alignment mechanisms into the latent diffusion process. This progressive alignment enables robust, viewpoint-invariant representations that are essential for both discriminative and generative tasks.

2. Latent Diffusion Formulation and Multi-View Encoding

VALDM utilizes the DDIM (Denoising Diffusion Implicit Models) formulation in latent space as its core. For multi-view anomaly detection in VSAD, a batch of $M$ camera images is first encoded through a pretrained VAE encoder $\mathcal{E}_{\mathrm{VAE}}$ into a joint latent tensor $Z_0 \in \mathbb{R}^{M \times C \times h \times w}$: $$Z_0 = \mathcal{E}_{\mathrm{VAE}}(I), \quad I = \{I_m\}_{m=1}^M$$ The forward (noising) process at time $t$ is: $$Z_t = \sqrt{\bar\alpha_t}\, Z_0 + \sqrt{1-\bar\alpha_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$ Denoising is conducted by a U-Net decoder parameterizing $\hat\epsilon_\theta(Z_t, t)$, trained using an $\ell_2$ noise prediction loss: $$\mathcal{L}_d = \mathbb{E}_{Z_0, \epsilon, t} \|\epsilon - \hat\epsilon_\theta(Z_t, t)\|_2^2$$ In conditional generation (e.g., BEV $\rightarrow$ street view), the latent diffusion process operates in a learned latent space ($\mathbb{R}^{64}$ dimension as in Stable Diffusion) and admits conditioning through textual embeddings (via CLIP) and spatial hints such as segmentation maps or view-specific tokens (Xu et al., 2 Sep 2024).

3. Geometric Alignment: Multi-View Alignment Module (MVAM)

The distinctive mechanism in VALDM is the Multi-View Alignment Module (MVAM), which is introduced at each U-Net decoder block to enforce geometric consistency across camera viewpoints. Pre-computed homographies $H_{i\to j} \in \mathbb{R}^{3 \times 3}$ map pixel or patch-center coordinates $p_i=[u, v, 1]^\top$ from view $i$ to view $j$: $p_j = H_{i\to j}\, p_i$ MVAM implements a local $R \times R$ search window around $p_j$ for fine correspondence and encodes the offset via a 2D Fourier positional embedding $\gamma(\Delta p_j)$. Query/key/value (QKV) projections are then constructed: $$\begin{aligned} q_i &= W_q\, X_i(p_i) \ k_j &= W_k\bigl[X_j(p_j') + \gamma(\Delta p_j)\bigr] \ v_j &= W_v\bigl[X_j(p_j') + \gamma(\Delta p_j)\bigr] \end{aligned}$$ Attention weights $\alpha_j$ are computed over all neighbor views and candidate windows: $$\alpha_j = \frac{\exp(q_i^\top k_j / \sqrt{d})}{\sum_{j'\in \mathcal{N}(i)} \exp(q_i^\top k_{j'}/\sqrt{d})}$$ The aligned feature for $p_i$ is then: $$\widetilde{X}_i(p_i) = \sum_{j \in \mathcal{N}(i)} \alpha_j v_j$$ This procedure, performed at all spatial locations and decoder layers, yields a progressively aligned latent representation robust to viewpoint-induced variations (Chen et al., 24 Nov 2025).

4. Progressive Denoising, Feature Fusion, and Global Refinement

VALDM’s diffusion process is designed to operate over $T$ fixed timesteps (e.g., $T=50$–100), with early steps targeting coarse, semantic alignment and later steps refining high-frequency details. After each MVAM-aligned decoder block, features across views are concatenated along the batch dimension. A lightweight Fusion Refiner Module (FRM) is subsequently employed to enhance global feature consistency. The FRM consists of a small 3×3-conv network $f$ and a Squeeze-and-Excitation module $\mathcal{A}$. It produces, for each view $m$: $$\begin{aligned} U_m^{(l)} &= f(\widetilde Z_{t,m}^{(l)}) \ s_m^{(l)} &= \mathcal{A}(f(\widetilde Z_{t,m}^{(l)})) \ Z^{\prime(l)}_m &= U_m^{(l)} \odot s_m^{(l)} \ F^{(l)}_{m} &= \widetilde Z^{(l)}_{t,m} + Z^{\prime(l)}_m \end{aligned}$$ A regularization loss $\mathcal{L}_r$ encourages agreement between features of neighboring views: $$\mathcal{L}_r = \frac{1}{L} \sum_{l=1}^L \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \|F_i^{(l)}-F_j^{(l)}\|_2^2$$ Total objective: $$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_d + \lambda\, \mathcal{L}_r$$ (Chen et al., 24 Nov 2025).

5. Anomaly Detection and Memory Bank Mechanism

For visual anomaly detection, VALDM leverages the denoised features from selected decoder layers (typically $l \in \{3,4\}$ per ablation findings). During training, refined features from normal data are stored in a memory bank $\mathcal{M}^{(l)}$. At inference, for each spatial location $(u,v)$ in layer $l$, the minimal $\ell_2$ distance to $\mathcal{M}^{(l)}$ is: $$d^{(l)}(u,v) = \min_{m \in \mathcal{M}^{(l)}} \|F_q^{(l)}(u,v) - m\|_2$$ Weighted summation across levels forms a pixel-wise anomaly map: $$S_{\mathrm{pixel}}(u,v) = \sum_l w_l\, d^{(l)}(u,v)$$ Sample-level and view-level anomaly scores are computed as maxima over their respective regions. This approach delivers substantial robustness to viewpoint shifts and textured backgrounds (Chen et al., 24 Nov 2025).

6. Applications: Generative and Discriminative Multi-View Tasks

Visual Anomaly Detection (VSAD)

VSAD, centered on VALDM, was evaluated on RealIAD (151K images, 30 classes, 5 views) and MANTA (137K images, 38 classes, 5 views). VSAD achieves pixel/view/sample AUROCs of 98.3/91.7/94.8% (RealIAD) and 96.8/93.9/94.5% (MANTA), outperforming existing multi-view and embedding-only baselines by 1–4% absolute, especially under large geometric transformations. Ablation demonstrates that removing MVAM or FRM notably degrades S-AUROC by 8–10% and 1–1.6%, respectively (Chen et al., 24 Nov 2025).

BEV-to-Street-View Image Synthesis

VALDM has been extended to conditional generation pipelines, notably for BEV-to-street conversion (Xu et al., 2 Sep 2024). The pipeline consists of a Neural View Transformation module performing geometric projection and UNet-based refinement of segmentation from BEV-space to camera space, followed by a diffusion-based generative model conditioned on the refined segmentation and text prompts. View adaptation is achieved by LoRA-style low-rank adapters for each camera, with semantic alignment enforced by explicit segmentation controllers (ControlNet). Evaluated on nuScenes, VALDM yields FID of 48.65 (versus 25.54 for BEVGen) and improved mIoU for vehicles (17.70 vs. 5.89), while ablations reveal the critical impact of view adaptation and shape refinement components (Xu et al., 2 Sep 2024).

7. Architectural and Training Details

The baseline diffusion model in VALDM is Stable Diffusion v2, adapted with view-alignment, FRM, and view-specific conditioning. In the VSAD configuration, optimization is performed using AdamW with lr=$1 \times 10^{-4}$ for 80 epochs, leveraging 4 A6000 GPUs. MVAM uses a 7×7 local window for patch matching. The memory bank for anomaly detection comprises high-level decoder outputs, as inclusion of lower-level layers introduces alignment noise that degrades detection accuracy (Chen et al., 24 Nov 2025). In generative applications, training proceeds in two stages: shape refinement (pseudo-label supervision using SegFormer) and diffusion adaptation (LoRA modules per camera), with prompt-based disentanglement of viewpoint and scene attributes (Xu et al., 2 Sep 2024).

---

VALDM’s integration of homography-guided spatial alignment, progressive denoising, and cross-view feature fusion establishes it as a domain-general strategy for multi-view, viewpoint-invariant representation learning. Empirical results in both discriminative and generative contexts substantiate its effectiveness, particularly under conditions of severe pose variation and appearance ambiguity.