DualDiff: Dual-Branch Diffusion Model

• DualDiff is a dual-branch diffusion model that separately conditions foreground and background using ControlNet modifications for precise scene synthesis in autonomous driving.
• It leverages Occupancy Ray Sampling to transform 3D occupancy information into rich 2D feature maps, effectively bridging geometric context with camera imagery.
• The framework incorporates Semantic Fusion Attention and a foreground-aware masked loss to enhance small-object fidelity, outperforming prior methods on key benchmarks.

^^^^2^^^^ refers to a class of dual-branch diffusion models designed for high-fidelity, controllable scene generation, with particular success in the context of autonomous driving perception. The DualDiff framework introduces architectural and algorithmic innovations enabling multi-modal, fine-grained control of both foreground and background content, leveraging rich geometric, semantic, and linguistic conditioning. The signature contributions include a dual-branch architecture based on ControlNet modifications to Stable Diffusion, Occupancy Ray Sampling for dense 3D scene conditioning, Semantic Fusion Attention for multi-modal feature integration, and a Foreground-Aware Masked loss tailored for detailed synthesis of small or distant objects. DualDiff and its video extension DualDiff+ set the state-of-the-art in several automated driving benchmarks in image and video generation, BEV segmentation, and 3D object detection (Li et al., 3 May 2025, Yang et al., 5 Mar 2025).

1. Dual-Branch Diffusion Model Architecture

DualDiff builds atop a frozen Stable Diffusion UNet, augmenting it with two parallel, lightweight ControlNet-style condition encoder branches:

• Background branch (denoted $\tau_\theta$): receives scene-layout and static background control.
• Foreground branch ($\mu_\theta$): handles object-level, dynamic foreground control.

At each reverse-diffusion timestep $t$, the main denoising UNet $\epsilon_\theta$ takes the noisy latent $z_t$, the timestep, and residual feature maps from both branches. These feature maps are derived from raw occupancy, semantic, and vectorized representations, subsequently aligned and fused. This dual-branch injection is implemented by cross-attention residuals into the UNet's layers, providing explicit, independent control over foreground and background (Li et al., 3 May 2025, Yang et al., 5 Mar 2025).

The general training objective is a foreground-aware masked mean squared error,

$$L(\theta) = \mathbb{E}_{z_t, \epsilon, c_\text{env}, v_b^*, v_f^*, t} \big\| \epsilon - \epsilon_\theta(z_t, t, c_\text{env}, \tau_\theta(v_b^*), \mu_\theta(v_f^*)) \big\|_2^2 \odot m,$$

where $c_\text{env}$ encodes the full numerical scene context; $v_b^*, v_f^*$ are semantically fused background/foreground feature maps; $m$ is the foreground-aware mask.

2. Occupancy Ray Sampling and Multi-Modal Condition Encoding

DualDiff introduces Occupancy Ray Sampling (ORS), which transforms a 3D occupancy grid $O \in \mathbb{R}^{H \times W \times D}$ into rich, camera-aligned 2D feature maps:

• For each image pixel $(u,v)$, a 3D ray is cast using camera intrinsics $K$, extrinsics $T$, and ego-pose $p_\text{ego}$.
• $N$ equidistant points are sampled along the ray: $$\hat s_\text{ego} = \{ p_\text{ego} + n \cdot r \mid n=1...N \}$$.
• Trilinear interpolation on $O$ yields dense volumetric features $v$ that capture both semantic class and geometry.

ORS provides a unified representation that bridges the gap between 3D occupancy, spatial layout, and 2D camera imagery essential for scene understanding and control (Li et al., 3 May 2025, Yang et al., 5 Mar 2025).

In parallel, DualDiff integrates four sets of vectorized/numerical scene features:

• Foreground object boxes: classes and 3D corners, embedded via CLIP and Fourier features.
• Vector map elements: lane and map geometry, similarly encoded.
• Camera pose: embedded by Fourier features and MLP/transformers.
• Text prompts: processed by CLIP and a learned projector.

All embeddings are concatenated into $c_\text{env}$ for global cross-modal context.

3. Semantic Fusion Attention Mechanism

The Semantic Fusion Attention (SFA) module is a three-stage transformer-based mechanism that fuses ORS-derived camera-view features with spatial and semantic context:

1. Visual Self-Attention: refines ORS features through intra-modality attention.
2. Spatial Grounding via Gated Cross-Attention: fuses spatial priors ($c_\text{spatial}$, e.g., map/box embeddings) with a learnable scaling gate $\tanh(\gamma)$ initialized at zero.
3. Textual Deformable Attention: aligns the spatially grounded features with textual semantics, employing learned positional offsets for greater flexibility in cross-modal alignment.

The result is a per-pixel feature map $v^*$ that integrates geometry, semantics, and layout, forming the input to both $\tau_\theta$ and $\mu_\theta$. This fusion is key for resolving complex scene attributes that require cross-modal context (Li et al., 3 May 2025, Yang et al., 5 Mar 2025).

4. Foreground-Aware Masked (FGM) Loss

DualDiff employs a Foreground-Aware Mask (FGM) in its denoising objective to focus training gradients on small, fine-grained, or distant objects commonly underrepresented in generative losses:

$$m_{ij} = \begin{cases} 2 - \frac{a_{ij}}{U \cdot V} & \text{if } (i,j) \in \text{foreground} \ 1 & \text{otherwise} \end{cases}$$

where $a_{ij}$ is the area of the foreground object’s projection at pixel $(i,j)$, and $U \times V$ is the image size. As a result, denoising error on pixels covering small objects is upweighted (as $m \rightarrow 2$), improving the fidelity of synthesized tiny or distant objects.

5. Training and Inference Procedures

The DualDiff model is trained and evaluated through the following protocol:

• Initialization: The Stable Diffusion UNet backbone is frozen; background and foreground branches are initialized from segmentation-pretrained ControlNet modules.
• Stage 1: Separate training of $\tau_\theta$ and $\mu_\theta$ for 80 epochs (learning rate $8 \times 10^{-5}$).
• Stage 2: Joint fine-tuning of both branches for 30 further epochs.
• Inference: A UniPC sampler is employed with 20 steps and a classifier-free guidance scale of 2; outputs are generated at $224 \times 400$ (nuScenes) or $320 \times 480$ (Waymo) resolution (Li et al., 3 May 2025).

6. Empirical Performance and Comparative Evaluation

On nuScenes and Waymo benchmarks, DualDiff achieves substantial advances over prior methods:

Metric	Baseline (Best Prior)	DualDiff	Abs. Gain
FID ↓ (nuScenes, 224×400)	16.20 (MagicDrive)	10.99	–5.21
BEV Segmentation Road mIoU ↑ (nuScenes)	61.26	62.75	+1.49
BEV Vehicle mIoU ↑ (nuScenes)	27.13	30.22	+3.09
3D Detection mAP ↑ (nuScenes, PV-RCNN)	12.30	13.99	+1.69
FID ↓ (Waymo)	17.16	11.45	–5.71

Ablation studies confirm the incremental benefits of ORS (+2.9 mIoU), added numerical representations (improved small object recall), SFA (–0.7 FID), and FGM loss (+1.0 vehicle mIoU). When used for synthetic data generation, incorporation of DualDiff outputs in downstream detection models yields notable mAP and NDS improvements (Li et al., 3 May 2025).

7. Extensions and Related Approaches

DualDiff+ extends the original framework to video, incorporating temporal modules (including spatio-temporal and temporal attention layers), and introduces Reward-Guided Diffusion (RGD) for enhanced video consistency and semantic alignment:

• RGD optimizes the diffusion process by maximizing a differentiable reward based on the distance between generated and reference video features in a frozen I3D network, fine-tuned via LoRA adapters.

Empirical results in image and video generation, as well as BEV tasks, consistently confirm DualDiff’s superiority over prior art in fidelity, segmentation, and detection (Yang et al., 5 Mar 2025).

This summary refers exclusively to the dual-branch diffusion models for autonomous driving and video generation introduced in "DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion" (Li et al., 3 May 2025) and "DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance" (Yang et al., 5 Mar 2025), and does not cover other unrelated uses of the "DualDiff" name in the literature.