DiffSemanticFusion: Semantic Diffusion Fusion

• DiffSemanticFusion is a family of diffusion-based methods that integrate semantic cues into denoising and feature fusion, enabling multi-modal signal integration.
• It leverages conditional diffusion, multi-view feature aggregation, and reliability mechanisms to align semantic inputs from images, maps, and text.
• The framework is applied in autonomous driving, image restoration, 3D feature extraction, and segmentation, demonstrating improved performance and efficiency.

DiffSemanticFusion denotes a family of diffusion-based semantic fusion formulations in which semantic information is injected into a diffusion process, or harvested from it, to combine complementary signals across modalities, views, or representations. In the most specific sense, it names the autonomous-driving framework "DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion" (Sun et al., 3 Aug 2025). In a broader sense used across recent work, it refers to systems that fuse semantics within diffusion-based generation, denoising, feature distillation, or decision modules, including joint multimodal fusion and super-resolution, multi-view 3D feature aggregation, interactive retrieval, autonomous-driving scene generation, single-image cross-spectral fusion, and domain-generalized segmentation (Jie et al., 11 Sep 2025, Dutt et al., 2023, Zhang et al., 23 Mar 2026, Li et al., 3 May 2025, Zhang et al., 2 Feb 2026, Ji et al., 2024).

1. Scope and representative formulations

The literature uses DiffSemanticFusion in two closely related ways. One is as a proper method name for semantic raster BEV fusion with online HD map diffusion in trajectory prediction and planning (Sun et al., 3 Aug 2025). The other is as a descriptive label for architectures in which diffusion models are explicitly conditioned on semantics, or in which diffusion-derived semantic features are fused across time, modality, or view. This broader usage is explicit in the descriptions of FS-Diff, Diff3F, ADaFuSE, DualDiff, MagicFuse, DIFF, DSPFusion, and DiTFuse (Jie et al., 11 Sep 2025, Dutt et al., 2023, Zhang et al., 23 Mar 2026, Li et al., 3 May 2025, Zhang et al., 2 Feb 2026, Ji et al., 2024, Tang et al., 30 Mar 2025, Li et al., 8 Dec 2025).

A common misconception is that semantic guidance in diffusion fusion is synonymous with text prompting. The recent literature is broader. Semantics may come from clarity-aware image embeddings, diffusion U-Net cross-attention maps, segmentation heads, heterogeneous traffic graphs, vectorized maps, occupancy semantics, or per-agent dialogue context. Another misconception is that diffusion semantic fusion is restricted to image generation; the same idea is used for 3D correspondence, segmentation backbones, trajectory prediction, planning, and interactive retrieval (Jie et al., 11 Sep 2025, Dutt et al., 2023, Ji et al., 2024, Sun et al., 3 Aug 2025, Zhang et al., 23 Mar 2026).

Setting	Representative method	Semantic-fusion mechanism
Joint multimodal fusion and super-resolution	FS-Diff (Jie et al., 11 Sep 2025)	clarity-aware semantic guidance in conditional denoising
3D semantic descriptors	Diff3F (Dutt et al., 2023)	multi-view diffusion feature aggregation and lifting
Interactive text-to-image retrieval	ADaFuSE (Zhang et al., 23 Mar 2026)	adaptive diffusion-text fusion with semantic-aware experts
Controllable image fusion	DiTFuse (Li et al., 8 Dec 2025)	joint text-image token fusion in a DiT backbone
Multi-view driving scene generation	DualDiff (Li et al., 3 May 2025)	ORS with Semantic Fusion Attention
Trajectory prediction and planning	DiffSemanticFusion (Sun et al., 3 Aug 2025)	semantic raster BEV fusion with online HD map diffusion
Single-image cross-spectral fusion	MagicFuse (Zhang et al., 2 Feb 2026)	per-timestep multi-domain knowledge fusion
Cross-domain semantic segmentation	DIFF (Ji et al., 2024)	multi-step fusion of diffusion features and cross-attention maps
Degraded image fusion	DSPFusion (Tang et al., 30 Mar 2025)	degradation and semantic dual-prior guidance

2. Common architectural principles

Across these systems, the dominant pattern is conditional diffusion in which semantic signals alter either the denoiser input, the denoiser internals, or the representation being denoised. In FS-Diff, the reverse chain is conditioned on low-resolution source modalities and clarity-aware semantics,

$$p_\theta(F_{0:T}\mid x,y)=p(F_T)\prod_{t=1}^{T}p_\theta(F_{t-1}\mid F_t,x,y),$$

with the denoiser receiving the cross-modal global representation $\hat{s}_t$, the noise-level embedding $\gamma_t$, and the selected semantic embedding $e^{ci}$ (Jie et al., 11 Sep 2025). In the explicit DiffSemanticFusion framework for autonomous driving, a truncated DDPM is applied to online lane vectors, with noisy vectors $$\tau_k^i=\sqrt{\bar{\alpha}_i}v_k+\sqrt{1-\bar{\alpha}_i}\epsilon$$ and a denoiser conditioned on fused BEV features and map context (Sun et al., 3 Aug 2025). In MagicFuse, semantics are fused even more directly at the noise level through

$$\epsilon_t^{f}=w\,\epsilon_t^\psi + (1-w)\,\epsilon_t^\phi,$$

where the weight is later modulated by a radiation category map derived from a segmentation head (Zhang et al., 2 Feb 2026).

A second recurring pattern is feature-space semantic fusion rather than pixel-space fusion. Diff3F extracts intermediate diffusion features from a Stable Diffusion U-Net decoder across multiple timesteps, combines them with DINOv2 features by normalized concatenation with $\alpha=0.5$, then lifts them onto a 3D surface by back-projection and multi-view averaging (Dutt et al., 2023). DIFF similarly concatenates decoder hidden states and cross-attention maps across timesteps and layers,

$$F_{\text{diff}}=F\big(\oplus_{t,l}[V_{t,l}^{\text{inter}},A_{t,l}^{\text{cross}}]\big),$$

and uses the fused representation as a segmentation backbone (Ji et al., 2024). ADaFuSE moves the same principle into retrieval: it replaces static additive fusion with an adaptive gate $\lambda$ and a semantic-aware mixture-of-experts residual, producing a normalized fused query embedding (Zhang et al., 23 Mar 2026).

A third principle is that semantic fusion is often paired with an explicit reliability mechanism. FS-Diff selects content embeddings according to a clarity sensing mechanism built on CA-CLIP, choosing the most reliable modality under clear/blur conditions (Jie et al., 11 Sep 2025). ADaFuSE downweights unreliable generated images when text-image alignment is weak, and assigns more conservative generated-image weights than the fixed weighting used by DAR (Zhang et al., 23 Mar 2026). DualDiff separates foreground and background into two ControlNet-style branches and adds a foreground-aware masked loss to emphasize tiny objects (Li et al., 3 May 2025). This suggests that recent DiffSemanticFusion systems treat semantics not merely as extra information, but as confidence-weighted structure that must be filtered before or during denoising.

3. Low-level vision and image-fusion systems

In low-level vision, DiffSemanticFusion is most directly realized by joint image fusion and restoration models. FS-Diff formulates simultaneous multimodal fusion and super-resolution as conditional generation from pure Gaussian noise, using a clarity sensing and semantic extraction module, Bidirectional Feature Mamba for global cross-modal representation learning, and a modified U-Net denoiser with semantic cross-attention injection (Jie et al., 11 Sep 2025). Its training objective is the DDPM noise-prediction MSE

$$L=\mathbb{E}\big[\|f_\theta(\hat{s}_t,\gamma_t,e^{ci})-\epsilon\|_2^2\big],$$

without additional perceptual or gradient losses. The method is trained jointly across $\hat{s}_t$0, $\hat{s}_t$1, and $\hat{s}_t$2 magnifications for 800,000 steps, uses inference steps $\hat{s}_t$3, and was evaluated on LLVIP, M3FD, MSRS, Harvard MRI-PET, MRI-SPECT, Lytro, MFI, and AVMS. Reported downstream results include detection mAP@0.50 of 0.933 on LLVIP and 0.795 on AVMS, and segmentation mIoU of 0.749 on MSRS and 0.665 on AVMS (Jie et al., 11 Sep 2025).

DSPFusion addresses degraded infrared-visible fusion by separating modality-specific degradation priors from joint semantic priors, then restoring a high-quality semantic prior in a compact latent space via a Semantic Prior Diffusion Model (Tang et al., 30 Mar 2025). Its architecture combines a Degradation Prior Embedding Network, a joint Semantic Prior Embedding Network, latent diffusion over semantic tokens, and a Restormer-based restoration-and-fusion network equipped with a Dual-Prior Guidance Module and a Prior-Guided Fusion Module. Because the diffusion model operates on $\hat{s}_t$4 semantic tokens rather than full-resolution images, DSPFusion uses $\hat{s}_t$5 timesteps in practice and is reported as approximately 0.119 s/image on a single RTX 4090, over 29× faster than image-space DDFM on MSRS. The method achieves strong results across degraded scenarios such as VI blur, rain, low-light, over-exposure, and IR low-contrast, random noise, and stripe noise, and improves YOLOv8 nighttime pedestrian detection on LLVIP to AP@50 $\hat{s}_t$6 and mAP $\hat{s}_t$7 (Tang et al., 30 Mar 2025).

MagicFuse extends semantic fusion to the single-image regime, where only a degraded visible image is available (Zhang et al., 2 Feb 2026). It introduces intra-spectral knowledge reinforcement and cross-spectral knowledge generation as two latent diffusion streams, then fuses the timestep-wise probabilistic noises from both streams through a learned weighting operator in a Multi-domain Knowledge Fusion branch. Semantic constraints are imposed by a segmentation head that produces a radiation category map to modulate the fusion weight and avoid collapse toward visible-only guidance. On MFNet, the reported visual metrics include EN 7.29, MI 4.13, and PSNR 63.49, while semantic evaluation with SegFormer yields 62.19 mIoU for the full fusion output; on FMB, the corresponding semantic score is 55.41 mIoU (Zhang et al., 2 Feb 2026).

DiTFuse generalizes the same theme into an instruction-driven Diffusion Transformer that jointly encodes two images and natural-language instructions in a shared latent space (Li et al., 8 Dec 2025). It uses an SDXL VAE, a Transformer text encoder with the pretrained Phi-3 tokenizer, task and subtask tags, hybrid attention, and a 32-layer Omnigen-style DiT fine-tuned with LoRA. The training objective is a rectified-flow loss,

$\hat{s}_t$8

under multi-degradation masked-image modeling rather than paired fusion supervision alone. DiTFuse reports, among other metrics, MSRS performance of MSE 0.021 and PSNR 66.63, FMB instruction-conditioned segmentation mIoU overall 0.4760, and an RTX 3090 latency of 53.551 s/sample with approximately 3.846B parameters (Li et al., 8 Dec 2025).

4. Feature, geometry, and segmentation formulations

DiffSemanticFusion is not limited to image synthesis. Diff3F shows that diffusion features can be fused across views and timesteps to produce class-agnostic semantic descriptors for untextured meshes and point clouds (Dutt et al., 2023). The method renders depth and normal maps for meshes, or depth and Canny edge maps for point clouds, uses ControlNet-conditioned Stable Diffusion with DDIM sampling and 30 inference steps, extracts 1280-D diffusion features from a U-Net decoder, time-aggregates features from later denoising steps with linearly increasing weights, fuses them with DINOv2 features, and then lifts them to 3D through visibility-aware back-projection and mean aggregation across approximately 100 views. Reported correspondence results include SHREC’19 accuracy 26.41% at 1% error tolerance with 1.69 average error, SHREC’20 accuracy 72.60% with 0.93 error, and FAUST average geodesic error 5.29 cm (Dutt et al., 2023). The central claim is that even when per-view conditional generations are inconsistent, the associated diffusion features remain semantically stable enough to aggregate.

DIFF applies the same intuition to domain-generalized semantic segmentation (Ji et al., 2024). It freezes Stable Diffusion v1-5 as a backbone, extracts U-Net decoder features and cross-attention maps at multiple steps and multiple layers, and fuses them with a residual bottleneck network. A second contribution is implicit posterior knowledge learning, in which a conditional branch uses class names and class masks to form a path-controlled diffusion trajectory, while an unconditional branch is aligned to it through a consistency loss. The final objective is

$\hat{s}_t$9

with $\gamma_t$0. Using 50 inversion steps with exponential timestep rescheduling, DIFF reports 58.01 mIoU on GTA5 $\gamma_t$1 Cityscapes, 53.60 on GTA5 $\gamma_t$2 BDD100K, 59.85 on GTA5 $\gamma_t$3 Mapillary Vistas, 46.32 on ACDC, and 30.66 on Dark Zurich (Ji et al., 2024). The method therefore treats diffusion semantic fusion as trajectory-level feature aggregation rather than as image restoration or generation.

These feature-centric systems clarify an important boundary in the term. In low-level fusion, semantics guide the creation of a fused image. In Diff3F and DIFF, by contrast, semantics are the fused product. This suggests that DiffSemanticFusion can denote either a semantic control mechanism for generative fusion or a semantic representation-learning mechanism derived from diffusion dynamics.

5. Driving, planning, and retrieval formulations

In autonomous driving, DiffSemanticFusion appears both as an explicit framework and as an architectural motif. The named DiffSemanticFusion framework builds a unified BEV feature space $\gamma_t$4 from dense BEV features, graph features, and raster image features, then stabilizes online HD maps with a truncated diffusion module applied to lane vectors (Sun et al., 3 Aug 2025). In planning experiments, the BEV tensor is sized 512×128×128. On nuScenes trajectory prediction, integrating the diffusion module into QCNet with StreamMapNet online maps improves ADE/FDE/MR from 0.354/0.717/0.068 to 0.336/0.673/0.0549, described as an approximately 5.1% improvement. On NAVSIM, the same framework reports state-of-the-art or best-in-class EPDMS results, including 85.1 on Navtest with a ResNet34 backbone, 86.5 with V2-99, and NavHard gains of +15% and +19% in the two backbone settings (Sun et al., 3 Aug 2025).

DualDiff addresses a related but distinct problem: multi-view driving scene generation with richer semantic conditioning (Li et al., 3 May 2025). Its design introduces Occupancy Ray Sampling as a semantic-rich 3D representation aligned to each camera view, a Semantic Fusion Attention module that performs self-attention, gated self-attention with spatial descriptors, and deformable attention with text, and a foreground-aware masked loss that gives greater weight to tiny objects through box-area-dependent masking. The system uses Stable Diffusion v1.5 with dual ControlNet-style branches for foreground and background. On nuScenes at 224×400, DualDiff reports FID 10.99, road mIoU 62.75, vehicle mIoU 30.22, mAP 13.99, and NDS 24.98; on Waymo it reports FID 11.45 and 3D detection AP 10.40 (Li et al., 3 May 2025).

ADaFuSE transfers the same semantic-fusion logic to interactive text-to-image retrieval (Zhang et al., 23 Mar 2026). Here the fused object is not an image or a map, but a retrieval query embedding constructed from dialogue text and a diffusion-generated visual proxy. The paper argues that simple embedding addition indiscriminately incorporates generative noise and reports degradation for up to 55.62% of samples under static fusion, with average rank drops of approximately 7,500 on degraded queries in DAR. ADaFuSE replaces this with a dual-branch module: an adaptive gate

$\gamma_t$5

and a semantic-aware mixture-of-experts residual. Across four benchmarks, it surpasses DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase, and reduces the average rank drop on degraded queries to approximately 20 (Zhang et al., 23 Mar 2026).

Taken together, these systems show that in decision-centric settings the semantics being fused are often topological, behavioral, or relational rather than photometric. The diffusion component is used to denoise vector maps, align 3D occupancy with text and geometry, or calibrate generated visual evidence against dialogue intent.

6. Evaluation themes, limitations, and research directions

Evaluation protocols vary by domain, but several regularities recur. Low-level vision systems are usually measured with PSNR, SSIM, VIF, $\gamma_t$6, LPIPS, MSE, and task-driven metrics such as mAP or mIoU (Jie et al., 11 Sep 2025, Tang et al., 30 Mar 2025, Li et al., 8 Dec 2025, Zhang et al., 2 Feb 2026). Feature-centric systems use correspondence accuracy, geodesic error, or cross-domain segmentation mIoU (Dutt et al., 2023, Ji et al., 2024). Driving systems use FID, BEV segmentation, 3D detection metrics, ADE/FDE/MR, or EPDMS (Li et al., 3 May 2025, Sun et al., 3 Aug 2025). Retrieval systems use Hits@10 and rank degradation analyses (Zhang et al., 23 Mar 2026). A plausible implication is that DiffSemanticFusion is best understood as a design pattern whose outputs may be images, semantic fields, maps, trajectories, or query embeddings; the unifying criterion is semantic conditioning or semantic aggregation inside a diffusion-derived pipeline.

The main limitations are equally consistent. Computational cost remains substantial in image-generation variants: FS-Diff reports 58.47M parameters, 64,391.6G FLOPs at 128×128, and about 74 s inference for $\gamma_t$7 steps on an RTX 3090, while DiTFuse reports approximately 3.846B parameters and 53.551 s/sample on an RTX 3090 (Jie et al., 11 Sep 2025, Li et al., 8 Dec 2025). Feature-centric models incur heavy inversion or multi-view costs: Diff3F takes about 2–3 minutes per shape for 100 views on an RTX 4090, and DIFF relies on multi-step inversion rather than a single feed-forward pass (Dutt et al., 2023, Ji et al., 2024). Conditioning quality is another failure point. FS-Diff can underperform under heavy fog, extreme darkness, modality misalignment, or when Gaussian DDPM assumptions are violated; MagicFuse remains constrained by the absence of real infrared sensing at inference; Diff3F is affected by occlusion, sparsity, and diffusion-model bias; DiffSemanticFusion for online maps cannot correct severe global misalignment or missing entire lanes; ADaFuSE still depends on the quality of the upstream generator and on learned router calibration (Jie et al., 11 Sep 2025, Zhang et al., 2 Feb 2026, Dutt et al., 2023, Sun et al., 3 Aug 2025, Zhang et al., 23 Mar 2026).

Future directions reported across these works converge on lighter backbones, fewer denoising steps, stronger semantic priors, and tighter coupling to downstream tasks. FS-Diff explicitly suggests lighter U-Net variants, adaptive noise schedules, improved semantic priors, and external memory (Jie et al., 11 Sep 2025). DiTFuse points to broader unified restoration tasks and higher-fidelity latent encoders (Li et al., 8 Dec 2025). The autonomous-driving literature highlights topology-aware losses, uncertainty modeling, and temporal conditioning (Sun et al., 3 Aug 2025, Li et al., 3 May 2025). ADaFuSE suggests more explicit uncertainty calibration and expert specialization, while DIFF suggests adapters or partial fine-tuning of diffusion backbones (Zhang et al., 23 Mar 2026, Ji et al., 2024). This suggests that the next stage of DiffSemanticFusion research will be less about proving that semantics can be fused inside diffusion models, and more about making such fusion reliable, controllable, and computationally tractable across heterogeneous downstream tasks.