Addressing Detail Bottlenecks in Latent Diffusion for RGB-to-SWIR Image Translation

Abstract: Latent diffusion models (LDMs) enable efficient image-to-image translation but discard fine spatial details during compression, degrading downstream perception tasks. We identify two bottlenecks: the autoencoder, which loses spatial information, and the conditioning pathway, which further degrades the source signal through naive downsampling. We propose two lightweight, backbone-agnostic fixes: a Source-Conditioned Autoencoder (SCAE) that injects high-resolution source features into the decoder via skip connections, and a Learnable Guidance Encoder (LGE) that replaces naive downsampling with a learned conditioning signal. Evaluated on RGB-to-SWIR translation for driving scenes with two denoiser backbones (U-Net and DiT), our approach improves detection mAP by up to 2x over the latent diffusion baseline, with up to 3.4x gains on small objects (COCO-small, <32^2 px^2), while achieving state-of-the-art FID. We further show that FID and detection performance are poorly correlated, motivating multi-axis evaluation. Results generalise zero-shot to the public RASMD benchmark. We will publicly release test data with annotations, all checkpoints, and training code.

• The paper introduces SCAE and LGE modules to preserve fine spatial details lost during autoencoder compression and naive conditioning.
• The approach improves perceptual metrics (e.g., FID, LPIPS) and detection mAP, achieving over 3.4× performance gains on small objects.
• The method features minimal computational overhead, is backbone-agnostic, and generalizes well to unseen datasets for practical RGB-to-SWIR translation.

Addressing Detail Bottlenecks in Latent Diffusion for RGB-to-SWIR Image Translation

Introduction

This work targets architectural deficiencies in latent diffusion models (LDMs) for cross-modal image-to-image (I2I) translation, specifically from RGB to short-wave infrared (SWIR) imagery. The study identifies two primary sources of spatial detail loss—autoencoder compression and naive conditioning pathway—and introduces two backbone-agnostic modules: Source-Conditioned Autoencoder (SCAE) and Learnable Guidance Encoder (LGE). Extensive experiments demonstrate significant improvements in both perceptual metrics and downstream detection performance, establishing new state-of-the-art results for RGB-to-SWIR translation.

Motivation and Problem Analysis

Standard LDMs achieve computational efficiency by compressing images into a latent space, but this abstraction discards fine spatial details essential for downstream vision tasks. When used for translating RGB to SWIR—where material reflectance, not pigment, governs appearance—standard LDMs fail to preserve pedestrians, sign text, and small objects, thereby degrading detection and practical utility.

Two architectural bottlenecks are identified:

1. Autoencoder Bottleneck: The compression-driven encoder-decoder pipeline loses spatial information, particularly affecting small and distant objects.
2. Conditioning Bottleneck: The naive resizing of source images to latent resolution in the conditioning path distorts or discards structural information vital to accurate translation.

Architectural Contributions

Source-Conditioned Autoencoder (SCAE)

SCAE extends the traditional autoencoder by introducing a mirrored source encoder, $E^{\text{src}}$, which extracts multi-scale features from the high-resolution RGB input. These features are injected into the decoding process via skip connections, enabling the decoder $D$ to access fine-grained spatial information at reconstruction time. End-to-end training of the SCAE on the target domain suppresses modality leakage and maximizes structure preservation.

Learnable Guidance Encoder (LGE)

LGE replaces the conventional resizing-based conditioning with a learnable, lightweight convolutional encoder, $\tau_\psi$, co-optimized with the denoiser. The resulting latent-resolution signal, $z^{\text{src}}$, is concatenated with the denoiser input, providing detailed spatial context during iterative sampling.

The overall pipeline supports multiple denoiser architectures (U-Net and DiT), demonstrating backbone-agnostic deployment.

Figure 1: DP-LDM/DP-LDiT pipeline featuring SCAE and LGE modules, highlighting how source features and learned conditioning signals are injected into the diffusion process.

Experimental Setup

The translation task is evaluated on a paired, registered RGB–SWIR dataset comprising 31,999 training pairs and two test splits (900 pairs from Day 1, 1,799 pairs from Day 2). Ground-truth bounding boxes for Pedestrian, Vehicle, and Cyclist classes facilitate task-level detection evaluation. Zero-shot generalization is assessed on the RASMD dataset, which differs in acquisition hardware and geographic context.

Metrics include FID, LPIPS, mAP@50, mAP@50–95, and size-stratified AP, quantifying both perceptual similarity and structural preservation.

Results and Analysis

Image Quality and Structural Fidelity

DP-LDiT achieves the best FID (16.90) and LPIPS (0.142) on Day 1, outperforming both pixel-space DDPM and latent baselines by significant margins. DP variants consistently improve FID by >45% relative to their baselines. Qualitative evaluations confirm that DP-LDiT recovers SWIR-specific reflectance properties (e.g., sign backgrounds, clothing material), while standard LDMs and Pix2pix-turbo either lose fine structure or leak RGB appearance.

Figure 2: DP-LDiT~f8 recovers pedestrians and traffic signs lost by standard LDiT~f8, matching real SWIR detail without DDPM's computational cost.

Figure 3: DP-LDiT preserves object boundaries and SWIR reflectance, outperforming baseline models in both spatial detail and modality fidelity.

Downstream Detection Performance

DP-LDM and DP-LDiT nearly double detection mAP over standard latent diffusion baselines, with up to $3.4\times$ improvements on small objects ($<32^2$ px$^2$). SCAE reconstruction approaches the real SWIR ceiling, while naive conditioning (LDM) yields <0.4 mAP@50.

Figure 4: DP variants uniquely improve both FID and mAP@50, demonstrating that FID alone is insufficient for perception-oriented evaluation.

A significant claim is substantiated: FID and detection performance are poorly correlated. High FID does not guarantee retained structural detail necessary for downstream detection, underscoring the necessity for multi-axis evaluation protocols.

Ablation Study

Isolating LGE and SCAE, SCAE yields greater improvements individually, but the highest gains occur when both modules are combined. SCAE’s skip connections compensate for imperfect latent codes by providing direct high-resolution features, while LGE enhances the latent itself. The architectural bottleneck cannot be solved by training objectives alone; LDM-LPL’s perceptual loss yields only marginal performance enhancements.

Figure 5: Both SCAE and LGE are needed for optimal pedestrian detection and spatial fidelity; their removal degrades either semantic or structural quality.

Size-Stratified and Cross-Dataset Evaluation

DP-LDM and DP-LDiT deliver $>3\times$ AP gains for small objects, validating precise spatial information recovery. Zero-shot transfer to RASMD demonstrates generalization of SWIR imaging physics across hardware and domain shifts; DP-LDiT achieves 68.67 FID on RASMD, outperforming all baselines.

Figure 6: DP-LDiT generalizes to unseen hardware and environments, correctly translating SWIR-specific properties with minimal domain adaptation.

Efficiency Considerations

Parameter and computational overhead of DP modifications are minimal (+0.15–0.19 GB), with negligible inference time increases (~2%), making the approach suitable for scalable, practical deployment. Unlike pixel-space DDPM, DP-LDiT accommodates larger batch sizes while retaining comparable detection performance.

Implications and Future Directions

This work provides a decisive demonstration that architectural design, not only training objective, governs spatial fidelity in latent diffusion I2I translation. The backbone-agnostic approach and lightweight parameter overhead position DP-LDiT as a robust candidate for cross-modal translation tasks in safety-critical perception applications, where fine detail preservation is essential.

Further directions include integrating material-level segmentation priors, expanding to unsupervised or semi-supervised modalities, and validating the utility of translated imagery for data augmentation in scenario coverage. Extending the evaluation beyond detection (e.g., semantic segmentation, tracking) will clarify the broader impact of detail-preserving latent diffusion.

Conclusion

The paper establishes two precision-preserving modules—SCAE and LGE—that address inherent bottlenecks in latent diffusion for cross-modal translation. The improvements in structural fidelity and downstream perception are validated across rigorous benchmarks and architectures, with demonstrated zero-shot transfer. Task-level evaluation is essential for practical I2I translation; FID alone cannot capture the impact on downstream vision pipelines. This study sets a precedent for future work in modality translation, emphasizing architectural innovation and multi-dimensional evaluation (2606.19961).