Differentiable Neural Exposure Fusion

• Differentiable Neural Exposure Fusion is a paradigm that replaces hand-crafted weighting with trainable neural operators to merge differently exposed images.
• The dynamic-scene implementation uses a multi-module CNN with color mapping, merging, and guided de-ghosting to generate artifact-free pseudo-HDR images.
• In 3D rendering, NeEF fuses bracketed 3D Gaussians into HDR scene representations using differentiable operations and softmax-normalized MLPs.

Differentiable Neural Exposure Fusion (NeEF) denotes a fully differentiable, learned exposure-fusion paradigm in which multiple differently exposed observations are mapped to a single fused result by trainable neural components rather than by hand-crafted weighting rules. In the literature provided here, the concept maps directly to the end-to-end CNN proposed for dynamic-scene exposure fusion in "Learnable Exposure Fusion for Dynamic Scenes" (Bouzaraa et al., 2018), and the term is later used explicitly in SeHDR to denote a learned fusion module that integrates bracketed 3D Gaussians into HDR Gaussians for novel-view rendering (Li et al., 23 Sep 2025). Across these formulations, NeEF replaces explicit CRF inversion, explicit geometric warping, or classical Mertens-style pixel weighting with differentiable neural operators, while preserving gradient flow through fusion and, in the 3D setting, through rendering.

1. Conceptual scope and problem definition

Exposure fusion (EF) and HDR reconstruction address related but distinct problems. Classical EF directly merges LDR exposures into a display-ready LDR image, whereas HDR reconstruction estimates scene radiance, typically by inverting a camera response function and then tone-mapping the recovered HDR signal for display. The dynamic-scene NeEF formulation is explicitly positioned on the EF side: it seeks a single high-quality LDR image with balanced contrast and saturation, visually similar to a tone-mapped HDR image, but without reconstructing HDR radiance and without estimating or using the camera response function (Bouzaraa et al., 2018).

This distinction matters because the two major NeEF instantiations operate in different domains. The 2018 formulation processes exposure-bracketed LDR images and outputs a pseudo-HDR LDR image aligned to a chosen reference view. By contrast, SeHDR uses NeEF inside a 3D Gaussian Splatting pipeline: it starts from single-exposure multi-view LDR inputs, synthesizes bracketed 3D Gaussians with varying linear colors, and then fuses those bracketed Gaussians into an HDR scene representation for novel-view rendering (Li et al., 23 Sep 2025).

A persistent difficulty in exposure fusion is misalignment. In sequential exposure bracketing, scene motion and camera motion break the alignment assumption and produce ghost artifacts, especially when exposure differences are large. The 2018 work emphasizes that registration is ill-posed under severe exposure variation because intensity ranges and saturation patterns differ substantially between images, making optical flow or patch-based correspondence unreliable in under- or over-exposed regions (Bouzaraa et al., 2018). SeHDR addresses a related but different ill-posedness: recovering HDR radiance from single-exposure multi-view inputs is underdetermined because saturated highlights and crushed shadows lose information, so the method regularizes recovery by learning base Gaussians in linear radiance space, synthesizing bracketed Gaussians, and fusing them with NeEF (Li et al., 23 Sep 2025).

For context, classical exposure fusion is often written as

$$I_{\mathrm{EF}}(\mathbf{x})= \frac{\sum_{i=1}^{N} w_i(\mathbf{x})\, I_i(\mathbf{x})} {\sum_{i=1}^{N} w_i(\mathbf{x})}, \qquad w_i(\mathbf{x}) \propto C_i(\mathbf{x})^{\alpha}\, S_i(\mathbf{x})^{\beta}\, W_i(\mathbf{x})^{\gamma},$$

where $C$, $S$, and $W$ denote contrast, saturation, and well-exposedness. NeEF departs from this formulation by learning the fusion mapping itself rather than explicitly predicting these weights (Bouzaraa et al., 2018).

2. Dynamic-scene NeEF as end-to-end CNN exposure fusion

The 2018 dynamic-scene formulation adopts a learnable, multi-module CNN architecture inspired by FlowNet-style encoder-decoder designs with long-range skip concatenations. It is organized into three cascaded, fully differentiable modules: a color mapping sub-network, an exposures merging sub-network, and a guided de-ghosting sub-network. The pipeline operates directly on concatenated LDR inputs and does not explicitly predict optical flow, alignment fields, or per-pixel fusion weights (Bouzaraa et al., 2018).

The color mapping sub-network is designed to reduce exposure-induced appearance discrepancies that make registration ill-posed. In the 2-LDR case, the reference view is the under-exposed image and the non-reference view is the over-exposed image. The sub-network takes the concatenated pair and estimates a brighter-exposure instance of the reference image. Architecturally, it uses 5 convolutional layers and 5 deconvolutional layers, with 32 filters per layer and FlowNet-style $4 \times 4$ stride-2 down/up-sampling. Long-range skip concatenations aggregate multi-scale features to improve reconstruction and reduce block artifacts (Bouzaraa et al., 2018).

The exposures merging sub-network receives the reference LDR image, the color-mapped bright estimate of that reference, the non-reference LDR image, and a ghost-fused image computed by classical exposure fusion from the inputs. Its role is to produce an initial pseudo-HDR estimate aligned to the reference view. This module uses 3 convolutional layers and 3 deconvolutional layers, with 16 filters per layer and the same FlowNet-style down/up-sampling and skip concatenation strategy (Bouzaraa et al., 2018).

The guided de-ghosting sub-network refines that initial fused estimate. It takes the initial pseudo-HDR and the ghost-fused prior, suppresses ghost artifacts, and enhances detail, again through an encoder-decoder with 3 convolutional and 3 deconvolutional layers and 16 filters per layer. The resulting output is a final artifact-free fused LDR image with extended dynamic range (Bouzaraa et al., 2018).

A defining property of this formulation is the absence of explicit warping. Misalignment robustness is instead attributed to learned cross-exposure color mapping, multi-scale feature aggregation, the availability of auxiliary details from the non-reference image, and the guided de-ghosting stage. The paper explicitly states that no spatial transformer or differentiable warping is used; robustness comes from learned mappings across exposures and content, together with the ghost-fused prior (Bouzaraa et al., 2018).

3. Inputs, supervision, and training protocol in the 2D formulation

The 2018 method is unusual in how it constructs supervision for dynamic-scene exposure fusion. The key idea is to build training tuples in which misaligned inputs share a correct exposure-fusion target. Stereo datasets provide aligned multi-exposure stacks per view and misaligned counterparts from the other view. Ground-truth fused images are produced from the aligned exposure stack of the reference view using classical exposure fusion, while the network is trained on misaligned inputs taken from the other stereo view. This decouples alignment from supervision: the model always sees misaligned inputs, but the target remains a proper fused image aligned to the reference view (Bouzaraa et al., 2018).

In the 2-LDR training protocol, the reference is the under-exposed image from the left camera and the non-reference is the over-exposed image from the right camera, with exposure ratio at least 8. The reported dataset composition includes Middlebury Stereo 2005, 2006, and 2014, as well as a custom outdoor dataset captured with two IDS uEye cameras. Ground-truth fusion images are computed from the full exposure stack of the reference view rather than from only two exposures, because using only 2 LDRs under very high exposure ratios can inject artifacts into supervision. The 2-LDR setup uses 3080 training pairs and 352 validation pairs, with flips along vertical, horizontal, and diagonal axes as augmentation (Bouzaraa et al., 2018).

The 3-LDR extension changes both the reference definition and the architecture. The reference becomes the mid-exposed image, while the non-reference images are under-exposed and over-exposed views from the other stereo view or a free-motion dataset. Two color-mapping networks are used to estimate the under-exposed and over-exposed instances of the reference mid-exposed image. In this 3-LDR configuration, all sub-networks use 3 convolutional and 3 deconvolutional layers with 16 filters, and the architecture is augmented with two extra non-downsampling convolutions after each encoder and decoder level, except at the lowest-resolution level where four extra convolutions are added (Bouzaraa et al., 2018).

Training uses Caffe, resizes inputs to $800 \times 480$, concatenates them channel-wise, and optimizes with SGD with momentum 0.9 and learning rate $10^{-2}$ with polynomial decay of power 0.9 on an NVIDIA TITAN X. Batch size, epochs, and the exact reconstruction loss are not explicitly stated. The paper indicates that supervised regression to the ground-truth EF image is used; a typical pixel-wise $L_1$ or $L_2$ loss is implied, but not formally specified (Bouzaraa et al., 2018).

A common misconception is that “fully differentiable” in this formulation means every component of classical exposure fusion has been embedded into the network. That is not the case. The ghost-fused image is precomputed outside the learnable pipeline and supplied as an input tensor. The learnable modules are end-to-end differentiable with respect to their parameters, but the classical EF prior itself is not learned (Bouzaraa et al., 2018).

4. NeEF in SeHDR: fusion of bracketed 3D Gaussians

SeHDR redefines NeEF in a 3D radiance-field setting. The scene is represented as a collection of 3D Gaussian primitives

$$G_i = (\mu_i, \Sigma_i, \alpha_i, C_i), \qquad i=1,\dots,N,$$

with center $C$0, covariance $C$1, opacity $C$2, and color $C$3 encoded by spherical harmonics (SH) coefficients in a linear radiance space. For viewing direction $C$4, the per-Gaussian linear color is

$C$5

and differentiable front-to-back alpha compositing yields the rendered linear color at each pixel (Li et al., 23 Sep 2025).

The exposure model follows

$C$6

where $C$7 is LDR intensity, $C$8 is HDR irradiance, $C$9 is a CRF or tone mapper, and $S$0 is exposure time. Starting from base Gaussians learned from single-exposure multi-view LDR inputs, SeHDR constructs bracketed 3D Gaussians by holding geometry fixed and scaling linear radiance according to sampled exposures: $S$1 These bracketed Gaussians are then fused by NeEF into HDR Gaussians (Li et al., 23 Sep 2025).

In this setting, NeEF is a learned, per-Gaussian, per-channel fusion mechanism. For Gaussian $S$2, exposure index $S$3, and channel $S$4, well-exposedness is defined as

$S$5

where $S$6 is a learnable per-channel standard deviation initialized to 0.2. Three small MLPs, one per channel and parameterized by $S$7, map exposure features to pre-normalized weights: $S$8 These normalized weights are then applied not in LDR space but in the linear HDR domain: $S$9 For SH-parameterized Gaussians, fusion is carried out coefficient-wise: $W$0 The fused HDR SH coefficients define the view-dependent HDR radiance used for rendering (Li et al., 23 Sep 2025).

This formulation differs fundamentally from the 2018 one. The earlier NeEF variant synthesizes a fused 2D LDR image aligned to a reference view; SeHDR’s NeEF fuses radiance-bearing 3D primitives in SH space to obtain HDR Gaussians for novel-view synthesis. A plausible implication is that NeEF has evolved from an image-space deghosting-and-fusion operator into a radiance-field fusion module.

5. Differentiability, objectives, and optimization

Differentiability is central in both formulations, but its operational meaning differs. In the 2018 image-fusion pipeline, differentiability refers to the fact that the cascaded color-mapping, merging, and de-ghosting sub-networks consist of standard convolutions, deconvolutions, concatenations, and elementwise nonlinearities, so gradients backpropagate jointly through the three modules. The model is trained end-to-end with supervised regression to a ground-truth fused image aligned to the reference view (Bouzaraa et al., 2018).

In SeHDR, differentiability spans a broader graph. Gradients backpropagate through the softmax-normalized MLP weights, the well-exposedness term, the exposure-scaling operation, the SH summation, the front-to-back Gaussian compositing rule, and the differentiable rasterizer. If the fixed tone mapper $W$1 is used inside losses, gradients also pass through its derivative. This means NeEF is not just differentiable as a fusion block; it is differentiable as part of a full 3D rendering pipeline (Li et al., 23 Sep 2025).

SeHDR uses two explicit training losses. The LDR reconstruction loss renders LDR views by tone-mapping the fused HDR radiance at the base exposure $W$2: $W$3 and then applies

$W$4

A linear self-supervision loss compares the base linear view $W$5 to the fused HDR view scaled to $W$6: $W$7 with total loss

$W$8

This linear self-supervision is reported to stabilize base linear radiance estimation and improve HDR reconstruction (Li et al., 23 Sep 2025).

The reported SeHDR implementation initializes NeEF with $W$9 per channel, uses small per-channel MLPs, trains jointly with Gaussian parameters and SH coefficients using Adam, and evaluates HDR outputs after $4 \times 4$0-law compression

$4 \times 4$1

On synthetic HDR-NeRF scenes with single-exposure training, the paper reports approximately PSNR 25.37, SSIM 0.810, and LPIPS 0.117 without HDR supervision, and approximately PSNR 37.05, SSIM 0.965, and LPIPS 0.032 with HDR supervision as an upper bound (Li et al., 23 Sep 2025).

6. Relations to adjacent work, misconceptions, and limitations

Several related papers implement differentiable neural fusion pipelines even when they do not use the label NeEF explicitly. "DeepFuse" learns exposure fusion for static, aligned image pairs with tied-weight branches, additive feature fusion, and a no-reference MEF-SSIM loss, but assumes static scenes and good alignment (Prabhakar et al., 2017). "Deep Exposure Fusion with Deghosting via Homography Estimation and Attention Learning" combines differentiable homography estimation, attention-guided feature gating, U-Net merging, and adversarial learning for two handheld exposures, explicitly addressing camera motion and moving objects (Chen et al., 2020). "End-to-End Differentiable Learning to HDR Image Synthesis for Multi-exposure Images" embeds differentiable CRF inversion and HDR synthesis inside training so that exposure-transfer networks are supervised through the HDR imaging process itself (Kim et al., 2020). These works define adjacent design axes: unsupervised image-space fusion, alignment-aware deghosting, and differentiable HDR physics.

Two misconceptions recur across the NeEF literature. First, NeEF is not inherently equivalent to HDR radiance recovery. The 2018 formulation outputs an LDR fused image and does not model CRF inversion or recover HDR radiance (Bouzaraa et al., 2018). Second, differentiability does not imply explicit geometry handling. The 2018 method avoids explicit warping entirely, whereas the 2020 homography-attention model includes differentiable warping, and SeHDR fuses exposure-conditioned radiance at the level of 3D Gaussians rather than 2D pixels (Chen et al., 2020).

The limitations are correspondingly domain-specific. In the 2018 system, early FlowNet baselines produced block artifacts until extra long-range skip concatenations were introduced; the paper also notes that explicit geometric alignment is absent, so extremely large parallax or strong non-rigid deformation could remain challenging, even though the qualitative experiments show strong robustness (Bouzaraa et al., 2018). In SeHDR, extreme saturation in the single-exposure inputs may still leave insufficient signal to recover HDR radiance; strong specularities can challenge SH modeling; fixed CRF selection per scene can mismatch the true camera response; few viewpoints or heavy occlusion can impair geometry and color estimation; and dynamic scenes are outside the assumed setting (Li et al., 23 Sep 2025).

Taken together, these works suggest that NeEF is best understood not as a single architecture but as a differentiable fusion principle. In 2D image fusion, it denotes end-to-end neural synthesis of artifact-free pseudo-HDR LDR outputs from exposure-bracketed inputs. In radiance-field modeling, it denotes learned, exposure-quality-aware fusion of bracketed scene representations in a linear HDR domain. The common substrate is the replacement of hand-crafted exposure-fusion rules with trainable differentiable mappings that can be optimized jointly with upstream representation learning and downstream rendering objectives.