ReFusion: Adaptive Fusion Frameworks

• ReFusion is a suite of innovative methodologies in machine learning and computer vision that adaptively fuses representations, losses, and signals across diverse tasks.
• Its frameworks employ advanced techniques such as latent diffusion, bi-level retrieval fusion, and meta-learning to optimize performance and efficiency with rigorous hyperparameter tuning.
• ReFusion demonstrates practical scalability and applicability in areas like image restoration, language understanding, 3D inpainting, and SLAM, delivering state-of-the-art results and accelerated inference.

ReFusion is a term that has been independently employed across diverse domains in machine learning and computer vision to denote methods and frameworks leveraging innovative fusion strategies—either of representations, losses, or signals—for challenging problems in restoration, fusion, inpainting, retrieval augmentation, LLM decoding, and 3D reconstruction. This article presents an overview of major ReFusion frameworks and their technical underpinnings as described in the cited literature.

1. Latent-Space Diffusion for Large-Scale Image Restoration

ReFusion (Luo et al., 2023) is a general-purpose image restoration pipeline formulated on diffusion models, specifically utilizing a mean-reverting stochastic differential equation (SDE) inspired by IR-SDE. Its two central technical advances are (i) a U-Net–based latent-space diffusion strategy, and (ii) extensive tuning of diffusion hyperparameters to jointly optimize both distortion and perceptual metrics.

The latent diffusion process operates not in pixel space, but in a compressed latent domain: a U-Net encoder maps the input, potentially very large (up to $6000\times4000\times3$), into a latent $z \in \mathbb{R}^{C \times H' \times W'}$. Diffusion denoising is performed in this compact latent, with each reverse diffusion step incurring dramatically reduced computational cost (e.g., $\sim$4 GFLOP for a $1024^2$ patch versus $\sim$63 GFLOP in pixel space). A U-Net decoder with multiple skip connections restores full-resolution output, maintaining high-frequency detail. Unlike VAE-GAN compressive priors, this avoids adversarial objectives, stabilizes reconstruction, and prevents GAN-induced instabilities.

Training is structured in two phases: (1) pretraining the U-Net compressor via paired LQ-HQ reconstruction losses; (2) freezing the compressor and learning the diffusion model on latent pairs with a simple MSE noise prediction loss: $$L_\mathrm{diffusion}(\theta) = \mathbb{E}_{x_0, \epsilon, t} \left[ \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2 \right]$$ Hyperparameters—noise level $\lambda$, diffusion step count $T$, patch size, optimizer choice (Lion [chen2023symbolic]), and scheduler (cosine decay)—are rigorously ablated. The final configuration ($\lambda=50$, $T=100$, patch size $z \in \mathbb{R}^{C \times H' \times W'}$0) yields strong empirical gains on both distortion (PSNR/SSIM/RMSE) and perceptual (LPIPS/FID) scores.

Refusion demonstrates task-agnostic extensibility: only minimal architectural or dataset-specific modifications are needed to handle a broad array of restoration tasks, including shadow removal (NTIRE 2023), stereo super-resolution, bokeh effect transformation, and HR non-homogeneous dehazing. It achieves state-of-the-art or highly competitive results while substantially accelerating inference (2–5× over IR-SDE).

2. Bi-Level Retrieval Fusion for Efficient Language Understanding

A distinct ReFusion (Wu et al., 2024) addresses the integration of retrieval-based external knowledge into LLMs for non-knowledge-intensive (NKI) natural language understanding tasks. Classical retrieval-augmented techniques concatenate retrieved texts to inputs, inflating sequence length and quadratic self-attention costs.

ReFusion’s architecture instead fuses compact retrieval representations directly into Transformer hidden states via layerwise adaptive selection of fusion operators. This is accomplished with a retrieval fusion module that, at selected layers, augments [CLS] token (or other positions) using

• a reranker-based softmax-weighted sum over retrieved hidden vectors
• an ordered-mask-based Gumbel-Softmax mechanism for fine-grained vector selection

An architecture search module with learned “gates” determines per-layer whether reranker-based, mask-based, or native processing is optimal. The overall design employs bi-level optimization: (1) model weights are trained for a fixed fusion configuration; (2) fusion architectures are searched with validation loss. This approach avoids input bloat and keeps attention costs at $z \in \mathbb{R}^{C \times H' \times W'}$1—essentially the vanilla base model with a minor additive term.

Empirically, ReFusion achieves average accuracy gains of +2.5 points over strong non-retrieval baselines and substantially outperforms concatenative retrieval augmentation, with best or competitive results on 10/15 NKI benchmarks. The model remains robust with low result variance while incurring nearly no computational overhead beyond a vanilla Transformer.

3. Meta-Learning Losses for Image Fusion

In image fusion, deep learning methods face the challenge of lacking ground truth for the fused target. ReFusion (Bai et al., 2023) introduces a meta-learning-based framework which jointly learns a fusion module, a source reconstruction module, and—critically—a loss proposal network that parameterizes pixelwise intensity and gradient weights for the fusion loss.

The loss proposal module dynamically generates adaptive pixelwise weights $z \in \mathbb{R}^{C \times H' \times W'}$2, enforcing a learnable loss: $z \in \mathbb{R}^{C \times H' \times W'}$3 with

$z \in \mathbb{R}^{C \times H' \times W'}$4

and a similar form for the gradient component. This meta-optimized loss landscape allows the fusion network to selectively preserve information from inputs without human-crafted heuristics. The meta-learning procedure alternates between meta-training (inner update), meta-testing (outer update), and full training phases, optimizing the loss proposal module such that fused outputs are maximally reconstructive of the sources.

Evaluations across infrared-visible, medical, multi-focus, and multi-exposure fusion benchmarks show superior information retention, enhanced texture, and sharper reconstructions compared to prior methods. Ablation confirms the benefit of learnable, dynamically weighted losses and the necessity of the meta-learning cycle.

4. Diffusion Models for 3D Scene Inpainting via Reference Personalization

RefFusion (Mirzaei et al., 2024) for 3D scene inpainting addresses the ill-posed inverse problem of synthesizing plausible completions for masked regions in reconstructed 3D environments. Its principal innovation is the multi-scale personalization of a 2D inpainting latent diffusion model (LDM) to a specific reference view of the scene, using low-rank (LoRA) updates restricted to attention and text encoder weights.

This reference adaptation trains the LDM on global and local crops of the reference, conditioning on random rectangular masks and “inpaint” prompts, such that the diffusion prior tightly matches the style and geometry of the scene. For 3D optimization, the system employs Gaussian-splatting representations and score distillation sampling (SDS). Gradients from the personalized LDM are backpropagated to the 3D scene parameters, enabling the generation of multi-view consistent details. The training objective combines a global SDS loss, local SDS loss, reconstruction loss on unmasked geometry, adversarial regularization around boundaries, and monocular depth alignment.

Ablations show that reference personalization significantly sharpens completed regions and reduces variance in gradients, improving both user-rated and automatic perceptual metrics (e.g., LPIPS). The method generalizes to object removal, insertion, outpainting, and sparse-view scene completion in diverse settings.

5. Slot-Level Parallel Decoding for LLMs

ReFusion (Li et al., 15 Dec 2025) introduces a masked diffusion LLM employing slot-level, plan-and-infill parallel decoding to address the inefficiencies of both autoregressive models (slow sequential inference) and classical masked diffusion (KV cache-incompatible, incoherent outputs).

Here, the target sequence $z \in \mathbb{R}^{C \times H' \times W'}$5 is partitioned into slots (fixed contiguous subsequences, e.g., $z \in \mathbb{R}^{C \times H' \times W'}$6 tokens). A diffusion-based planning step predicts “draft tokens” per slot, scores them, and selects a batch of weakly dependent slots for parallel infilling via autoregressive decoding within each slot. Causal attention masking enables full KV cache reuse and maintains local coherence. The learning complexity is dramatically reduced from $z \in \mathbb{R}^{C \times H' \times W'}$7 mask patterns (token-level MDM) to $z \in \mathbb{R}^{C \times H' \times W'}$8 slot permutations (with $z \in \mathbb{R}^{C \times H' \times W'}$9), accelerating both training and inference.

This design achieves 18$\sim$0 raw speedup over previous MDMs and 2.33$\sim$1 speedup over an 8B autoregressive LLM (Qwen3-8B) on average, with no loss—and in some benchmarks, an improvement—in generation quality. Once slots are filled, errors are not correctable (future work could enable intra-slot refinement).

6. Residual-Driven Strategies in Image Restoration and SLAM

Two other frameworks, Resfusion (Shi et al., 2023) and ReFusion in 3D RGB-D SLAM (Palazzolo et al., 2019), exploit residual signals differently. In image restoration, Resfusion incorporates the residual between observed degraded and clean images into the forward diffusion process; the reverse chain, initialized closer to the data, requires only a fraction of standard DDPM sampling steps ($\sim$2), yielding state-of-the-art restoration with only five denoising steps, and accelerated generation when extended to standard image synthesis tasks.

In 3D SLAM (Palazzolo et al., 2019), ReFusion fuses RGB-D sensor data into a TSDF volumetric map and leverages registration residuals to robustly mask dynamic scene regions. Free-space carving, GPU-parallel sparse voxel hashing, and direct photometric/geometric tracking yield robust, real-time, class-agnostic mapping in highly dynamic environments, with performance on par with, or better than, neural or surfel-based SLAM on established benchmarks.

7. Comparative Table of Major ReFusion Frameworks

Paper/Domain	Fusion Target	Distinguishing Method
(Luo et al., 2023) Image Restoration	Latent codes	Latent U-Net, Hyperparameter ablation
(Wu et al., 2024) Language Understanding	Transformer states	Bi-level retrieval fusion, NAS
(Bai et al., 2023) Image Fusion	Loss function	Meta-learned, pixelwise task-adaptive loss
(Mirzaei et al., 2024) 3D Scene Inpainting	LDM gradients	Scene-specific LoRA, SDScore distillation
(Li et al., 15 Dec 2025) LLM Decoding	Token slots	Slot-wise plan-and-infill with causal cache
(Shi et al., 2023) Image Restoration	Diffusion states	Residual-driven accelerated DDPM
(Palazzolo et al., 2019) 3D SLAM	TSDF/residuals	Residual masking, GPU voxel hashing

8. Insights and Outlook

Across domains, ReFusion approaches demonstrate that sophisticated fusion—not necessarily of data but of representations, losses, or schedule—is central to achieving high performance and computational efficiency. Whether integrating retrievals in LLMs, fusing multi-modal images, enabling editable and realistic 3D inpainting, or unlocking parallel decoding in LLMs, these frameworks share an emphasis on algorithmic adaptability and leveraging task structure.

A plausible implication is that as learning models scale and diversify, the role of adaptive, learnable fusion—especially via meta-learning, personalized priors, or structural decomposition—will become increasingly central not only for performance, but also for practical deployment in constrained computational environments. Further, adopting residual-aware or reference-conditioned mechanisms, as illustrated, can yield substantial benefits in both convergence and output quality.