Hallucination-Aware Diffusion Prior (HAD)

• HAD is a diffusion-assisted 3D reconstruction framework that uses pixel-wise hallucination scores to mask unreliable outputs and improve multiview consistency.
• It integrates a pretrained multiview encoder and a U-Net based score estimator to selectively filter diffusion-generated novel views during sparse-view reconstruction.
• Multi-sampling fusion in HAD selects pixels with the lowest hallucination risk, leading to enhanced metrics like PSNR and SSIM while reducing reconstruction artifacts.

to=arxiv_search.search 大发快三彩票json {"query":"Hallucination-Aware Diffusion Prior HAD diffusion hallucination prior", "max_results": 10, "sort_by": "relevance"} to=arxiv_search.search 北京赛车能 json {"query":"Hallucination-Aware Diffusion Prior for 3D Reconstruction (Liu et al., 16 May 2026)", "max_results": 5, "sort_by": "relevance"} to=arxiv_search.search 盈立json {"query":"Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation (Oorloff et al., 24 Feb 2025)", "max_results": 5, "sort_by": "relevance"} Hallucination-Aware Diffusion Prior (HAD) most directly denotes a diffusion-assisted sparse-view 3D reconstruction framework in which diffusion-generated novel views are not treated as uniformly reliable supervision, but are scored for hallucination at the pixel level and selectively masked before they influence the reconstructed 3D scene (Liu et al., 16 May 2026). In a broader technical usage, the term also names a family of methods and design principles that make diffusion-based generation or reconstruction sensitive to unsupported content: some works intervene at inference time through score or attention modulation, others regularize training with detector-based losses, and others robustify solver updates when a pretrained diffusion prior is used inside an inverse-problem pipeline (Oorloff et al., 24 Feb 2025). Taken together, these works treat hallucination not as a purely perceptual defect but as a failure of instance faithfulness, multiview consistency, or support under the relevant data distribution.

1. Definition and scope

In its strictest sense, HAD is the method introduced for sparse-view 3D reconstruction, where diffusion priors improve novel-view realism but can also inject “aliens” or other non-existent artifacts into the scene; the central remedy is to estimate a hallucination score map for each generated view and exclude unreliable pixels from 3D supervision (Liu et al., 16 May 2026). The method is formulated around calibrated inputs

$P=\{(I_n, C_n)\}_{n=1}^N,$

a 3D Gaussian Splatting backbone, a pretrained diffusion prior for view refinement, a hallucination score network, and a multi-sampling fusion rule.

A broader interpretation is supported by adjacent work, although those papers usually do not define a probabilistic prior in the strict sense. “Adaptive Attention Modulation” modulates self-attention temperature and injects masked perturbations during diffusion sampling, but is best understood as “hallucination-aware inference-time guidance” rather than a learned prior (Oorloff et al., 24 Feb 2025). “Mitigating Diffusion Model Hallucinations with Dynamic Guidance” implements a directionally selective correction to the reverse score through a noisy-sample classifier, which suggests an anisotropic prior correction focused on artifact-prone directions (Triaridis et al., 6 Oct 2025). “DynamicDPS” uses an unconditional diffusion model plus data consistency to refine outputs from conditional medical reconstruction models, which makes it a practical post hoc corrective prior for hallucination reduction rather than a retrained diffusion prior (Kim et al., 3 Mar 2025). “MobilePicasso” adds a hallucination-aware loss and detector-based data filtering in image editing, but explicitly frames this as a training regularizer rather than a formal prior (Kwon et al., 7 Oct 2025).

Work	Role relative to HAD	Core mechanism
HAD	Direct instantiation	Pixel-wise hallucination scores and masked supervision
AAM	Prior-like inference control	Adaptive self-attention temperature and masked perturbation
Dynamic Guidance	Anisotropic prior correction	Dynamic classifier-guided score sharpening
DynamicDPS	Post hoc corrective prior	Unconditional diffusion prior plus data consistency
MobilePicasso	Training-time surrogate	Hallucination-aware loss and data filtering

This suggests a useful distinction. In the narrow sense, HAD is a named reconstruction framework. In the broader sense, HAD refers to diffusion systems that encode, estimate, or operationalize hallucination risk and then use that signal to reshape denoising, guidance, supervision, or solver updates.

2. HAD for sparse-view 3D reconstruction

The canonical HAD formulation addresses sparse-view 3D reconstruction under 3D Gaussian Splatting. Given a small set of calibrated RGB images, the scene is represented by anisotropic Gaussians, and novel views are rendered and refined by a pretrained diffusion prior before being reused as supervision (Liu et al., 16 May 2026). The paper’s central claim is that diffusion augmentation improves photorealism but remains vulnerable to hallucinated content inconsistent with the observed scene; once such content is used for supervision, it becomes baked into the 3D model.

HAD therefore reframes diffusion-assisted reconstruction as a selective-supervision problem. The high-level pipeline is: train or update a 3DGS model; sample novel poses; render those poses from the current 3D representation; refine the rendered views with a pretrained diffusion prior; estimate a pixel-wise hallucination score map for each refined image; threshold that score map into a mask; and use only reliable pixels for 3D optimization (Liu et al., 16 May 2026). The paper instantiates the diffusion prior with Difix3D and modifies the training schedule relative to Difix3D by using a single-phase strategy in which original input views and masked diffusion-augmented views are used from the start.

The 3DGS optimization objective is

$$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$

For input views,

$$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$

For diffusion-generated novel views, HAD applies a hallucination mask: $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$ Here $m$ is the hallucination mask and $\neg m$ denotes its inversion, so only reliable pixels contribute to supervision.

The diffusion-generated view itself is written as

$$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$

where $G$ is the pretrained diffusion prior, $R_{\Phi}(\hat{c})$ is the current 3DGS rendering at novel pose $\hat{c}$, and $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$0 is the chosen conditioning input view. This formulation makes the prior explicit but treats its outputs as fallible. The novelty of HAD lies less in changing $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$1 itself than in explicitly modeling which parts of $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$2 should be trusted.

3. Hallucination score estimation and multiview consistency

The main technical novelty of HAD is the hallucination score estimator. For a diffusion-generated novel view $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$3 at target pose $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$4, the score branch predicts a dense hallucination map

$$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$5

where

$$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$6

Here $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$7 is a frozen multiview encoder, $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$8 is the set of original input views, $$\arg \min_{\Phi} \ \lambda_{\mathrm{input}} \mathcal{L}_{\mathrm{input}} + \lambda_{\mathrm{novel}} \mathcal{L}_{\mathrm{novel}}. \tag{6}$$9 is the current 3DGS rendering at the target pose, and $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$0 is the learnable score branch (Liu et al., 16 May 2026).

The feedforward novel view synthesis model LVSM supplies the multiview encoder. The rationale is explicit: diffusion priors are strong at photorealistic synthesis but weaker on fidelity, whereas feedforward NVS models like LVSM are better at preserving multiview consistency, so LVSM features are a better basis for deciding whether a generated target-view pixel is plausible given the observed input views (Liu et al., 16 May 2026). In supplementary details, the score network uses the target pose and the three nearest input views among the 9 input views, selected by camera-pose proximity, and the predictor is a three-layer U-Net.

Ground-truth hallucination scores are derived from the per-pixel mean absolute error between the diffusion-generated image and the true image at the same viewpoint. The score branch is trained with an $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$1 loss

$$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$2

Predicted score maps are thresholded into binary masks using a fixed threshold of $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$3 (Liu et al., 16 May 2026). The paper does not introduce a soft weighting scheme in the reconstruction loss; the implemented masking is binary.

This design makes HAD a multiview-consistency prior proxy. Rather than assuming a diffusion output is reliable because it is realistic, the method asks whether each pixel is supported by the input-view set as represented through LVSM features. A plausible implication is that HAD shifts the role of the diffusion prior from unconditional completion to confidence-weighted view augmentation.

4. Multi-sampling and score-based fusion

A second defining feature of HAD is multi-sampling. Difix3D refines a target view using a single reference input view, but HAD argues that one conditioning view can omit context available from others. It therefore generates multiple diffusion outputs for the same target pose, each conditioned on a different sampled input view, and uses hallucination scores to fuse them (Liu et al., 16 May 2026).

For a target pose, HAD samples $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$4 reference views and produces

$$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$5

where $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$6 is the diffusion-generated image conditioned on the $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$7-th reference and $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$8 is its hallucination score map. Fusion is pixelwise: $$\mathcal{L}_{\mathrm{input}} = 0.8\,L_1\!\left(R_{\Phi}(c), i\right) + 0.2\,\mathrm{CD\text{-}SSIM}\!\left(R_{\Phi}(c), i\right). \tag{7}$$9 Each pixel in the final fused view is therefore taken from the candidate with the lowest hallucination score at that location.

The paper also evaluates weighted averaging, but ArgMin performs better: weighted averaging gives PSNR $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$0, SSIM $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$1, and LPIPS $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$2, whereas ArgMin gives PSNR $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$3, SSIM $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$4, and LPIPS $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$5 (Liu et al., 16 May 2026). Increasing the number of candidate versions improves quality monotonically in the reported ablation:

• $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$6: PSNR $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$7
• $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$8: PSNR $$\mathcal{L}_{\mathrm{novel}} = L_1\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right) + \mathrm{CD\text{-}SSIM}\!\left(\neg m \odot R_{\Phi}(c), \neg m \odot i\right). \tag{8}$$9
• $m$0: PSNR $m$1

The final system uses $m$2 as a trade-off between quality and compute. This mechanism is important because it shows that HAD is not only a detector-and-mask framework; it is also a confidence-based selector over multiple prior completions. That makes the prior effectively conditional not just on the current rendering and one reference image, but on a small ensemble of cross-view hypotheses filtered by hallucination risk.

5. Relations to adjacent hallucination-aware diffusion mechanisms

Several adjacent papers clarify what HAD is and is not. “Adaptive Attention Modulation” studies hallucinations in diffusion image generators and attributes them to erroneous internal feature routing during early denoising, especially self-attention-mediated amplification or suppression of noisy intermediate structures. It introduces an inference-time temperature-scaled attention rule

$m$3

optimizes $m$4 online against a PatchCore anomaly score on $m$5, and adds masked perturbation of anomalous regions at selected early timesteps (Oorloff et al., 24 Feb 2025). This is highly relevant to HAD because it operationalizes hallucination risk on intermediate states, but the paper explicitly describes the result as closer to “hallucination-aware inference-time guidance” than to a full probabilistic prior.

“Mitigating Diffusion Model Hallucinations with Dynamic Guidance” offers a different route. It interprets hallucinations as off-support samples caused by oversmoothed scores between modes and modifies the reverse score by dynamically selecting a noisy-sample classifier target: $m$6 The correction is applied only over a timestep interval and is designed to sharpen the score selectively along artifact-prone directions while preserving benign semantic variation (Triaridis et al., 6 Oct 2025). This suggests a HAD formulation in which hallucination-awareness is anisotropic and trajectory-dependent.

“DynamicDPS” addresses hallucinations from conditional medical reconstruction models by refining their outputs with an unconditional diffusion prior plus data consistency. Its posterior-guided score is written as

$m$7

and the method adds adaptive starting-time selection and Wolfe line search, while distinguishing intrinsic hallucinations from extrinsic hallucinations in measurement-null-space terms (Kim et al., 3 Mar 2025). This is not a named HAD framework, but it provides a medical-imaging instance of the same principle: use a diffusion prior as a corrective module against unsupported content.

“Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates” isolates the prior update itself as a hallucination injection point in Bayes-rule-based diffusion inverse solvers. Its Robust Prior Update probes local instability by iteratively perturbing against the prior displacement, evaluates the prior at the perturbed point, and re-anchors the resulting displacement at the current iterate before leaving the measurement update unchanged (Jin et al., 1 Jun 2026). This directly supports a HAD view in which hallucination awareness is implemented at the solver level, not by retraining the prior but by robustifying how it is queried.

Other works move further away from a strict prior interpretation. “MobilePicasso” adds a hallucination-aware training loss

$m$8

with $m$9 based on a pretrained hallucination detector, and performs detector-based data filtering for high-resolution image editing (Kwon et al., 7 Oct 2025). “OmniScaleSR” uses explicit scale-conditioned modulation inside a latent diffusion super-resolution system to regulate how much the pretrained prior is allowed to invent detail as scale increases (Chai et al., 4 Dec 2025). Both are closely related to HAD, but each paper describes its contribution as control, regularization, or conditioning rather than as a formal hallucination-aware prior.

6. Empirical performance, limitations, and implications

On DL3DV, HAD reports PSNR $\neg m$0, SSIM $\neg m$1, and LPIPS $\neg m$2, compared with Difix3D at PSNR $\neg m$3, SSIM $\neg m$4, and LPIPS $\neg m$5, for an improvement of about $\neg m$6 dB PSNR, $\neg m$7 SSIM, and $\neg m$8 LPIPS (Liu et al., 16 May 2026). On MipNeRF360, HAD reports $\neg m$9, compared with Difix3D at $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$0, corresponding to $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$1 dB PSNR, $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$2 SSIM, and $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$3 LPIPS. The ablation sequence on DL3DV isolates the contributions of hallucination awareness, multi-sampling, and the single-phase schedule:

• Difix3D: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$4
• Difix3D + HAD: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$5
• Difix3D + HAD + M.S.: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$6
• full HAD: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$7

The hallucination score estimator itself benefits strongly from the pretrained LVSM encoder. The reported MAE values are:

• Retrained Difix3D: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$8
• HAD without pretrained encoder: $$i_g = G\!\left(R_{\Phi}(\hat{c}) \mid i_{\mathrm{ref}}\right), \tag{9}$$9
• HAD full: $G$0 Supplementary ablations show that removing the 3DGS rendering input only weakly affects MAE, whereas removing the pretrained encoder degrades both score quality and final reconstruction (Liu et al., 16 May 2026). The method also generalizes beyond Difix3D: the score network can identify hallucinations in GenFusion and Stable Virtual Camera without fine-tuning, and integrating HAD into GenFusion on DL3DV improves performance from $G$1 to $G$2.

The limitations are equally explicit. HAD depends on the quality and domain generalization of the pretrained multiview encoder; score-network training still requires curated tuples of inputs, 3DGS renders, diffusion outputs, and ground-truth views; the evaluation emphasizes seen-region reconstruction rather than hallucination-aware modeling of truly unseen regions; and multi-sampling increases runtime because each target pose requires multiple diffusion runs (Liu et al., 16 May 2026). A broader implication, supported by related work, is that hallucination awareness in diffusion systems is often strongest when it is attached to intermediate states, local feature routing, or solver proposals rather than only to final outputs (Oorloff et al., 24 Feb 2025).

Taken together, these results establish HAD as a selective-supervision framework in which diffusion priors remain valuable but cease to be treated as infallible. The broader literature suggests the same principle in different technical forms: estimate hallucination risk on trajectories or proposals, couple that risk to attention, score, solver, or masking decisions, and favor instance faithfulness over realism alone (Jin et al., 1 Jun 2026). In that sense, HAD names both a specific 3D reconstruction method and a wider research direction: diffusion priors that are queried, filtered, or corrected with explicit awareness of when their completions are likely to exceed the evidence.