---
title: 'CorrFill: Training-Free Reference Guidance'
url: https://www.emergentmind.com/topics/reference-guidance
type: topic
---

# CorrFill: Training-Free Reference Guidance

CorrFill is a training-free module for **reference-based image inpainting** that improves how faithfully a diffusion model restores missing content from a provided reference image. In this setting, the input consists of a damaged target image \(I_{\text{tar} \in \mathbb{R}^{h \times w \times 3}\), a binary mask \(M \in \{0,1\}^{h \times w}\) marking the missing region, and a reference image \(I_{\text{ref} \in \mathbb{R}^{h \times w \times 3}\) depicting the same scene, often from a nearby viewpoint or another moment. The objective is not merely plausible completion, but recovery of the target’s original content according to the actual geometry and content visible in the reference. CorrFill addresses the observation that diffusion-based inpainting methods often exploit the reference only loosely, producing outputs that are semantically plausible yet geometrically unfaithful. Its central contribution is to extract weak target-reference correspondence cues from diffusion self-attention during denoising and convert them into explicit guidance, without retraining the backbone [2501.02355].

## 1. Task definition and failure mode addressed

Reference-based image inpainting differs from generic exemplar-based editing because the missing region is expected to be reconstructed according to the original scene rather than replaced by any semantically compatible content. In the formulation used by CorrFill, the reference is therefore not auxiliary style information; it is an evidential constraint on the geometry and identity of the missing target region [2501.02355].

The method is motivated by a limitation in existing diffusion-based approaches. Conditioning schemes such as global or spatial image prompts, and stitched-image setups in which reference and target are concatenated side by side, allow information flow from the reference but do not explicitly specify which target patch should attend to which reference patch. The result is a characteristic failure mode: unwanted objects may appear, scene layout may be incorrect, and copied content may originate from semantically related but geometrically wrong regions [2501.02355].

This makes **faithfulness** the governing criterion of CorrFill. The method is not primarily concerned with enlarging the expressive capacity of the backbone. Instead, it introduces explicit correspondence constraints during sampling so that the inpainting process becomes aware of geometric correlation already latent in the model’s own attention maps [2501.02355].

## 2. Position within reference-guided generation

CorrFill belongs to a broader class of reference-guided generation methods, but its mechanism is specific. In supervised encoder-decoder restoration, reference information can be injected through explicit alignment and fusion modules, as in TransRef’s multi-scale reference embedding for inpainting [2306.11528]. In training-free diffusion guidance, control can be introduced through external gradients or latent-space optimization without changing the pretrained generator, as in Semantic Diffusion Guidance [2112.05744]. CorrFill is closer to the latter direction, but it specializes the idea to reference-based inpainting by deriving **online correspondences from self-attention itself** rather than from an external similarity model [2501.02355].

Its implementation is built on **Stable Diffusion v2 Inpainting** in a latent diffusion setting. Following the stitched-image formulation, the reference and target are concatenated horizontally into
\[
I_{\text{ref;tar} \in \mathbb{R}^{h \times 2w \times 3}.
\]
After VAE encoding,
\[
\epsilon(I_{\text{ref;tar}) \in \mathbb{R}^{h' \times 2w' \times d},
\]
the latent is concatenated with a noise latent \(N^\epsilon \in \mathbb{R}^{h' \times 2w' \times d}\) and a downsampled mask \(M^\epsilon \in \{0,1\}^{h' \times 2w'}\), producing
\[
z_T \in \mathbb{R}^{h' \times 2w' \times (2d+1)}.
\]
At each denoising step, the U-Net receives the current latent together with correspondence information estimated from the previous step, creating a cyclic refinement process rather than a static conditioning pipeline [2501.02355].

A plausible implication is that CorrFill treats correspondence as an evolving latent variable of the denoising trajectory rather than as a preprocessing artifact. That distinguishes it from reference alignment pipelines that compute matching only once before generation.

## 3. Online correspondence estimation from self-attention

CorrFill starts from the claim that self-attention maps inside the inpainting diffusion model already contain primitive target-reference correlation signals, including in masked regions. For denoising step \(t\), self-attention maps are collected from multiple layers, heads are averaged, all maps are rescaled to a common spatial resolution, and the results are summed to form an aggregated attention tensor
\[
A_t \in \mathbb{R}^{(h' \times 2w') \times (h' \times 2w')}.
\]
Only the block corresponding to queries from the target and keys/values from the reference is retained, giving
\[
A_t^{\text{tar2ref} \in \mathbb{R}^{h' \times w' \times h' \times w'}.
\]
Rather than matching from a single layer or timestep, CorrFill accumulates temporal consensus:
\[
C_t = \sum_{\tau=t}^{T} A_{\tau}^{\text{tar2ref}},
\]
where \(T\) is the initial diffusion timestep. The correspondence for target token \((i,j)\) is then chosen by nearest-neighbor selection:
\[
P_t(i,j) = \arg\max_{(\hat{i},\hat{j})} C_t(i,j,\hat{i},\hat{j}).
\]
This converts the model’s own self-attention into a dynamic correspondence estimator [2501.02355].

Because raw correspondences are noisy, CorrFill refines them by **filtering** and **smoothing**. Filtering targets a failure mode in which many target tokens collapse onto a few heavily attended reference tokens. These are called **dominant tokens**. If a reference token is selected by more than a threshold number of target tokens, empirically \(4\), those assignments are treated as outliers and removed from the valid set. The excluded correspondences are stored as \(P_t^o\), because they remain useful later as regions to suppress in attention masking [2501.02355].

Smoothing is based on the observation that correspondences near the mask boundary are more reliable than those deep inside the hole. CorrFill therefore defines a displacement field
\[
D_t(i,j) = P_t(i,j) - (i,j)
\]
and a consensus-weight matrix
\[
W_t(i,j) = C_t(i,j,P_t(i,j)).
\]
Outlier correspondences receive zero weight. The displacement is then replaced by a weighted neighborhood average,
\[
D_t^\ast(i,j) = \frac{1}{|W_t(i,j)|} \sum_{(\hat{i},\hat{j}) \in \mathcal{N}(i,j)} D_t(\hat{i},\hat{j}) \cdot W_t(\hat{i},\hat{j}),
\]
with
\[
|W_t(i,j)| = \sum_{(\hat{i},\hat{j}) \in \mathcal{N}(i,j)} W_t(\hat{i},\hat{j}),
\]
and the refined correspondence becomes
\[
P_t^\ast(i,j) = D_t^\ast(i,j) + (i,j).
\]
This suppresses isolated erroneous matches and enforces locally coherent geometry [2501.02355].

## 4. Guidance mechanisms during denoising

Once correspondences have been estimated at step \(t+1\), CorrFill uses them to guide the next denoising step \(t\) in two ways: **attention masking** and **latent tensor optimization** [2501.02355].

For attention masking, the method modifies the self-attention affinity matrix. Standard attention uses
\[
\frac{QK^\top}{\sqrt{d_a}},
\]
where \(Q\) and \(K\) are query and key matrices and \(d_a\) is the attention embedding dimension. CorrFill replaces this with
\[
\frac{QK^\top + m_t}{\sqrt{d_a}},
\]
where
\[
m_t \in \mathbb{R}^{(h' \times 2w') \times (h' \times 2w')}
\]
is a correspondence-derived additive mask. For an inlier correspondence \(P_{t+1}(i,j)\), the mask boosts affinity within a neighborhood of the matched reference location and sets all other reference positions to \(-\infty\), while leaving non-reference parts unchanged. For an outlier correspondence \(P_{t+1}^o(i,j)\), the mask only blocks the suspect dominant region, preventing repeated attention to irrelevant reference areas [2501.02355].

Attention masking alone is not always strong enough, so CorrFill also performs **latent tensor optimization**. It reshapes and resizes target-to-reference attention maps from all self-attention layers, and optimizes the current latent \(z_t\) so that those attention distributions approach a one-hot map centered at the estimated correspondence. The paper states that the objective uses normalized attention, a sigmoid mapping into \([0,1]\), and a weighted binary cross-entropy against the one-hot encoding \(E(P_{t+1})\). Although the printed equation is truncated typographically, the intended meaning is explicit: target-to-reference attention should be pushed toward the estimated geometric match. Conceptually, the update is of the form
\[
z_t \leftarrow z_t - \eta \nabla_{z_t} S.
\]
This latent update is applied layerwise using gradients from the auxiliary objective [2501.02355].

These two mechanisms form a **cyclic enhancement loop**. Correspondences estimated from self-attention at one step constrain the next denoising step through attention masks and latent updates; the resulting denoising pass yields new self-attention maps; these produce better correspondences; and the cycle continues. This suggests that CorrFill is best understood as a self-bootstrapping control mechanism internal to diffusion sampling rather than a static attachment to the U-Net [2501.02355].

## 5. Empirical behavior and ablation evidence

CorrFill is evaluated on **RealEstate10K** and **MegaDepth**, with emphasis on reference faithfulness. The implementation uses \(512\times512\) target and reference images, a stitched latent size \(64\times128\), and **DDIM sampling with 50 steps**. It is attached to multiple baselines, including Paint-by-Example, IP-Adapter-Plus, a side-by-side baseline, and LeftRefill [2501.02355].

On RealEstate10K, the reported PSNR improvements are:

| Method | Baseline PSNR | With CorrFill |
|---|---:|---:|
| Paint-by-Example | 20.03 | 21.57 |
| IP-Adapter-Plus | 21.26 | 25.10 |
| Side-by-side | 23.32 | 25.81 |
| LeftRefill | 26.71 | 26.97 |

The gains are largest for methods lacking explicit geometric grounding, while even LeftRefill, already the strongest baseline and already based on a stitched formulation with prompt tuning, improves slightly in PSNR and also gains in SSIM and LPIPS [2501.02355].

On MegaDepth, improvements are smaller. The authors attribute this to more severe viewpoint changes and dynamic scene variation, which make 2D correspondence estimation harder and sometimes reduce the benefit of strict adherence to the reference. Even so, CorrFill improves Paint-by-Example, IP-Adapter-Plus, and the side-by-side baseline across all metrics, while LeftRefill changes only marginally [2501.02355].

The ablation study clarifies the role of each component. For the side-by-side baseline on RealEstate10K, PSNR progresses from \(23.32\) to \(23.70\) with attention masking alone, to \(24.36\) with outlier filtering, to \(24.37\) with correspondence smoothing, and to \(25.81\) after adding latent \(z_t\) optimization. For LeftRefill, attention masking alone slightly hurts performance, \(26.71\) to \(26.45\), but filtering and smoothing recover the loss, and latent optimization reaches \(26.97\) [2501.02355].

This pattern suggests two things. First, latent optimization provides the largest quantitative gain. Second, stronger baselines are more sensitive to noisy correspondence estimates, so refinement through filtering and smoothing is especially important before applying stronger guidance [2501.02355].

## 6. Limitations, scope, and significance

CorrFill’s limitations follow directly from its correspondence-based design. It can fail in scenes with repetitive structures or complex geometry because correspondences become ambiguous. It also struggles under large viewpoint changes, where 2D patch correspondences are insufficient. The paper gives the example of a statue with incorrect head orientation, illustrating that even roughly correct 2D matching may be inadequate when the underlying object transformation requires 3D reasoning [2501.02355].

Accordingly, CorrFill should not be interpreted as a full 3D-aware alignment system. Its strength lies in the regime where reliable 2D geometric correlation exists between target and reference. In that regime, it increases faithfulness without introducing new trainable modules, adapters, retraining, or fine-tuning of the diffusion backbone [2501.02355].

Its significance is therefore methodological rather than architectural. CorrFill does not propose a new inpainting backbone. It shows that internal diffusion self-attention already contains usable correspondence evidence, and that this evidence can be turned into a practical control signal through temporal consensus, outlier filtering, spatial smoothing, self-attention masking, and latent optimization. In that sense, its main contribution is a training-free mechanism for making existing diffusion inpainting systems more faithful to the reference image [2501.02355].

Source: https://www.emergentmind.com/topics/reference-guidance