---
title: 'HarmonPaint: Diffusion Inpainting Framework'
url: https://www.emergentmind.com/topics/harmonpaint
type: topic
---

# HarmonPaint: Diffusion Inpainting Framework

HarmonPaint is a training-free text-guided diffusion inpainting framework that operates on a pretrained Stable Diffusion Inpainting model and modifies its attention computation at inference time so that newly synthesized content is both structurally plausible and stylistically consistent with the visible part of the image [2507.16732]. It is formulated for masked image completion with an optional text prompt, but its central concern is not merely filling a hole: it is harmonization, understood as simultaneous preservation of structural fidelity and transfer of style information from the unmasked region to the masked region. The method is especially motivated by stylized images, including paintings, sketches, and transferred-art styles, where a generated region may satisfy the prompt yet still appear visually pasted-in if its texture, color statistics, or rendering style diverge from the surrounding image [2507.16732].

## 1. Problem setting and conceptual scope

HarmonPaint addresses text-guided image inpainting under three inputs: an image with a missing or masked region, a binary mask indicating where content should be generated, and a text prompt describing what should appear in that region [2507.16732]. The target output is an image in which the generated content matches the prompt, fits the geometry and layout of the surrounding image, and blends stylistically with the unmasked region.

The paper treats two failure modes as fundamental. The first is **structural failure**, in which the inpainted object has poor shape, broken contours, awkward spatial placement, or weak boundary alignment with the visible context. The second is **harmony failure**, in which the object is recognizable but its texture, palette, or artistic medium differs from the surrounding image, producing a collage-like result rather than a coherent scene [2507.16732]. This framing is particularly important for stylized images, where style consistency is not a peripheral aesthetic property but a core criterion of perceptual plausibility.

In HarmonPaint, “training-free” means that the method uses a pretrained Stable Diffusion Inpainting model and does not perform retraining, fine-tuning, or auxiliary-network learning. The model weights remain fixed; only attention computation and guidance during denoising are altered [2507.16732]. This places HarmonPaint within the broader class of inference-time intervention methods rather than architecture-retraining approaches.

## 2. Diffusion backbone and overall pipeline

HarmonPaint is built on top of Stable Diffusion Inpainting. The paper encodes an image \(x_0\) into a latent representation
\[
z_0 = \mathcal{E}(x_0),
\]
and optimizes the denoising model with
\[
\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].
\]
Within the U-Net attention blocks, the feature projections are
\[
Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.
\]
The method uses classifier-free guidance scale \(7.5\) and 50-step DDIM sampling [2507.16732].

The high-level pipeline consists of standard latent denoising augmented by three inference-time mechanisms. First, the **Self-Attention Masking Strategy (SAMS)** modifies encoder self-attention for structural fidelity. Second, the **Mask-Adjusted Key-Value Strategy (MAKVS)** modifies decoder self-attention for style transfer from the visible region into the masked region. Third, an **Attention Steer Loss** \(\mathcal{L}_s\) biases prompt-related cross-attention toward the masked area [2507.16732].

The paper also introduces an **Efficient Division Strategy**, which partitions denoising into two time intervals using \(\eta\). In the reported setup, \(\eta=0.6\). Structural fidelity is emphasized in \([\eta T,T]\), and stylistic harmony is emphasized in \([0,\eta T]\), following the paper’s timestep convention [2507.16732]. A plausible implication is that HarmonPaint treats structure and style as separable, though coupled, stages of diffusion control.

## 3. Attention mechanisms for structure and harmony

HarmonPaint’s central technical claim is that harmonized inpainting can be achieved by intervening in self-attention differently in the encoder and decoder [2507.16732].

### Encoder-side structural control: SAMS

SAMS operates on self-attention maps in encoder layers 2–6. The method distinguishes three interaction types: masked-to-masked, unmasked-to-unmasked, and masked-to-unmasked. Its objective is to suppress only the masked–unmasked terms while preserving the two within-region terms. Given a resized and flattened mask \(M_f\), the masked and unmasked attention components are
\[
A^{self}_{in}=M_{f} \times M_{f}^{\top} \odot A^{self},
\]
\[
A^{self}_{out}=(\mathbf{1}-M_{f}) \times (\mathbf{1}-M_{f})^{\top} \odot A^{self},
\]
and the modified attention is
\[
\widehat{A}^{self} = A^{self}_{in} + A^{self}_{out}.
\]

To avoid the brittleness of a hard binary separation, the paper softens the mask using
\[
\widehat{M}_{f} = (1-\tau)M_{f} + \frac{\tau}{HW},
\]
with \(\tau=0.1\) in experiments [2507.16732]. The interpretation given by the paper is that self-attention encodes layout, and excessive masked–unmasked coupling contaminates the evolving representation of the region to be generated.

### Prompt localization: Attention Steer Loss

The Attention Steer Loss uses cross-attention maps at resolutions 16 and 32, averages them into
\[
A^{cross} \in \mathbb{R}^{HW \times L},
\]
and restricts token attention to the masked region:
\[
\widehat{A}^{cross}_i=A^{cross}_i \odot M_{f}.
\]
The loss is
\[
\mathcal{L}_{s}=-\sum_{i} \log(1-\prod_{j} (\mathbf{1}-\widehat{A}^{cross}_{i,j})).
\]
This encourages prompt-related tokens to place attention mass inside the target region rather than diffusing it across the visible background [2507.16732]. The paper states that this loss is used to refine the noise map, although the exact latent or noise update equation is not explicitly given.

### Decoder-side style transfer: MAKVS

MAKVS is applied in the final 8 decoder layers and is motivated by the paper’s claim that decoder self-attention keys and values carry style information [2507.16732]. For each patch \(i\), the adjusted key is
\[
\widetilde{K}(i) = \begin{cases} K(i),&\text{ if } M_{f}(i)=0, \\
\bar{K}, &\text{ otherwise}, \end{cases}
\]
where \(\bar{K}\) is the mean key over the unmasked region. The same construction is used for \(\widetilde{V}\).

Rather than replacing original self-attention entirely, HarmonPaint concatenates original and style-adjusted keys:
\[
\widetilde{A}^{self}=\text{Softmax}\left(\frac{Q \times [K, \lambda \widetilde{K}]^{\top} }{\sqrt{d}}\right),
\]
and computes
\[
\widetilde{f}'_{t} = \widetilde{A}^{self} \times \begin{bmatrix} V \\
\widetilde{V} \end{bmatrix}.
\]
The scaling parameter \(\lambda\) controls style strength. The paper reports \(\lambda=1.4\) as effective on stylized data, while suggesting \(\lambda=0.8\) for natural-image inpainting [2507.16732].

## 4. Empirical evaluation

HarmonPaint is evaluated on two stylized benchmarks constructed from MSCOCO and OpenImages, each using 50 WikiArt reference styles and including both segmentation masks and bounding-box masks [2507.16732]. The reported metrics are CLIP Score (CS), Image Reward (IR), Aesthetic Score (AS), and CMMD.

On Stylized-MSCOCO with segmentation masks, HarmonPaint reports CS \(28.86\), IR \(-0.56\), AS \(6.55\), and CMMD \(0.103\). On Stylized-MSCOCO with bounding-box masks, it reports \(28.27\), \(-0.46\), \(5.96\), and \(0.080\). On Stylized-OpenImages, the corresponding segmentation-mask scores are \(23.49\), \(-0.58\), \(5.43\), and \(0.136\), while the bounding-box-mask scores are \(23.46\), \(-0.77\), \(5.36\), and \(0.080\) [2507.16732]. These are the best reported values in the provided comparison tables against Blended Latent Diffusion, ControlNet Inpainting, Stable Diffusion Inpainting, BrushNet, and PowerPaint.

The supplementary user study compares 20 incomplete images across 40 participants with two questions: prompt alignment and style consistency/visual harmony. HarmonPaint receives \(53.250\%\) on the first question and \(60.500\%\) on the second, substantially above the reported alternatives [2507.16732].

The ablation studies support the separation of structural and stylistic control. Relative to a baseline with CS \(19.01\), IR \(-0.78\), and AS \(5.31\), adding SAMS yields \(19.93\), \(-0.88\), and \(5.21\); adding \(\mathcal{L}_s\) yields \(21.74\), \(-0.89\), and \(5.28\); removing MAKVS from the full method yields \(23.45\), \(-0.97\), and \(5.26\); and the full method yields \(23.46\), \(-0.77\), and \(5.36\) [2507.16732]. Another ablation shows that masking only masked–unmasked attention is preferable to masking masked–masked or unmasked–unmasked interactions.

## 5. Position within the harmonization literature

HarmonPaint belongs to a broader line of harmonization research but occupies a specific niche: training-free diffusion inpainting with explicit structural and stylistic attention control [2507.16732]. Earlier painterly harmonization systems include optimization-based local-statistics transfer in “Deep Painterly Harmonization” [1804.03189], dual-domain spatial/frequency modeling in PHDNet [2212.08846], adversarial residual learning in PHARNet [2311.08646], object-style hallucination in ArtoPIH [2312.10263], progressive low-to-high-level stylization in ProPIH [2312.10264], and mask-aware latent diffusion conditioning in PHDiffusion [2308.02228].

A distinct cluster of recent methods pursues training-free or zero-shot painterly harmonization via pretrained diffusion models. FreePIH performs latent optimization with a frozen Stable Diffusion model and late-step denoising [2311.14926], while TF-GPH uses image-wise attention sharing with similarity reweighting to combine style and content references without prompts [2404.12900]. Relative to these methods, HarmonPaint is more tightly coupled to text-guided inpainting and to within-image style transfer from unmasked to masked regions rather than from explicit external style references. This suggests a different operating regime: less reference-based editing and more context-conditioned completion.

The term “harmonization” also appears in non-diffusion image-editing and scene-understanding work, including semantic-feature-guided photographic harmonization [2006.00809], continuous high-resolution harmonization [2303.01681], and curve-based high-resolution foreground color mapping [2109.05750]. HarmonPaint’s contribution is narrower but more specialized: it treats harmonization as an attention-routing problem inside a pretrained inpainting diffusion model.

## 6. Limitations and significance

The principal limitation stated for HarmonPaint is that performance degrades when more than \(90\%\) of the image is masked, especially when only one edge remains visible [2507.16732]. MAKVS depends on the unmasked region as a style source; if too little context remains, style transfer becomes underconstrained. The paper also leaves some implementation details under-specified, most notably the exact inference-time update associated with the Attention Steer Loss. No explicit runtime, FLOPs, or memory benchmarks are reported [2507.16732].

Within its intended scope, HarmonPaint is significant because it turns harmonized inpainting into a problem of selective attention manipulation rather than retraining. The encoder suppresses masked–unmasked structural interference, the decoder injects style statistics from visible regions, and cross-attention is steered toward the masked target. This combination makes HarmonPaint a specific answer to a recurrent failure mode in diffusion inpainting: generating content that is semantically plausible yet stylistically alien to its surroundings [2507.16732]. A plausible implication is that the method’s broader value lies not only in its reported benchmarks, but in the formulation of harmonization as coordinated control over structure-bearing and style-bearing attention pathways.

Source: https://www.emergentmind.com/topics/harmonpaint