HarmonPaint: Diffusion Inpainting Framework

• HarmonPaint is a training-free text-guided diffusion inpainting framework that uses modified attention to blend generated content with the unmasked image region.
• It employs SAMS, MAKVS, and an Attention Steer Loss to separately control structural fidelity and style transfer during the diffusion process.
• Evaluations show improved CLIP Score and Aesthetic Score, highlighting its strength in harmonizing stylized images despite challenges with extensive masking.

HarmonPaint is a training-free text-guided diffusion inpainting framework that operates on a pretrained Stable Diffusion Inpainting model and modifies its attention computation at inference time so that newly synthesized content is both structurally plausible and stylistically consistent with the visible part of the image (Li et al., 22 Jul 2025). It is formulated for masked image completion with an optional text prompt, but its central concern is not merely filling a hole: it is harmonization, understood as simultaneous preservation of structural fidelity and transfer of style information from the unmasked region to the masked region. The method is especially motivated by stylized images, including paintings, sketches, and transferred-art styles, where a generated region may satisfy the prompt yet still appear visually pasted-in if its texture, color statistics, or rendering style diverge from the surrounding image (Li et al., 22 Jul 2025).

1. Problem setting and conceptual scope

HarmonPaint addresses text-guided image inpainting under three inputs: an image with a missing or masked region, a binary mask indicating where content should be generated, and a text prompt describing what should appear in that region (Li et al., 22 Jul 2025). The target output is an image in which the generated content matches the prompt, fits the geometry and layout of the surrounding image, and blends stylistically with the unmasked region.

The paper treats two failure modes as fundamental. The first is structural failure, in which the inpainted object has poor shape, broken contours, awkward spatial placement, or weak boundary alignment with the visible context. The second is harmony failure, in which the object is recognizable but its texture, palette, or artistic medium differs from the surrounding image, producing a collage-like result rather than a coherent scene (Li et al., 22 Jul 2025). This framing is particularly important for stylized images, where style consistency is not a peripheral aesthetic property but a core criterion of perceptual plausibility.

In HarmonPaint, “training-free” means that the method uses a pretrained Stable Diffusion Inpainting model and does not perform retraining, fine-tuning, or auxiliary-network learning. The model weights remain fixed; only attention computation and guidance during denoising are altered (Li et al., 22 Jul 2025). This places HarmonPaint within the broader class of inference-time intervention methods rather than architecture-retraining approaches.

2. Diffusion backbone and overall pipeline

HarmonPaint is built on top of Stable Diffusion Inpainting. The paper encodes an image $x_0$ into a latent representation

$z_0 = \mathcal{E}(x_0),$

and optimizes the denoising model with

$$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$

Within the U-Net attention blocks, the feature projections are

$$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$

The method uses classifier-free guidance scale $7.5$ and 50-step DDIM sampling (Li et al., 22 Jul 2025).

The high-level pipeline consists of standard latent denoising augmented by three inference-time mechanisms. First, the Self-Attention Masking Strategy (SAMS) modifies encoder self-attention for structural fidelity. Second, the Mask-Adjusted Key-Value Strategy (MAKVS) modifies decoder self-attention for style transfer from the visible region into the masked region. Third, an Attention Steer Loss $\mathcal{L}_s$ biases prompt-related cross-attention toward the masked area (Li et al., 22 Jul 2025).

The paper also introduces an Efficient Division Strategy, which partitions denoising into two time intervals using $\eta$. In the reported setup, $\eta=0.6$. Structural fidelity is emphasized in $[\eta T,T]$, and stylistic harmony is emphasized in $[0,\eta T]$, following the paper’s timestep convention (Li et al., 22 Jul 2025). A plausible implication is that HarmonPaint treats structure and style as separable, though coupled, stages of diffusion control.

3. Attention mechanisms for structure and harmony

HarmonPaint’s central technical claim is that harmonized inpainting can be achieved by intervening in self-attention differently in the encoder and decoder (Li et al., 22 Jul 2025).

Encoder-side structural control: SAMS

SAMS operates on self-attention maps in encoder layers 2–6. The method distinguishes three interaction types: masked-to-masked, unmasked-to-unmasked, and masked-to-unmasked. Its objective is to suppress only the masked–unmasked terms while preserving the two within-region terms. Given a resized and flattened mask $z_0 = \mathcal{E}(x_0),$0, the masked and unmasked attention components are

$z_0 = \mathcal{E}(x_0),$1

$z_0 = \mathcal{E}(x_0),$2

and the modified attention is

$z_0 = \mathcal{E}(x_0),$3

To avoid the brittleness of a hard binary separation, the paper softens the mask using

$z_0 = \mathcal{E}(x_0),$4

with $z_0 = \mathcal{E}(x_0),$5 in experiments (Li et al., 22 Jul 2025). The interpretation given by the paper is that self-attention encodes layout, and excessive masked–unmasked coupling contaminates the evolving representation of the region to be generated.

Prompt localization: Attention Steer Loss

The Attention Steer Loss uses cross-attention maps at resolutions 16 and 32, averages them into

$z_0 = \mathcal{E}(x_0),$6

and restricts token attention to the masked region: $z_0 = \mathcal{E}(x_0),$7 The loss is

$z_0 = \mathcal{E}(x_0),$8

This encourages prompt-related tokens to place attention mass inside the target region rather than diffusing it across the visible background (Li et al., 22 Jul 2025). The paper states that this loss is used to refine the noise map, although the exact latent or noise update equation is not explicitly given.

Decoder-side style transfer: MAKVS

MAKVS is applied in the final 8 decoder layers and is motivated by the paper’s claim that decoder self-attention keys and values carry style information (Li et al., 22 Jul 2025). For each patch $z_0 = \mathcal{E}(x_0),$9, the adjusted key is

$$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$0

where $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$1 is the mean key over the unmasked region. The same construction is used for $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$2.

Rather than replacing original self-attention entirely, HarmonPaint concatenates original and style-adjusted keys: $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$3 and computes

$$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$4

The scaling parameter $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$5 controls style strength. The paper reports $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$6 as effective on stylized data, while suggesting $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$7 for natural-image inpainting (Li et al., 22 Jul 2025).

4. Empirical evaluation

HarmonPaint is evaluated on two stylized benchmarks constructed from MSCOCO and OpenImages, each using 50 WikiArt reference styles and including both segmentation masks and bounding-box masks (Li et al., 22 Jul 2025). The reported metrics are CLIP Score (CS), Image Reward (IR), Aesthetic Score (AS), and CMMD.

On Stylized-MSCOCO with segmentation masks, HarmonPaint reports CS $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$8, IR $$\mathcal{L}=\mathbb{E}_{z_{t},t,y,\epsilon\sim \mathcal{N}(0, 1)}\left[ {\left \| \epsilon - \epsilon_{\theta}(z_t,t,y)\right \|}_2^2 \right].$$9, AS $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$0, and CMMD $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$1. On Stylized-MSCOCO with bounding-box masks, it reports $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$2, $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$3, $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$4, and $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$5. On Stylized-OpenImages, the corresponding segmentation-mask scores are $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$6, $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$7, $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$8, and $$Q=W^{q}f_{t}, \quad K=W^{k}f_{t}, \quad V=W^{v}f_{t}.$$9, while the bounding-box-mask scores are $7.5$0, $7.5$1, $7.5$2, and $7.5$3 (Li et al., 22 Jul 2025). These are the best reported values in the provided comparison tables against Blended Latent Diffusion, ControlNet Inpainting, Stable Diffusion Inpainting, BrushNet, and PowerPaint.

The supplementary user study compares 20 incomplete images across 40 participants with two questions: prompt alignment and style consistency/visual harmony. HarmonPaint receives $7.5$4 on the first question and $7.5$5 on the second, substantially above the reported alternatives (Li et al., 22 Jul 2025).

The ablation studies support the separation of structural and stylistic control. Relative to a baseline with CS $7.5$6, IR $7.5$7, and AS $7.5$8, adding SAMS yields $7.5$9, $\mathcal{L}_s$0, and $\mathcal{L}_s$1; adding $\mathcal{L}_s$2 yields $\mathcal{L}_s$3, $\mathcal{L}_s$4, and $\mathcal{L}_s$5; removing MAKVS from the full method yields $\mathcal{L}_s$6, $\mathcal{L}_s$7, and $\mathcal{L}_s$8; and the full method yields $\mathcal{L}_s$9, $\eta$0, and $\eta$1 (Li et al., 22 Jul 2025). Another ablation shows that masking only masked–unmasked attention is preferable to masking masked–masked or unmasked–unmasked interactions.

5. Position within the harmonization literature

HarmonPaint belongs to a broader line of harmonization research but occupies a specific niche: training-free diffusion inpainting with explicit structural and stylistic attention control (Li et al., 22 Jul 2025). Earlier painterly harmonization systems include optimization-based local-statistics transfer in “Deep Painterly Harmonization” (Luan et al., 2018), dual-domain spatial/frequency modeling in PHDNet (Cao et al., 2022), adversarial residual learning in PHARNet (Wang et al., 2023), object-style hallucination in ArtoPIH (Niu et al., 2023), progressive low-to-high-level stylization in ProPIH (Niu et al., 2023), and mask-aware latent diffusion conditioning in PHDiffusion (Lu et al., 2023).

A distinct cluster of recent methods pursues training-free or zero-shot painterly harmonization via pretrained diffusion models. FreePIH performs latent optimization with a frozen Stable Diffusion model and late-step denoising (Li et al., 2023), while TF-GPH uses image-wise attention sharing with similarity reweighting to combine style and content references without prompts (Hsiao et al., 2024). Relative to these methods, HarmonPaint is more tightly coupled to text-guided inpainting and to within-image style transfer from unmasked to masked regions rather than from explicit external style references. This suggests a different operating regime: less reference-based editing and more context-conditioned completion.

The term “harmonization” also appears in non-diffusion image-editing and scene-understanding work, including semantic-feature-guided photographic harmonization (Sofiiuk et al., 2020), continuous high-resolution harmonization (Chen et al., 2023), and curve-based high-resolution foreground color mapping (Liang et al., 2021). HarmonPaint’s contribution is narrower but more specialized: it treats harmonization as an attention-routing problem inside a pretrained inpainting diffusion model.

6. Limitations and significance

The principal limitation stated for HarmonPaint is that performance degrades when more than $\eta$2 of the image is masked, especially when only one edge remains visible (Li et al., 22 Jul 2025). MAKVS depends on the unmasked region as a style source; if too little context remains, style transfer becomes underconstrained. The paper also leaves some implementation details under-specified, most notably the exact inference-time update associated with the Attention Steer Loss. No explicit runtime, FLOPs, or memory benchmarks are reported (Li et al., 22 Jul 2025).

Within its intended scope, HarmonPaint is significant because it turns harmonized inpainting into a problem of selective attention manipulation rather than retraining. The encoder suppresses masked–unmasked structural interference, the decoder injects style statistics from visible regions, and cross-attention is steered toward the masked target. This combination makes HarmonPaint a specific answer to a recurrent failure mode in diffusion inpainting: generating content that is semantically plausible yet stylistically alien to its surroundings (Li et al., 22 Jul 2025). A plausible implication is that the method’s broader value lies not only in its reported benchmarks, but in the formulation of harmonization as coordinated control over structure-bearing and style-bearing attention pathways.