Token Painter: Text-Guided Inpainting

• Token Painter is a training-free method for text-guided image inpainting that selectively generates masked tokens while preserving the unmasked background using MAR models.
• Dual-Stream Encoder Information Fusion (DEIF) fuses textual and contextual cues to enhance prompt adherence and boundary coherence during inpainting.
• Adaptive Decoder Attention Score Enhancing (ADAE) modulates decoder attention to boost inpainting detail and ensure visual harmony between generated content and background.

Searching arXiv for the primary paper and closely related work on Token Painter, MAR inpainting, and stroke-based Painter models. Token Painter is a training-free method for text-guided image inpainting built on Mask AutoRegressive (MAR) models. Given an image $I$, a binary mask $M$, and a text prompt $T$, it generates content for the masked region while preserving the unmasked background and maintaining visual harmony with the surrounding context. The method is organized around two inference-time components—Dual-Stream Encoder Information Fusion (DEIF) and Adaptive Decoder Attention Score Enhancing (ADAE)—and is positioned as an alternative to diffusion-based inpainting for settings where local controllability and background preservation are central requirements (Jiang et al., 28 Sep 2025).

1. Problem formulation and motivation

The task addressed by Token Painter is text-guided image inpainting: the masked region should be filled according to the prompt, the unmasked background should be preserved, and the final image should remain visually coherent. The paper frames this as distinct from ordinary inpainting because the completion is not determined solely by surrounding pixels; the text prompt must strongly influence the generated content inside the mask (Jiang et al., 28 Sep 2025).

The paper argues that diffusion-based inpainting methods are limited in this setting because they denoise the entire image latent jointly. In that formulation, masked and unmasked regions are processed together, prompt control can be weakened by surrounding context, and background consistency may also be disrupted. It further notes that blending-based corrections do not fully resolve lighting or color seams and add extra cost (Jiang et al., 28 Sep 2025).

Against this background, Token Painter adopts MAR as the base generative framework. The central claim is that MAR is naturally suitable for inpainting because it generates image tokens step by step, predicts only unknown tokens, and leaves known background tokens fixed. The paper writes the generic MAR factorization as

$$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$

and adapts it for inpainting to

$$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$

where $I_b$ denotes background tokens and the union of the generated sets $S^k$ corresponds to the inpainting tokens (Jiang et al., 28 Sep 2025).

2. MAR adaptation and the failure modes of naive conditioning

Token Painter treats only the masked tokens $I_p$ as unknown and keeps background tokens $I_b$ fixed during generation. This is one reason the method is described as training-free: it does not retrain the MAR backbone, but changes token handling at inference and adds inference-time modules (Jiang et al., 28 Sep 2025).

The paper identifies two naive conditioning strategies and treats their failure modes as the immediate motivation for Token Painter. In the T+B setting, text tokens and background tokens are both input. According to the paper’s decoder attention visualizations, attention then spreads broadly over the image, especially background regions; the output matches context but follows the prompt weakly. In the T-only setting, only text tokens are input; attention becomes concentrated in the masked region, prompt semantics are better preserved, but the generated content is often visually disharmonious with the surrounding image (Jiang et al., 28 Sep 2025).

This analysis yields the method’s main design premise: background tokens are useful for context, but they can overwhelm text semantics; text tokens preserve semantics, but without context they degrade harmony. A plausible implication is that the central challenge is not merely multimodal conditioning, but the controlled redistribution of attention between semantic and contextual token streams.

3. Dual-Stream Encoder Information Fusion

DEIF is the encoder-side mechanism that fuses semantic and contextual information into a new guidance representation. The paper first restricts background interaction to tokens near the mask boundary using a dilated mask $M_d$, and defines a selected background subset as

$M$0

It then runs two encoder branches:

$M$1

Here $M$2 is the guidance stream enriched with boundary context, while $M$3 is the text-only semantic stream (Jiang et al., 28 Sep 2025).

Before fusion, the two streams are aligned statistically. The paper defines

$M$4

and then normalizes and shifts both streams to the shared distribution:

$M$5

The purpose of this alignment is to make the two sources more compatible before fusion (Jiang et al., 28 Sep 2025).

The actual fusion is performed in the frequency domain. The paper argues that low-frequency components are more associated with semantic structure and high-frequency components with contextual style and detail. After transforming the aligned streams into frequency space, it uses a modified Gaussian

$M$6

and fuses them as

$M$7

The paper describes the fused representation as the final guidance tokens $M$8, obtained after inverse transformation. In the reported ablation, the modified Gaussian performs better than linear, constant, and quadratic fusion (Jiang et al., 28 Sep 2025).

4. Adaptive Decoder Attention Score Enhancing

After DEIF, Token Painter modifies decoder attention scores through ADAE. The decoder input is

$M$9

formed by concatenating guidance tokens $T$0 with image tokens $T$1. Standard attention notation is then used:

$T$2

ADAE selectively amplifies entries of $T$3 rather than changing the model weights (Jiang et al., 28 Sep 2025).

The first component, ADAE-G, strengthens attention from inpainting tokens to guidance tokens. The paper introduces an adaptive coefficient

$T$4

and defines

$T$5

Its stated effect is to improve prompt-detail alignment by forcing masked tokens to attend more strongly to the fused semantic-contextual guidance (Jiang et al., 28 Sep 2025).

The second component, ADAE-I, strengthens interactions among inpainting tokens themselves. The masked tokens are split during generation into unknown tokens $T$6 and predicted tokens $T$7, with $T$8, and the paper defines

$T$9

Attention is then modified as

$$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$0

This is intended to improve internal coherence and visual quality as more masked tokens become available during autoregressive decoding (Jiang et al., 28 Sep 2025).

The full inference pipeline is correspondingly staged: encode prompt and image, split image tokens into background and inpainting sets, run the T+B and T-only encoder branches, align and fuse them through DEIF, feed $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$1 into the decoder, apply ADAE, autoregressively generate masked tokens, and decode the final latent tokens with VQ-VAE (Jiang et al., 28 Sep 2025).

5. Experimental setting and reported performance

The paper evaluates Token Painter against SDI, HD-Painter, FreeCond, PowerPaint, BrushNet, and Meissonic on two benchmarks. EditBench contains 240 annotated images with loose masks, where captions describe the masked object. BrushBench contains 600 text-image pairs with tight masks similar to segmentation masks, where captions describe the entire image. Evaluation uses IR, HPS, PS, AS, PSNR, LPIPS, SSIM, and CLIP-S (Jiang et al., 28 Sep 2025).

On EditBench, Token Painter reports the following values: IR $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$2, PS $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$3, HPS $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$4, AS $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$5, PSNR $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$6, LPIPS $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$7, SSIM $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$8, and CLIP-S $$p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),$$9, where CLIP-S is second best. On BrushBench, it reports IR $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$0, PS $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$1, HPS $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$2, AS $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$3, PSNR $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$4, LPIPS $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$5, SSIM $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$6, and CLIP-S $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$7 (Jiang et al., 28 Sep 2025).

The ablation study is structurally important because it decomposes the contribution of the two main modules. On BrushBench, the paper reports the following progression:

• Baseline (T+B only): IR 4.23, PS 19.47, PSNR 26.26, CLIP-S 6.42
• +DEIF: IR 12.41, PS 44.26, PSNR 26.35, CLIP-S 14.42
• +ADAE-G: IR 12.76, PS 46.28, PSNR 26.27, CLIP-S 14.45
• +ADAE-I: IR 13.01, PS 47.90, PSNR 26.39, CLIP-S 14.46

The paper interprets this as showing that DEIF gives the largest improvement, while ADAE-G and ADAE-I further improve prompt alignment and image quality. It also notes that PSNR changes little because MAR preserves the background tokens (Jiang et al., 28 Sep 2025).

A hyperparameter study reports final values $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$8, $$p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),$$9, and $I_b$0. Increasing these coefficients improves alignment or structure only up to a point; values that are too large cause distortion or chaotic colors (Jiang et al., 28 Sep 2025).

6. Conceptual position, related systems, and scope

Token Painter belongs to a broader line of research in which autoregressive models operate over structured visual tokens rather than directly over pixels. An earlier example is "Painter: Teaching Auto-regressive LLMs to Draw Sketches", which generates virtual brush strokes in string form from text prompts and canvas feedback, using an autoregressive LLM with a visual encoder and residual cross-attention blocks (Pourreza et al., 2023). Token Painter differs fundamentally in representation and objective: it does not emit stroke descriptions, but instead manipulates latent image tokens within a MAR inpainting pipeline (Jiang et al., 28 Sep 2025).

It is also related more indirectly to work on visual tokenization such as "WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction", which addresses the trade-off between compression and reconstruction fidelity via Group-wise Lookup-free Quantization (GQ) and Generative Decoding (GD) (Zhuang et al., 7 Aug 2025). That work concerns the quality of discrete tokenization and reconstruction, whereas Token Painter is an inference-time method for conditioning masked-token generation on text and context. A plausible implication is that advances in tokenizer quality could affect the upper bound of MAR-based inpainting quality, even though Token Painter itself is not a tokenizer design.

Several misconceptions are explicitly precluded by the paper’s setup. Token Painter is not a new pretrained image generator; it does not finetune the underlying MAR model. Its contribution lies in inference-time token routing, encoder-side fusion, and decoder attention modification (Jiang et al., 28 Sep 2025). Conversely, the paper does not claim universal superiority of MAR over diffusion in all generative settings; its argument is specific to text-guided image inpainting, where preserving background tokens and generating only masked tokens is especially advantageous (Jiang et al., 28 Sep 2025).

The method’s stated limitations are correspondingly narrow but important. Performance depends on the quality and suitability of the base MAR model; the attention-enhancement hyperparameters must be tuned carefully; over-enhancement can cause distortions; and the evaluation is carried out on specific benchmarks, so broader generalization remains an open empirical question (Jiang et al., 28 Sep 2025). Within that scope, Token Painter is best understood as a targeted MAR-based inpainting framework that attempts to reconcile prompt fidelity, background preservation, and contextual harmony without retraining the base model.