---
title: 'Token Painter: Text-Guided Inpainting'
url: https://www.emergentmind.com/topics/token-painter
type: topic
---

# Token Painter: Text-Guided Inpainting

Searching arXiv for the primary paper and closely related work on Token Painter, MAR inpainting, and stroke-based Painter models.
Token Painter is a **training-free** method for **text-guided image inpainting** built on **Mask AutoRegressive (MAR)** models. Given an image \(I\), a binary mask \(M\), and a text prompt \(T\), it generates content for the masked region while preserving the unmasked background and maintaining visual harmony with the surrounding context. The method is organized around two inference-time components—**Dual-Stream Encoder Information Fusion (DEIF)** and **Adaptive Decoder Attention Score Enhancing (ADAE)**—and is positioned as an alternative to diffusion-based inpainting for settings where local controllability and background preservation are central requirements [2509.23919].

## 1. Problem formulation and motivation

The task addressed by Token Painter is **text-guided image inpainting**: the masked region should be filled according to the prompt, the unmasked background should be preserved, and the final image should remain visually coherent. The paper frames this as distinct from ordinary inpainting because the completion is not determined solely by surrounding pixels; the text prompt must strongly influence the generated content inside the mask [2509.23919].

The paper argues that diffusion-based inpainting methods are limited in this setting because they denoise the **entire image latent** jointly. In that formulation, masked and unmasked regions are processed together, prompt control can be weakened by surrounding context, and background consistency may also be disrupted. It further notes that blending-based corrections do not fully resolve lighting or color seams and add extra cost [2509.23919].

Against this background, Token Painter adopts **MAR** as the base generative framework. The central claim is that MAR is naturally suitable for inpainting because it generates image tokens step by step, predicts only unknown tokens, and leaves known background tokens fixed. The paper writes the generic MAR factorization as

$$
p(S^1,\ldots,S^V)=\prod_{v}^{V} p(S^v \mid T, S^1,\ldots,S^{v-1}),
$$

and adapts it for inpainting to

$$
p(S^1,\ldots,S^k)=\prod_{k}^{K} p(S^k \mid T, I_b, S^1,\ldots,S^{k-1}),
$$

where \(I_b\) denotes background tokens and the union of the generated sets \(S^k\) corresponds to the inpainting tokens [2509.23919].

## 2. MAR adaptation and the failure modes of naive conditioning

Token Painter treats only the masked tokens \(I_p\) as unknown and keeps background tokens \(I_b\) fixed during generation. This is one reason the method is described as **training-free**: it does not retrain the MAR backbone, but changes token handling at inference and adds inference-time modules [2509.23919].

The paper identifies two naive conditioning strategies and treats their failure modes as the immediate motivation for Token Painter. In the **T+B** setting, text tokens and background tokens are both input. According to the paper’s decoder attention visualizations, attention then spreads broadly over the image, especially background regions; the output matches context but follows the prompt weakly. In the **T-only** setting, only text tokens are input; attention becomes concentrated in the masked region, prompt semantics are better preserved, but the generated content is often visually disharmonious with the surrounding image [2509.23919].

This analysis yields the method’s main design premise: background tokens are useful for context, but they can overwhelm text semantics; text tokens preserve semantics, but without context they degrade harmony. A plausible implication is that the central challenge is not merely multimodal conditioning, but the controlled redistribution of attention between semantic and contextual token streams.

## 3. Dual-Stream Encoder Information Fusion

**DEIF** is the encoder-side mechanism that fuses semantic and contextual information into a new guidance representation. The paper first restricts background interaction to tokens near the mask boundary using a dilated mask \(M_d\), and defines a selected background subset as

$$
I'_{b}=(I\odot(M_{d}-M))[:p\cdot N, :].
$$

It then runs two encoder branches:

$$
\begin{aligned}
T_{gb} &= ME(Concat(T,I'_b,I_{p_1}))[:L,:] \in \mathbb{R}^{L\times D}, \\
T_{gt} &= ME(T)[:,:] \in \mathbb{R}^{L\times D}.
\end{aligned}
$$

Here \(T_{gb}\) is the guidance stream enriched with boundary context, while \(T_{gt}\) is the text-only semantic stream [2509.23919].

Before fusion, the two streams are aligned statistically. The paper defines

$$
\begin{aligned}
\gamma_f &= a\cdot\mu(T_{gb}) + (1-a)\cdot\mu(T_{gt}), \\
\zeta_f &= a\cdot\sigma(T_{gb}) + (1-a)\cdot\sigma(T_{gt}),
\end{aligned}
$$

and then normalizes and shifts both streams to the shared distribution:

$$
\begin{aligned}
T'_{gb} &= \gamma_f\cdot\left(\frac{T_{gb}-\mu(T_{gb})}{\sigma(T_{gb})}\right)+\zeta_f, \\
T'_{gt} &= \gamma_f\cdot\left(\frac{T_{gt}-\mu(T_{gt})}{\sigma(T_{gt})}\right)+\zeta_f.
\end{aligned}
$$

The purpose of this alignment is to make the two sources more compatible before fusion [2509.23919].

The actual fusion is performed in the frequency domain. The paper argues that **low-frequency** components are more associated with semantic structure and **high-frequency** components with contextual style and detail. After transforming the aligned streams into frequency space, it uses a modified Gaussian

$$
MG(l)=\exp\left(-\left(\frac{|l-L/2|}{\varphi}\right)^{\tau}\right)
$$

and fuses them as

$$
T_{gf}(l)=(1-MG(l))\cdot T_{gb}+MG(l)\cdot T_{gt}.
$$

The paper describes the fused representation as the final guidance tokens \(T_{gf}\), obtained after inverse transformation. In the reported ablation, the **modified Gaussian** performs better than **linear**, **constant**, and **quadratic** fusion [2509.23919].

## 4. Adaptive Decoder Attention Score Enhancing

After DEIF, Token Painter modifies decoder attention scores through **ADAE**. The decoder input is

$$
X \in \mathbb{R}^{(L+HW)\times D},
$$

formed by concatenating guidance tokens \(T_{gf}\) with image tokens \(I\). Standard attention notation is then used:

$$
Q,K,V \in \mathbb{R}^{(L+HW)\times D'}, \qquad
A = \frac{QK^T}{\sqrt{D'}}.
$$

ADAE selectively amplifies entries of \(A\) rather than changing the model weights [2509.23919].

The first component, **ADAE-G**, strengthens attention from inpainting tokens to guidance tokens. The paper introduces an adaptive coefficient

$$
\alpha = \log_N HW
$$

and defines

$$
A'_{ij}= \begin{cases}
\alpha^{\lambda_1}\cdot A_{ij}, & X_i\in I_p \text{ and } X_j\in T_{gf},\\
A_{ij}, & \text{otherwise}.
\end{cases}
$$

Its stated effect is to improve prompt-detail alignment by forcing masked tokens to attend more strongly to the fused semantic-contextual guidance [2509.23919].

The second component, **ADAE-I**, strengthens interactions among inpainting tokens themselves. The masked tokens are split during generation into unknown tokens \(I_{p_1}\) and predicted tokens \(I_{p_2}\), with \(N_1+N_2=N\), and the paper defines

$$
\beta = \log_{N_2+1}{N_1}.
$$

Attention is then modified as

$$
A'_{ij}= \begin{cases}
\beta^{\lambda_3}\cdot \alpha^{\lambda_2}\cdot A_{ij}, & X_i\in I_{p_1} \text{ and } X_j\in I_{p_2},\\
A_{ij}, & \text{otherwise}.
\end{cases}
$$

This is intended to improve internal coherence and visual quality as more masked tokens become available during autoregressive decoding [2509.23919].

The full inference pipeline is correspondingly staged: encode prompt and image, split image tokens into background and inpainting sets, run the T+B and T-only encoder branches, align and fuse them through DEIF, feed \(T_{gf}\) into the decoder, apply ADAE, autoregressively generate masked tokens, and decode the final latent tokens with VQ-VAE [2509.23919].

## 5. Experimental setting and reported performance

The paper evaluates Token Painter against **SDI**, **HD-Painter**, **FreeCond**, **PowerPaint**, **BrushNet**, and **Meissonic** on two benchmarks. **EditBench** contains **240 annotated images** with loose masks, where captions describe the masked object. **BrushBench** contains **600 text-image pairs** with tight masks similar to segmentation masks, where captions describe the entire image. Evaluation uses **IR**, **HPS**, **PS**, **AS**, **PSNR**, **LPIPS**, **SSIM**, and **CLIP-S** [2509.23919].

On **EditBench**, Token Painter reports the following values: **IR \(-2.49\)**, **PS \(55.37\)**, **HPS \(23.00\)**, **AS \(3.92\)**, **PSNR \(28.03\)**, **LPIPS \(24.92\)**, **SSIM \(9.41\)**, and **CLIP-S \(26.06\)**, where CLIP-S is second best. On **BrushBench**, it reports **IR \(13.01\)**, **PS \(47.90\)**, **HPS \(28.36\)**, **AS \(4.22\)**, **PSNR \(26.39\)**, **LPIPS \(42.27\)**, **SSIM \(8.78\)**, and **CLIP-S \(14.46\)** [2509.23919].

The ablation study is structurally important because it decomposes the contribution of the two main modules. On BrushBench, the paper reports the following progression:

- **Baseline (T+B only)**: IR 4.23, PS 19.47, PSNR 26.26, CLIP-S 6.42  
- **+DEIF**: IR 12.41, PS 44.26, PSNR 26.35, CLIP-S 14.42  
- **+ADAE-G**: IR 12.76, PS 46.28, PSNR 26.27, CLIP-S 14.45  
- **+ADAE-I**: IR 13.01, PS 47.90, PSNR 26.39, CLIP-S 14.46

The paper interprets this as showing that **DEIF** gives the largest improvement, while **ADAE-G** and **ADAE-I** further improve prompt alignment and image quality. It also notes that PSNR changes little because MAR preserves the background tokens [2509.23919].

A hyperparameter study reports final values \(\lambda_1 = 0.3\), \(\lambda_2 = 0.1\), and \(\lambda_3 = 0.03\). Increasing these coefficients improves alignment or structure only up to a point; values that are too large cause distortion or chaotic colors [2509.23919].

## 6. Conceptual position, related systems, and scope

Token Painter belongs to a broader line of research in which autoregressive models operate over structured visual tokens rather than directly over pixels. An earlier example is **"Painter: Teaching Auto-regressive Language Models to Draw Sketches"**, which generates **virtual brush strokes** in string form from text prompts and canvas feedback, using an autoregressive LLM with a visual encoder and residual cross-attention blocks [2308.08520]. Token Painter differs fundamentally in representation and objective: it does not emit stroke descriptions, but instead manipulates latent image tokens within a MAR inpainting pipeline [2509.23919].

It is also related more indirectly to work on visual tokenization such as **"WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction"**, which addresses the trade-off between compression and reconstruction fidelity via **Group-wise Lookup-free Quantization (GQ)** and **Generative Decoding (GD)** [2508.05599]. That work concerns the quality of discrete tokenization and reconstruction, whereas Token Painter is an inference-time method for conditioning masked-token generation on text and context. A plausible implication is that advances in tokenizer quality could affect the upper bound of MAR-based inpainting quality, even though Token Painter itself is not a tokenizer design.

Several misconceptions are explicitly precluded by the paper’s setup. Token Painter is **not** a new pretrained image generator; it **does not finetune** the underlying MAR model. Its contribution lies in inference-time token routing, encoder-side fusion, and decoder attention modification [2509.23919]. Conversely, the paper does not claim universal superiority of MAR over diffusion in all generative settings; its argument is specific to **text-guided image inpainting**, where preserving background tokens and generating only masked tokens is especially advantageous [2509.23919].

The method’s stated limitations are correspondingly narrow but important. Performance depends on the quality and suitability of the base MAR model; the attention-enhancement hyperparameters must be tuned carefully; over-enhancement can cause distortions; and the evaluation is carried out on specific benchmarks, so broader generalization remains an open empirical question [2509.23919]. Within that scope, Token Painter is best understood as a targeted MAR-based inpainting framework that attempts to reconcile prompt fidelity, background preservation, and contextual harmony without retraining the base model.

Source: https://www.emergentmind.com/topics/token-painter