Token-Level Inpainting Module

• Token-Level Inpainting Modules are components that use discrete tokenization and autoregressive or parallel generation to synthesize missing image regions.
• They fuse semantic prompts with spatial context via dual-stream architectures and adaptive attention, enhancing local detail and global coherence.
• Recent methods demonstrate state-of-the-art performance by integrating loss functions, wavelet transforms, and progressive patch synthesis for optimal inpainting.

Token-level inpainting modules are specialized architectural components within modern image inpainting models that explicitly structure the inpainting process as the synthesis, prediction, and fusion of discrete semantic units—tokens—corresponding to spatial or latent partitions of the visual input. These modules enable controllable and high-fidelity restoration of missing image regions by leveraging tokenization strategies, autoregressive or parallel generation schemes, and guided fusion mechanisms. Recent advances employ token-level inpainting to balance local detail, global semantic coherence, and explicit conditioning (e.g., by text or explicit visual context) through designs that blend latent quantization, transformer-based attention, state-space models, wavelet mixing, and adaptive statistical alignment.

1. Discrete Tokenization and Autoregressive Modeling

Token-level inpainting typically operates on discrete representations derived from the input image using quantization schemes such as VQ-VAE. The image is partitioned into a grid of tokens $I \in \mathbb{R}^{H \times W \times D}$, and auxiliary modalities (e.g., text) are similarly tokenized ($T \in \mathbb{R}^{L \times D}$) (Jiang et al., 28 Sep 2025). Within the mask autoregressive (MAR) framework, only tokens within the masked region $I_p$ are generated, while known context tokens $I_b$ remain fixed. The generation proceeds in ordered steps:

$$p(S^1, ..., S^K) = \prod_{k=1}^K p(S^k \mid T, I_b, S^1, ..., S^{k-1}),$$

where each $S^k$ denotes a subset of inpainting tokens.

Autoregressive mechanisms are used in both pure transformer models and hybrid architectures that incorporate state-space dynamics, as in Mamba × Transformer (MxT), which processes sequences in $O(L)$ time while retaining long-range dependencies (Chen et al., 2024). Models such as "Token Painter" leverage the explicit token grid to enable local controllability and stable background preservation during inpainting (Jiang et al., 28 Sep 2025).

2. Guidance Fusion and Dual-Stream Architectures

Effective inpainting requires integrating semantic prompt signals with local visual context. Modules like Dual-Stream Encoder Information Fusion (DEIF) in "Token Painter" instantiate this via a two-fold process:

• Constructing a "semantic" stream from text-only guidance, and a "contextual" stream from text concatenated with selected background tokens near the mask.
• Aligning mean and variance statistics across both streams using a weighted blending parameter $a$ (typically $a=0.3$), followed by frequency-domain fusion. Specifically, low-frequency bands (semantics) rely more on the text-only stream, while high-frequency components (structure/texture) incorporate background context (Jiang et al., 28 Sep 2025).

Alternative approaches, such as the Structure–Texture Matching Attention in (Liu et al., 2022), combine direct global self-attention over texture tokens with a structural bridge via already inpainted (or known) patches, fusing multiple attention maps per patch for robust token selection.

3. Adaptive Attention and Score Enhancement

To maximize alignment between prompt, background, and generated tokens, modules adapt attention computation at the token level. The Adaptive Decoder Attention Score Enhancing (ADAE) mechanism, for example, dynamically boosts attention weights between:

• Masked-region tokens and fused guidance tokens (ADAE–G), and
• Tokens within the masked region, enhancing dependencies among partially generated tokens (ADAE–I).

This is achieved through adaptive exponentiated coefficients,

$A'_{ij} = \begin{cases} \alpha^{\lambda_1} A_{ij} & \text{if $X_i \in I_p$,$X_j \in T_{gf}$} \ (\alpha^{\lambda_2} \beta^{\lambda_3}) A_{ij} & \text{if $X_i \in I_{p,1}$,$X_j \in I_{p,2}$} \ A_{ij} & \text{otherwise,} \end{cases}$

where $\alpha$ and $\beta$ adapt as a function of current and remaining tokens, with exponents $\lambda_1, \lambda_2, \lambda_3$ set by cross-validation (Jiang et al., 28 Sep 2025). These enhancements intensify prompt-adherence and intra-region consistency, particularly in late-stage generation when ambiguity is highest.

4. Token-Mixing Beyond Transformers: Wavelet and State-Space Approaches

Token-level inpainting modules are not exclusive to transformer-based designs. WavePaint (Jeevan et al., 2023) employs "WaveMix" blocks, which perform per-channel 2D Haar wavelet transforms, splitting each feature map into sub-bands (LL, LH, HL, HH) for spatial and multi-resolution mixing. These modules process images as collections of tokens, facilitating token-level mixing with minimal parameter overhead and enabling efficient, mask-aware propagation of contextual information (Jeevan et al., 2023).

In MxT, the Hybrid Module parallelizes a Mamba SSM branch for pixel-level long-range modeling (flattening the feature tensor into a length-$L$ sequence, adding fixed positional embeddings, projecting into body/gate branches, SSM recurrence, and then gating), with a Spatially Reduced Self-Attention branch for global, patchwise context (Chen et al., 2024). Their sum passes through a global broadcasting normalization, which rebroadcasts the spatial mean across locations to stabilize and refine token features.

5. Progressive Inpainting, Patch Vocabulary, and Probabilistic Selection

Instead of parallel filling, some token-level modules employ progressive, patch-wise synthesis. For example, (Liu et al., 2022) fills masked patches iteratively: at each step, a patch’s token is predicted by matching (i) direct attention to global texture tokens and (ii) structure-bridged attention via known patches. All candidate patch tokens are scored by their cumulative attention weights; the most probable is inserted into the output in a “probabilistic diffusion” fashion. This ensures nonlocal, semantically grounded token selection at each iteration.

6. Loss Functions and Objective Design

Token-level inpainting modules are supervised by combinations of objectives targeting pixel-level accuracy, perceptual fidelity, and holistic visual realism. Common losses across recent work include:

• $\ell_1$ and $\ell_2$ (L1, L2) errors masked to the inpainting region (Jeevan et al., 2023, Chen et al., 2024).
• Perceptual losses computed on VGG feature activations over the masked region (Liu et al., 2022, Chen et al., 2024).
• Style losses on Gram matrices for maintaining statistical realism (Liu et al., 2022, Chen et al., 2024).
• Adversarial losses with discriminator networks only in select models and upsampling stages (Liu et al., 2022, Chen et al., 2024). Losses are aggregated with dataset-tuned weights; for example, (Chen et al., 2024) uses $$\{\alpha_1,\alpha_2,\alpha_3,\alpha_4\} = \{1, 250, 0.1, 0.001\}$$ for L1, style, perceptual, and adversarial terms respectively.

7. Comparative Performance and Ablation Studies

Token-level inpainting has yielded state-of-the-art performance in both text-guided and unconditional settings. Empirical results from "Token Painter" on the BrushBench benchmark demonstrate that DEIF yields the dominant improvements in prompt alignment and image quality, while the two-stage ADAE further enhances local detail and intra-hole structural consistency (Jiang et al., 28 Sep 2025). For example:

Inpainting Module	IR	PS	PSNR	CLIP-S
Baseline (no DEIF/ADAE)	4.23	19.47	26.26	6.42
+DEIF only	12.41	44.26	—	14.42
+ADAE–G	12.76	46.28	—	14.45
+ADAE–I	13.01	47.90	—	14.46

WavePaint achieves comparable FID and LPIPS metrics to much larger GANs and diffusion models while being dramatically more parameter- and memory-efficient (Jeevan et al., 2023). In MxT, the Hybrid Module boosts PSNR, SSIM, and LPIPS over both transformer-only and CNN-only baselines, with dual-level fusion modules yielding the highest holistic gains (Chen et al., 2024).

8. Synthesis and Emerging Directions

Token-level inpainting modules provide a unifying principle across text-guided, patchwise, and pixelwise image reconstruction architectures. By aligning token-based semantic reasoning with spatial context propagation, these modules enable efficient, high-fidelity, and controllable restoration of complex visual structures. Current trends include modular fusion of orthogonal mechanisms (e.g., state-space and attention), frequency-domain aggregation, and explicit statistical alignment for improved prompt adherence and context harmony (Jiang et al., 28 Sep 2025, Chen et al., 2024).

Ongoing research focuses on extending token-level techniques to multi-modal and high-resolution regimes, further reducing computational demands, and exploring new token-mixing strategies beyond standard transformer and convolutional constructs. Continued benchmarking with ablation studies across diverse datasets is likely to delineate performance frontiers and architectural trade-offs (Jiang et al., 28 Sep 2025, Jeevan et al., 2023, Liu et al., 2022, Chen et al., 2024).