Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Abstract: Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone. Diffusion Transformers (DiTs) can natively ingest heterogeneous tokens stemming from texts and images, but they lack mechanisms for determining where and how these tokens should influence the output. We introduce appearance pointers, compact tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing token load. Our approach introduces the first modality-agnostic interface for localized multimodal control in a DiT without retraining the base model from scratch. Across a range of metrics, our single model reaches or surpasses the performance of modality-specific state of the art methods, offering a simple and extensible path toward precise, region-aware, multimodal guidance in generative image synthesis.

• The paper introduces AppearancePointers, a lightweight region-correspondence and aggregation system that routes text, image, or multimodal appearance cues to specified image regions without retraining the base DiT.
• The method achieves leading or competitive results across regional image similarity, DINO-I, CLIP-IQA, and mask adherence, including a 45.44 MIoU score for single-pass multi-subject insertion.
• With roughly 3.33% parameter overhead and diffusion-step-independent pointer computation, the framework supports flexible region counts and zero-shot transfer, but performance declines for tiny faces and more than about ten regions.

Overview

"Appearance Pointers — Multimodal Region Control of Diffusion Transformers" (2607.19344) addresses a specific deficiency in Diffusion Transformer (DiT) based image generators: although modern DiTs such as FLUX natively ingest heterogeneous token streams from text and images, they possess no mechanism for determining where and how those tokens should influence the output. The paper introduces AppearancePointers, compact tokens produced by a lightweight region correspondence network that route a DiT toward the correct appearance cues at user-specified spatial locations. The authors claim, and support experimentally, that this is the first modality-agnostic interface for localized multimodal control in a DiT that does not require retraining the base model, and that a single model reaches or surpasses modality-specific state-of-the-art methods across most evaluation metrics.

The work is positioned against three families of prior approaches: attention manipulation (e.g., DreamRenderer, 3DIS), specialized adapters (ControlNet-style modules, GLIGEN), and inference-time guidance (BoxDiff, gradient-based alignment). The authors argue these are largely U-Net-era, slow, non-robust, and restricted to a single conditioning modality. A capability comparison table shows AppearancePointers as the only method supporting image and text regional conditioning simultaneously, fine and sparse region control, insertion, generation, and multimodal per-region conditioning within one framework.

Method

Given $n$ region–prompt pairs $\mathcal{R} = \{(R_i, P_i)\}$, where each mask $R_i$ is paired with a local prompt $P_i$ drawn from text tokens, reference-image tokens, or both, the goal is to generate an image whose regions follow their local prompts while the remainder follows a global text prompt.

Region correspondence network. Masks are augmented with coordinate information and encoded through the FLUX VAE; images use the VAE encoder and text uses the FLUX Kontext T5 encoder. A Region Correspondence Transformer $\Phi_{RC}$ processes mask, image, and text tokens jointly to produce two semantic feature maps per region, ${}^{I}M_i$ and ${}^{T}M_i$, targeting the image and text streams of the backbone DiT respectively. Two lightweight two-layer mask transformer blocks perform initial alignment with U-DiT-style token downsampling, followed by multimodal self-attention blocks with modality-specific QKV projections. Learnable placeholder tokens handle missing modalities, so the module operates on image+mask, text+mask, or image+text+mask inputs. Crucially, $\Phi_{RC}$ is diffusion-step independent: pointers are computed once per generation rather than per denoising step, and the number of regions can vary dynamically at inference.

Region aggregation. Naively injecting all region tokens would inflate the sequence length under $\mathcal{O}(N^2)$ self-attention. Instead, a Region Aggregation Transformer performs depth-wise attention over the region dimension at each patch location, collapsing the multi-region stack into a single semantic canvas via learnable [CLS] tokens. Two independent aggregation blocks yield the final text and image appearance pointers, which are concatenated into the FLUX Kontext dual streams alongside the raw local prompt tokens.

Supporting components. Because token reduction smooths fine detail, an aggregated boundary map of all region contours is encoded by the VAE and supplied to the DiT. Training uses flow matching with log-normal time sampling, LoRA rank 128 on the new conditional tokens, and Prodigy optimization on 8×A100 GPUs for three days. The added Region Aggregation and Correspondence modules total roughly 400M parameters — a 3.33% increase over the base model. Positional ID resampling (multiplying region IDs by a random integer in [1, 6]) lets the model generalize beyond the seven regions seen during training.

Dataset

Existing personalization datasets lack fine-grained region captions, so the authors construct AppearancePointers-37K through a three-stage pipeline: (i) LLM-driven scene captioning with Qwen 3 producing global, object-level, material, and grounding descriptions; (ii) synthesis with Flux.1 Dev followed by grounding with Grounded SAM; (iii) pose perturbations and material/texture edits via Flux Kontext, filtered by InternVL-based consistency scoring (thresholds 0.84 for pose, 0.79 for texture/material). The dataset spans small, medium, and large object scales and includes novel-viewpoint appearance references.

Experimental results

Evaluation generates 500 images per condition (text-only and image-only) with an average of five regions per image, using CLIP-IQA, global CLIP-T, and four region metrics: CLIP-I, DINO-I, region CLIP-T, and class-agnostic MIoU computed via SAM2 prompting.

For text-conditioned regions, the method achieves the best CLIP-IQA (95.02), CLIP-I (90.40), and DINO-I (56.09), second-best MIoU (40.35), against InstanceDiffusion, DreamRenderer, and Seg2Any. For image-conditioned regions, it outperforms MS-Diffusion and DreamRenderer* (which additionally receives depth supervision) with CLIP-I 93.29, DINO-I 69.31, and MIoU 40.97, while DreamRenderer* retains the best region CLIP-T (27.37). The authors note that baselines often fail to adhere strictly to masks, explaining their lower MIoU despite competitive CLIP-T.

In the multi-subject insertion setting, treating the background image as a region condition extends the framework to editing. Against iterative InsertAnything — a baseline given a conceptual advantage since it inserts objects sequentially — AppearancePointers matches CLIP-I (94.52), wins CLIP-T (27.69 vs 27.46), MIoU (45.44 vs 40.63), and CLIP-IQA (95.47 vs 85.82), while remaining competitive on DINO-I (75.11 vs 76.32). InsertAnything's quality degrades with repeated insertions, whereas the single-pass approach avoids this compounding error.

On the real-world SACap-eval benchmark, zero-shot transfer from synthetic training data is competitive with Seg2Any trained on SACap-1M (e.g., CLIP-IQA 89.27 vs 86.07), and fine-tuning on only 2.3% real data improves region adherence further. In the image-conditioned setting, zero-shot fine-grained control reaches DINO-I 72.63 and MIoU 62.46 versus MS-Diffusion's 46.86 and 53.53. Attention visualizations confirm that pointer tokens attend to the correct conditioning regions during generation.

Ablations attribute gains to individual components: removing region aggregation drops DINO-I from 69.31 to 54.47 (though it slightly raises MIoU to 41.49, since no fusion is required); removing contour guidance drops MIoU from 40.97 to 35.81; removing position ID resampling lowers DINO-I to 67.46; and removing the appearance-pointer mask yields marginal but consistent degradation. Performance scales gracefully with region count up to nine regions, with identity preservation (CLIP-I) stable around 95 while DINO-I declines from 85.22 to 74.80 due to occlusion and boundary overlap. Coarse masks (bounding boxes, ellipses) degrade DINO-I and MIoU relative to fine masks but preserve CLIP-I and aesthetics, indicating robustness to sparse input.

Limitations and open questions

The paper concedes several limitations directly. The method inherits FLUX Kontext's difficulty preserving intricate human facial identities, with identity preservation degrading when faces occupy small spatial regions. Adherence deteriorates at roughly ten or more prescribed regions, producing leakage artifacts and degraded regions. Dataset construction depends on VLM-based filtering, so residual verification errors may propagate into appearance-transfer performance. The authors also acknowledge a potential distribution-shift advantage in benchmarking against baselines trained elsewhere, partially mitigated by excluding distribution-sensitive metrics (FID/KID) and reporting real-data results. Open questions include extending the pointer mechanism to audio, video, 3D, and 4D conditioning modalities, and improving efficiency toward real-time editing.

Conclusion

AppearancePointers provide a lightweight, modular mechanism for spatially grounded multimodal control of DiTs, decoupling what should appear from where it appears without retraining the backbone. With roughly 3% parameter overhead, single-pass computation of pointers, and competitive-or-better results against unimodal specialists on both synthetic and real benchmarks, the approach offers a practical unified interface for region-aware generation and editing, subject to known constraints on facial identity fidelity and scaling to very large numbers of regions.