OmniText: Generalist Text-Image Manipulation

• OmniText Framework is a modular, training-free system for controllable text-image manipulation that unifies multiple TIM tasks like editing, insertion, and removal.
• It employs innovative attention manipulations—Self-Attention Inversion and Cross-Attention Reassignment—to suppress unwanted text features and preserve style fidelity in results.
• Its performance is validated on OmniText-Bench, demonstrating superior accuracy and versatility compared to specialist TIM models without the need for retraining.

OmniText Framework denotes a generalist approach to controllable text-image manipulation (TIM), unifying previously fragmented, specialist TIM methods into a modular, training-free solution capable of performing editing, insertion, removal, rescaling, repositioning, and style transfer for text within images—all without retraining or fine-tuning the underlying model. The framework is defined by its manipulation of cross- and self-attention mechanisms during inference, its novel latent optimization procedures, and its coverage of diverse TIM tasks and evaluation settings. The design of OmniText reflects rigorous attention to inference-time control, interpretability, and extensibility, establishing new baselines for both breadth and fidelity in text-image manipulation (Gunawan et al., 28 Oct 2025). OmniText should be distinguished from "OmniText Framework" usage in multimodal MLLM design (e.g., Baichuan-Omni), which applies generic token streams to all modalities, whereas OmniText focuses on training-free, controllable visual text editing.

1. Technical Overview and Architecture

OmniText is centered around a latent diffusion framework built atop TextDiffuser-2 (TextDiff-2), a pretrained text-inpainting diffusion model. It receives as input an image $I$, target text string $T$, target mask $M$, and, for style transfer or controlled editing, a style reference image.

At inference, OmniText applies modular attention manipulations directly to the diffusion process:

• Self-Attention Inversion (SAI): For pixels $i$ inside the mask $m$, self-attention weights $S^l_{i,j}$ at layer $l$ are inverted:

$$S^l_{i, j} = \max_j(S^l_{i, j}) + \min_j(S^l_{i, j}) - S^l_{i, j}$$

This suppresses focus on surrounding text, mitigating the hallucination of unwanted text features.

• Cross-Attention Reassignment (CAR): For regions within the mask, the cross-attention map $C^l_{i,j}$ is reassigned to ignore text tokens:

$$C^l_{i, j} = \begin{cases} 1, & \text{if } (i \in m \wedge j = E_d) \vee (i \notin m \wedge j = S_d) \ 0, & \text{otherwise} \end{cases}$$

Here, $E_d$ and $S_d$ are the end/start description tokens, ensuring background restoration.

These manipulations are non-parametric and applied at test time, requiring no further model fitting.

In addition, for text insertion, editing, or style-based manipulation, an attention-guided latent optimization is executed. The framework leverages a "grid trick" (Editor's term)—a latent arrangement incorporating the result of removal and reference style guidance.

2. Loss Functions and Optimization Strategies

Controllable inpainting is achieved by minimizing a composite loss during early inference diffusion steps, focusing on two new terms:

• Cross-Attention Content Loss ($\mathcal{L}_C$): For each character $c_k$ in the target string, focal loss ensures accurate rendering and placement:

$$\mathcal{L}_C = \sum_{k=1}^N FL(C^l_{i, j = c_k}, m_{c_k})$$

with

$$FL(p, l) = (1-(p \cdot l))^\gamma [ -(l \log p + (1-l) \log (1-p)) ]$$

• Self-Attention Style Loss ($\mathcal{L}_S$): KL-divergence aligns the self-attention profile inside the mask to that induced by the style reference:

$\mathcal{L}_S = D_{KL}(GT, S^l_{i \in m})$

where $GT$ is the pixel-normalized style mask for the reference:

$$GT = \frac{m_{\text{ref}}}{\sum_{j=1}^N (m_{\text{ref}})_j}$$

The overall optimization objective:

$$\mathcal{L} = \lambda_C \mathcal{L}_C + \lambda_S \mathcal{L}_S$$

These losses steer the early-step latent denoising process, updated via Adam optimizer, toward user-specified content ($T$) and style ($m_{\text{ref}}$), without model retraining.

3. Unified Generalist Capabilities

OmniText performs, within one framework, all principal TIM tasks—including those previously treated separately or requiring task-specialized models:

• Text Removal: Via SAI and CAR
• Text Editing: Remove then insert new text at the same location
• Text Insertion: Add text to existing blanks
• Text Rescaling/Repositioning: Remove then insert modified text at new scale/location
• Style-Controlled Editing/Insertion: Guide appearance using arbitrary style references

This contrasts with prior work, which typically supports only one or a subset of these manipulations, often lacking style control or requiring finetuning.

4. Evaluation Protocol: OmniText-Bench

To facilitate systematic, multi-task evaluation, OmniText introduces OmniText-Bench—a benchmark tailored for diverse TIM tasks, domain coverage, and style transfer assessment:

• Dataset Composition: 150 mockup-based sets covering real-world domains (print, apparel, devices, packaging) with precise masks, ground-truth text strings, style references, and outputs for each manipulation task.
• Tasks Covered: Insertion, Removal, Editing, Rescaling, Repositioning, and style-based variants.
• Metrics: Diverse quantitative and qualitative measures, including PSNR, FID, MS-SSIM, accuracy (ACC), normalized edit distance (NED), and MSE.

A plausible implication is increased reproducibility and comparability across future generalist TIM frameworks.

5. Comparative Performance and Limitations

Across established and novel benchmarks, OmniText surpasses generalist inpainting/synthesis methods (AnyText, TextDiff-2, UDiffText, DreamText) and rivals specialist models (TextCtrl, LaMa, ViTEraser) in both removal and editing tasks:

• Text Removal: Cleaner restoration (higher MS-SSIM, PSNR) with fewer artifacts.
• Text Editing: Highest style fidelity and competitive accuracy in rendering text, including orientation, shadow, stroke, and color consistency.
• Novel Manipulations: Only OmniText performs rescaling, repositioning, or style-based editing without retraining.
• Ablation: Both SAI and CAR are necessary for robust removal; content loss ensures accuracy, style loss enables precise transfer.

Limitations are noted:

• Duplicated Letters: Inherited from backbone limitations; more pronounced in loose or large masks.
• Style-Accuracy Tradeoff: Maximizing style fidelity can lower text accuracy.
• Seed Sensitivity: Latent optimization may yield variable results depending on initial noise.
• Script/Length Coverage: Restricted to short, single-line, alphanumeric text; longer, multi-line, or non-Latin scripts require dedicated backbone training.

6. Connections to Broader OmniText Paradigms

OmniText's manipulation of cross-/self-attention and loss-based latent optimization evokes the philosophical shift toward "OmniText Frameworks," in which all modalities (text, image, audio, video) are treated as structured token sequences for unified processing. In MLLMs such as Baichuan-Omni (Li et al., 11 Oct 2024), similar principles of modality textualization and joint attention enable instruction following across modalities, though their technical focus is broader multimodal alignment rather than controllable image text editing. The integration and structured tokenization in OmniParser (Wan et al., 28 Mar 2024) and multimodal prompt fusion in OmniLV (Pu et al., 7 Apr 2025) similarly reflect movement toward flexible, plug-and-play task handling.

7. Summary Table: TIM Task Comparison

Task	AnyText	TextDiff-2	UDiffText	DreamText	TextSSR	TextCtrl	ViTEraser	OmniText
Text Insertion	L	L	L	L	L			✓
Text Editing	L	L	L	L	L	✓		✓
Text Removal						L	✓	✓
Text Rescaling								✓
Text Repositioning								✓
Style Control								✓

L = possible but with major limitations, ✓ = natively supported.

8. Conclusion

OmniText exemplifies a training-free, generalist approach to controllable text-image manipulation. Its innovations in attention map inversion/reassignment and loss-based control enable accurate, style-faithful, and extensible manipulation for all principal TIM tasks, moving beyond limitations of both previous generalist and specialist methods. Its reproducible benchmark and modular losses support future research and application. The framework's technical and conceptual foundation aligns with broader trends in multimodal AI toward unified, text-driven control of perceptual tasks (Gunawan et al., 28 Oct 2025).