TextCtrl: Diffusion-based Scene Text Editing

• TextCtrl is a diffusion-based framework for Scene Text Editing that explicitly separates fine-grained glyph structure and style to ensure high-fidelity modifications in real-world images.
• It introduces a novel Glyph-adaptive Mutual Self-Attention (GaMuSa) mechanism to dynamically blend reconstruction and editing cues for precise content and style transfer.
• The framework outperforms prior GAN and diffusion methods on metrics like SSIM, FID, and text accuracy, setting new benchmarks for scene text editing evaluation.

TextCtrl is a diffusion-based framework for Scene Text Editing (STE) that achieves high-fidelity content modification and robust style preservation in real-world text-containing images. The method is distinguished by its explicit disentanglement of fine-grained style and glyph structure, as well as a novel Glyph-adaptive Mutual Self-Attention (GaMuSa) mechanism that efficiently transfers intricate style priors from the source image during inference. TextCtrl defines new standards for STE evaluation and outperforms both GAN-based and previous diffusion-based editing approaches across multiple style and text accuracy metrics (Zeng et al., 2024).

1. Scene Text Editing and Motivations

Scene Text Editing (STE) refers to altering text regions in natural images—such as signs or posters—while maintaining the intrinsic visual style (including font, color, background, geometric distortion) and ensuring the output string matches the user-specified target. Prior approaches relied on GAN-based decompositions (removal, inpainting, fusion), which struggle with generalization and introduce artifacts. Latent diffusion models improved inpainting quality, but prior work did not fully resolve style drift or glyph formation errors owing to weak alignment between content and style prompts. TextCtrl explicitly targets these deficits by introducing dual prior guidance—separate style and glyph structure representations—at both training and inference.

2. Fine-grained Style and Glyph Disentanglement

A central principle in TextCtrl is the construction of two distinct embedding streams:

• Glyph Structure Encoder (𝒯): For any target string $C_\mathrm{text}$, a Transformer encoder $\mathcal{T}$ computes a sequence of glyph-structure embeddings, $C_\mathrm{struct} \in \mathbb{R}^{L \times d}$, intended to encode precise character shapes, agnostic to font but robust to morphological detail. Pre-training uses a CLIP-based contrastive loss between $C_\mathrm{struct}[u]$ and the output of a frozen vision encoder applied to synthesized glyph images, with font-variance augmentation to ensure shape-centric representation:

$$L_\mathrm{clip} = -\cos(C_\mathrm{struct}[u], f_\mathrm{vis}(\mathrm{glyph}_u))$$

• Style Encoder (𝒮): A Vision Transformer backbone $\mathcal{S}$ maps the source image $I_\mathrm{source}$ to a set of style tokens $C_\mathrm{style}$, split into $c_\mathrm{texture}$ and $c_\mathrm{spatial}$. Style and spatial cues are disentangled via four auxiliary objectives: colorization, font-shape transfer, text removal, and text segmentation, with the composite loss

$\mathcal{T}$0

These modules provide orthogonal priors—structure for content accuracy, and style for visual coherence—addressing the entanglement shortcomings observed in previous STE systems.

3. Glyph-adaptive Mutual Self-Attention (GaMuSa)

The GaMuSa mechanism enhances style consistency at inference by dynamically blending self-attention activations from two diffusion branches:

• Reconstruction branch: Latent codes from the source image $\mathcal{T}$1 are obtained via DDIM inversion, then denoised to yield attention key/value tensors $\mathcal{T}$2.
• Editing branch: Denoising from random noise $\mathcal{T}$3 produces $\mathcal{T}$4 conditioned on the new text. At prescribed intervals, the intermediate editing image is decoded and a recognizer computes the embedding $\mathcal{T}$5, which is compared (cosine similarity) to the target glyph embedding $\mathcal{T}$6. The mixing coefficients

$\mathcal{T}$7

yield fused attention maps $\mathcal{T}$8 and likewise for $\mathcal{T}$9, replacing baseline self-attention in the editing branch. This adaptive fusion aligns editing progression to structural accuracy and style conformity.

4. Model Architecture, Training Objectives, and Guidance

TextCtrl is built around a Stable Diffusion VAE encoder/decoder paired with a UNet denoiser. Glyph structure embeddings replace/default the key/value roles in cross-attention at each UNet layer, ensuring text structure is precisely imposed. Style tokens are injected at UNet skip connections and in the midblock for fine style control. Training employs a multi-objective loss:

• Denoising loss: Standard diffusion $C_\mathrm{struct} \in \mathbb{R}^{L \times d}$0 error between predicted and true noise.
• Reconstruction loss: Combines perceptual, style, and direct pixel differences, e.g.,

$C_\mathrm{struct} \in \mathbb{R}^{L \times d}$1

• OCR/language loss: Cross-entropy between recognized and ground-truth output. No adversarial losses are used; all supervision and guidance is diffusion-based or extracted from auxiliary perceptual modules. Classifier-free guidance is implemented by stochastically dropping structure tokens during training and combining unconditional and conditional outputs at test time:

$C_\mathrm{struct} \in \mathbb{R}^{L \times d}$2

5. Evaluation: ScenePair Benchmark and Quantitative Results

Because existing evaluation protocols confound style and text accuracy or use only synthetic data, the ScenePair dataset was constructed. It contains 1,280 pairs of real-world text crops matched in style and geometry, with rich annotations (bounding quads, transcriptions, positions). The principal metrics used are:

• Style fidelity: SSIM, PSNR, MSE, FID, evaluated both on the edited crop and the entire image.
• Text accuracy: Word-level recognition accuracy (ACC) and Normalized Edit Distance (NED). Comparisons on ScenePair demonstrate that TextCtrl surpasses the closest GAN and diffusion baselines:

Method	SSIM (×10⁻²)	FID ↓	ACC (%)	NED
SRNet	26.66	49.22	17.84	0.478
MOSTEL	27.45	49.19	37.69	0.557
AnyText	30.73	51.79	51.12	0.734
TextCtrl	37.56	43.78	84.67	0.936

Ablations demonstrate the necessity of the glyph encoder (e.g. substituting a CLIP text encoder reduces accuracy by over 70 percentage points), the importance of multi-task pretraining for style embeddings, and the quantitative impact of the GaMuSa inference mechanism.

6. Limitations, Open Problems, and Future Directions

TextCtrl currently supports word- and short-phrase-level editing under modest geometric distortion but is challenged by arbitrary-length, complex-curved text and extreme perspectives, due to the absence of explicit geometric alignment modules (e.g. TPS grid predictors). Evaluation metrics such as SSIM and FID are imperfect proxies for style coherence. Open directions involve integrating geometric priors, devising improved style evaluation metrics, and extending TextCtrl to temporal domains (video, sequential frame consistency) and multi-lingual or ornate scripts.

7. Context Within the Scene Text Editing Landscape

TextCtrl establishes a new paradigm where explicit style-structure disentanglement and adaptive attention strategies eliminate “style drift” and glyph anomalies characteristic of previous methodologies (Zeng et al., 2024). The introduction of the ScenePair benchmark provides a realistic standard for comparing both content fidelity and style preservation in STE, forming a foundation for further innovation in content-aware, style-consistent text editing systems.