WordCon: Precise Word-Level Typography Control

• WordCon is a method for precise word-level typography control in scene text rendering, aligning text tokens to specific image regions using a TIA framework.
• It utilizes a novel 28k sample dataset with per-word masks and a hybrid PEFT strategy that fine-tunes key cross-attention components for enhanced stylistic control.
• Empirical evaluations show WordCon achieves high controllability (80.95% type control) and superior OCR performance, outpacing prior state-of-the-art models.

Searching arXiv for the WordCon paper and a closely related scene-text rendering context paper. WordCon is a method for word-level typography control in scene text rendering that combines a word-annotated training corpus, a Text-Image Alignment (TIA) framework, and a hybrid parameter-efficient fine-tuning (PEFT) strategy for diffusion-transformer-based text-to-image systems. Its stated objective is to make typographic attributes such as bold, italic, underline, and font choices attach to the correct word within a generated scene image rather than diffuse across neighboring tokens or collapse into prompt-level style transfer. The work identifies the central failure mode of prior systems as word-level misalignment: cross-attention in diffusion transformers does not adequately disentangle which prompt word should correspond to which local image region (Shi et al., 26 Jun 2025).

1. Task definition and problem setting

WordCon addresses a narrow but technically demanding subproblem of scene text rendering: precise word-level control over typography inside images. In this setting, the model must not only render readable text, but also map individual prompt words to the correct spatial support and apply the intended typographic attribute to that support only. The paper explicitly contrasts this with the behavior of existing text-to-image models, including state-of-the-art commercial systems, which often fail when a prompt requires selective styling of one word but not others (Shi et al., 26 Jun 2025).

A common misconception is that this problem is primarily one of richer prompting. The paper instead attributes the failure to the structure of the model’s internal alignment: cross-attention mechanisms in diffusion transformers do not effectively disentangle the correspondence between specific words in the prompt and their precise locations in the image. In that formulation, the bottleneck is not merely lexical specification but cross-modal binding between token-level text representations and local visual regions.

The method is therefore positioned as a control mechanism for scene text rendering, not as a generic text stylization layer detached from image synthesis. Its target operating regime includes generated scenes in which text must coexist with background content, perspective variation, and heterogeneous layouts.

2. Text-Image Alignment (TIA) framework

The Text-Image Alignment (TIA) framework is the supervisory core of WordCon. For a scene image and a textual prompt, TIA uses grounding models to produce word-level segmentation masks, namely spatial regions corresponding to individual words in the prompt. These masks serve as explicit anchors for supervision during training and convert an otherwise weakly aligned text-to-image objective into a word-region aligned optimization problem (Shi et al., 26 Jun 2025).

Architecturally, TIA sits between the grounding system and the text-to-image backbone. The grounding model decodes the prompt word by word and delineates corresponding spatial regions in the image. Those regions are then used to supervise the diffusion model so that its latent predictions and attention maps are driven toward the correct local supports. The paper describes this as a dual flow: grounding outputs guide the text-to-image model, thereby improving word-region correspondence.

The significance of TIA lies in the form of supervision it introduces. Rather than treating text rendering as a global text-conditioned generation task, it imposes fine-grained cross-modal supervision at the word level. This suggests a shift from prompt-level conditioning toward token-local alignment as the relevant control variable for typography.

3. Word-level controlled scene text dataset

WordCon is trained with a newly constructed word-level controlled scene text dataset. The dataset is created by an overview pipeline that begins with a corpus of diverse text content, renders text via an HTML renderer with typographic attributes and transparent background, and then composites the rendered text onto scene backgrounds generated by Flux.1. The compositing process varies positions, scales, and contexts, producing scene diversity rather than isolated text exemplars (Shi et al., 26 Jun 2025).

The resulting dataset contains 28k samples, spans five font types (serif, non-serif, sans-serif, script, monospace), includes texts ranging 3–70 characters, and maintains individual word segmentation masks for each image. The annotations are therefore mask-level and per-word, which is the supervision required by TIA and the downstream loss terms.

The paper characterizes this resource as the first large word-level controlled scene text dataset with mask-level word annotation. In practical terms, its importance is twofold. First, it supplies the spatial supervision necessary for training word-disentangled text rendering models. Second, it provides an evaluation substrate for controllability, OCR behavior, and visual quality under word-specific typographic constraints.

4. Hybrid PEFT design

WordCon’s adaptation strategy is a hybrid PEFT method that selectively reparameterizes only the parameters most relevant to text-image alignment. Concretely, it selectively reparameterizes only the text-attention components, specifically the key and value projections in the joint cross-attention blocks (Double-DiT layers) of the backbone. The paper motivates this design by arguing that these parameters govern the alignment between words and image regions (Shi et al., 26 Jun 2025).

The “hybrid” aspect combines two elements. The first is selective fine-tuning, which restricts adaptation to alignment-critical components instead of updating the full model. The second is reparameterization (LoRA-like) through learnable low-rank adapters. This yields a method described as both computationally efficient and portable.

Portability is treated as a first-class property. The paper states that the resulting lightweight modules can be inserted into diverse compatible pipelines, including artistic text rendering, text editing, and image-conditioned text rendering. It also reports compatibility with artistic LoRAs, with text-editing pipelines such as Flux-fill, and with image-conditioned settings using canny/depth/subject cues. A plausible implication is that WordCon is designed less as a single end-to-end model and more as a reusable alignment module for DiT-based scene text systems.

5. Optimization objectives and word disentanglement

WordCon introduces two supervisory losses beyond the baseline generation objective: a masked loss at the latent level and a joint-attention loss at the feature level. Together they are intended to force the model to learn the text region preferentially and to disentangle attention between different words (Shi et al., 26 Jun 2025).

The masked loss replaces the original conditional flow matching objective with a loss that restricts gradients to the controlled word regions:

$$\mathcal{L}_{mask} = \mathbb{E}_{t, p_t(z | \epsilon), p(\epsilon)} \left\| M_k \big( v_\Theta(z, t) - u_t(z | \epsilon) \big) \right\|_2^2$$

Here, $M_k$ is the union of the pixel regions corresponding to the $k$ controlled words, $v_\Theta$ is the predicted vector field, and $u_t$ is the conditional ground-truth flow. Pixels outside the controlled text region are zeroed out by the mask. The stated effect is to focus learning on typography localization rather than background reconstruction.

The joint-attention loss supervises token-specific attention maps against word-specific masks:

$$\mathcal{L}_{attn} = \mathbb{E}_{z, c, t} \left\| J_{attn}(z_t,\psi_{\vartheta}(c)_i) - M_i \right\|_2^2$$

In this expression, $J_{attn}(z_t,\psi_{\vartheta}(c)_i)$ is the joint-attention map between the latent state and the $i$-th word token, and $M_i$ is the ground-truth mask for that word. The supervision is intended to regularize the model so that each prompt word attends to its own region rather than leaking across adjacent tokens.

The total loss is

$$\mathcal{L}_{total} = \mathcal{L}_{mask} + \lambda_{attn} \mathcal{L}_{attn}$$

with $M_k$0. In the paper’s formulation, this combination couples spatial localization and token-region disentanglement. This suggests that controllable scene text rendering depends on supervising both where the model updates and which token is responsible for each update.

6. Empirical results, applications, and limitations

The reported evaluation compares WordCon with SD3, FLUX, Recraft, Gemini, GPT4o-img, and Ideogram along three axes: controllability, image quality, and OCR accuracy. WordCon achieves the highest reported controllability scores, with 80.95% for type control, 71.43% for word control, and 71.43% for total word-plus-type control. The summary also notes that the second-best total score is 52.38% by Ideogram. For OCR, WordCon reports 83.14% precision and 81.95% recall, compared with 85% and 83.65% for GPT4o (Shi et al., 26 Jun 2025).

The ablation study isolates the contribution of the loss terms to total controllability:

Setting	Total controllability
Original, no finetuning	8.67%
Finetuning with vanilla loss	27.22%
Adding masked loss	42.67%
Adding joint-attention loss	66.67%

These results are used to argue that both losses are functionally important: vanilla fine-tuning improves control over the unfine-tuned backbone, masked supervision further concentrates learning on text regions, and joint-attention supervision yields the largest additional gain in word-level control.

The qualitative evaluation reports three recurrent failure modes in competing systems: applying style to the wrong word, applying the correct style to an incorrect word, or failing to render the style at all. WordCon is reported to support multiple words in a prompt and to maintain competitive image aesthetics and distortion behavior. A user study with 20 expert users ranks it highest overall, especially for controllability.

The method’s application envelope includes artistic text rendering, text editing and placement control, image-conditioned rendering, and font selection and mixing. It is also reported to generalize to non-square images at inference. The main limitation stated in the paper concerns prompts containing repeated identical words: when the same word occurs multiple times, the model tends to style all instances, not just one. That limitation is informative because it shows that word-level control is not yet equivalent to unambiguous instance-level control.