---
title: 'WordCon: Precise Word-Level Typography Control'
url: https://www.emergentmind.com/topics/wordcon
type: topic
---

# WordCon: Precise Word-Level Typography Control

Searching arXiv for the WordCon paper and a closely related scene-text rendering context paper.
WordCon is a method for **word-level typography control in scene text rendering** that combines a word-annotated training corpus, a **Text-Image Alignment (TIA)** framework, and a **hybrid parameter-efficient fine-tuning (PEFT)** strategy for diffusion-transformer-based text-to-image systems. Its stated objective is to make typographic attributes such as bold, italic, underline, and font choices attach to the correct word within a generated scene image rather than diffuse across neighboring tokens or collapse into prompt-level style transfer. The work identifies the central failure mode of prior systems as **word-level misalignment**: cross-attention in diffusion transformers does not adequately disentangle which prompt word should correspond to which local image region [2506.21276].

## 1. Task definition and problem setting

WordCon addresses a narrow but technically demanding subproblem of scene text rendering: **precise word-level control** over typography inside images. In this setting, the model must not only render readable text, but also map individual prompt words to the correct spatial support and apply the intended typographic attribute to that support only. The paper explicitly contrasts this with the behavior of existing text-to-image models, including state-of-the-art commercial systems, which often fail when a prompt requires selective styling of one word but not others [2506.21276].

A common misconception is that this problem is primarily one of richer prompting. The paper instead attributes the failure to the structure of the model’s internal alignment: cross-attention mechanisms in diffusion transformers do not effectively disentangle the correspondence between specific words in the prompt and their precise locations in the image. In that formulation, the bottleneck is not merely lexical specification but **cross-modal binding** between token-level text representations and local visual regions.

The method is therefore positioned as a control mechanism for **scene text rendering**, not as a generic text stylization layer detached from image synthesis. Its target operating regime includes generated scenes in which text must coexist with background content, perspective variation, and heterogeneous layouts.

## 2. Text-Image Alignment (TIA) framework

The **Text-Image Alignment (TIA)** framework is the supervisory core of WordCon. For a scene image and a textual prompt, TIA uses **grounding models** to produce **word-level segmentation masks**, namely spatial regions corresponding to individual words in the prompt. These masks serve as explicit anchors for supervision during training and convert an otherwise weakly aligned text-to-image objective into a word-region aligned optimization problem [2506.21276].

Architecturally, TIA sits between the grounding system and the text-to-image backbone. The grounding model decodes the prompt word by word and delineates corresponding spatial regions in the image. Those regions are then used to supervise the diffusion model so that its latent predictions and attention maps are driven toward the correct local supports. The paper describes this as a **dual flow**: grounding outputs guide the text-to-image model, thereby improving word-region correspondence.

The significance of TIA lies in the form of supervision it introduces. Rather than treating text rendering as a global text-conditioned generation task, it imposes **fine-grained cross-modal supervision** at the word level. This suggests a shift from prompt-level conditioning toward token-local alignment as the relevant control variable for typography.

## 3. Word-level controlled scene text dataset

WordCon is trained with a newly constructed **word-level controlled scene text dataset**. The dataset is created by a synthesis pipeline that begins with a corpus of diverse text content, renders text via an **HTML renderer** with typographic attributes and transparent background, and then composites the rendered text onto scene backgrounds generated by **Flux.1**. The compositing process varies positions, scales, and contexts, producing scene diversity rather than isolated text exemplars [2506.21276].

The resulting dataset contains **28k samples**, spans **five font types (serif, non-serif, sans-serif, script, monospace)**, includes texts ranging **3–70 characters**, and maintains **individual word segmentation masks** for each image. The annotations are therefore mask-level and per-word, which is the supervision required by TIA and the downstream loss terms.

The paper characterizes this resource as the **first large word-level controlled scene text dataset with mask-level word annotation**. In practical terms, its importance is twofold. First, it supplies the spatial supervision necessary for training word-disentangled text rendering models. Second, it provides an evaluation substrate for controllability, OCR behavior, and visual quality under word-specific typographic constraints.

## 4. Hybrid PEFT design

WordCon’s adaptation strategy is a **hybrid PEFT** method that selectively reparameterizes only the parameters most relevant to text-image alignment. Concretely, it **selectively reparameterizes only the text-attention components**, specifically the **key and value projections** in the **joint cross-attention blocks (Double-DiT layers)** of the backbone. The paper motivates this design by arguing that these parameters govern the alignment between words and image regions [2506.21276].

The “hybrid” aspect combines two elements. The first is **selective fine-tuning**, which restricts adaptation to alignment-critical components instead of updating the full model. The second is **reparameterization (LoRA-like)** through learnable low-rank adapters. This yields a method described as both **computationally efficient** and **portable**.

Portability is treated as a first-class property. The paper states that the resulting lightweight modules can be inserted into diverse compatible pipelines, including **artistic text rendering**, **text editing**, and **image-conditioned text rendering**. It also reports compatibility with artistic LoRAs, with text-editing pipelines such as **Flux-fill**, and with image-conditioned settings using **canny/depth/subject** cues. A plausible implication is that WordCon is designed less as a single end-to-end model and more as a reusable alignment module for DiT-based scene text systems.

## 5. Optimization objectives and word disentanglement

WordCon introduces two supervisory losses beyond the baseline generation objective: a **masked loss at the latent level** and a **joint-attention loss** at the feature level. Together they are intended to force the model to learn the text region preferentially and to disentangle attention between different words [2506.21276].

The **masked loss** replaces the original conditional flow matching objective with a loss that restricts gradients to the controlled word regions:
$$
\mathcal{L}_{mask} = \mathbb{E}_{t, p_t(z | \epsilon), p(\epsilon)} \left\| M_k \big( v_\Theta(z, t) - u_t(z | \epsilon) \big) \right\|_2^2
$$
Here, \(M_k\) is the union of the pixel regions corresponding to the \(k\) controlled words, \(v_\Theta\) is the predicted vector field, and \(u_t\) is the conditional ground-truth flow. Pixels outside the controlled text region are zeroed out by the mask. The stated effect is to focus learning on typography localization rather than background reconstruction.

The **joint-attention loss** supervises token-specific attention maps against word-specific masks:
$$
\mathcal{L}_{attn} = \mathbb{E}_{z, c, t} \left\| J_{attn}(z_t,\psi_{\vartheta}(c)_i) - M_i \right\|_2^2
$$
In this expression, \(J_{attn}(z_t,\psi_{\vartheta}(c)_i)\) is the joint-attention map between the latent state and the \(i\)-th word token, and \(M_i\) is the ground-truth mask for that word. The supervision is intended to regularize the model so that each prompt word attends to its own region rather than leaking across adjacent tokens.

The total loss is
$$
\mathcal{L}_{total} = \mathcal{L}_{mask} + \lambda_{attn} \mathcal{L}_{attn}
$$
with \(\lambda_{attn} = 0.01\). In the paper’s formulation, this combination couples spatial localization and token-region disentanglement. This suggests that controllable scene text rendering depends on supervising both **where** the model updates and **which token** is responsible for each update.

## 6. Empirical results, applications, and limitations

The reported evaluation compares WordCon with **SD3, FLUX, Recraft, Gemini, GPT4o-img, and Ideogram** along three axes: **controllability**, **image quality**, and **OCR accuracy**. WordCon achieves the highest reported controllability scores, with **80.95%** for type control, **71.43%** for word control, and **71.43%** for total word-plus-type control. The summary also notes that the second-best total score is **52.38%** by Ideogram. For OCR, WordCon reports **83.14% precision** and **81.95% recall**, compared with **85%** and **83.65%** for GPT4o [2506.21276].

The ablation study isolates the contribution of the loss terms to **total controllability**:

| Setting | Total controllability |
|---|---:|
| Original, no finetuning | 8.67% |
| Finetuning with vanilla loss | 27.22% |
| Adding masked loss | 42.67% |
| Adding joint-attention loss | 66.67% |

These results are used to argue that both losses are functionally important: vanilla fine-tuning improves control over the unfine-tuned backbone, masked supervision further concentrates learning on text regions, and joint-attention supervision yields the largest additional gain in word-level control.

The qualitative evaluation reports three recurrent failure modes in competing systems: applying style to the wrong word, applying the correct style to an incorrect word, or failing to render the style at all. WordCon is reported to support **multiple words in a prompt** and to maintain competitive image aesthetics and distortion behavior. A **user study with 20 expert users** ranks it highest overall, especially for controllability.

The method’s application envelope includes **artistic text rendering**, **text editing and placement control**, **image-conditioned rendering**, and **font selection and mixing**. It is also reported to generalize to **non-square images at inference**. The main limitation stated in the paper concerns prompts containing repeated identical words: when the same word occurs multiple times, the model tends to style **all instances**, not just one. That limitation is informative because it shows that word-level control is not yet equivalent to unambiguous **instance-level** control.

Source: https://www.emergentmind.com/topics/wordcon