DiT-based Text Style Transfer

• DiT-based text style transfer models are advanced generative architectures that accurately map content glyphs to a specified style using dual encoders and transformer blocks.
• They integrate content and style features via cross-attention mechanisms to achieve high fidelity in typography and artistic text rendering.
• Demonstrated in systems like UTDesign and FonTS, these models set new performance benchmarks in OCR accuracy, font consistency, and style agreement.

Diffusion Transformer (DiT)-based models for text style transfer constitute a specialized class of generative architectures that enable precise control over both the typography and artistic style of text rendered within images. These models are demonstrated as core components of advanced design frameworks such as UTDesign (Zhao et al., 23 Dec 2025) and FonTS (Shi et al., 2024), where they establish new state-of-the-art performance in style-consistent text synthesis, typography control, and adaptability to multilingual and multimodal usage scenarios. The following exposition systematically details the underlying principles, architectures, training regimes, datasets, evaluation protocols, and extensibility of DiT-based text style transfer systems.

1. Design Objective and System Overview

The primary objective of DiT-based text style transfer models is style-preserving glyph synthesis, where arbitrary sets of content glyphs (target characters to render) are mapped into a designated style as specified by reference glyph samples or external visual prompts. This functionality is critical in design contexts requiring high precision—such as automated poster, banner, or advertisement production—and is further extended to achieve conditional text generation in complex graphic settings (Zhao et al., 23 Dec 2025). FonTS adopts a two-stage pipeline: typography control fine-tuning (TC-FT) for word-level typographic attributes, and a style control adapter (SCA) for visual style injection via reference images (Shi et al., 2024). UTDesign proceeds through a three-stage curriculum: (i) style-transfer DiT training on synthetic data, (ii) multi-modal condition encoder alignment, and (iii) post-training for conditional generation in real designs.

2. Network Architectures and Conditioning Mechanisms

At the core of these systems is the Diffusion Transformer (DiT), an adaptation of the Flux.1-dev and RF-DiT architectures, working in either unconditional or multi-modal conditional regimes. The models accept inputs comprising noisy latent tensors $x_t \in \mathbb{R}^{H'\times W'\times C}$ (denoising targets), reference glyph sets $\mathcal{R}_c$ (content) and $\mathcal{R}_s$ (style), and continuous timestep $t \in [0, 1]$ (Zhao et al., 23 Dec 2025).

For glyph style transfer, UTDesign utilizes dual encoders:

• Content encoder $\mathcal{C}$: DINOv2-Large ViT, projects content glyph images to latent embedding space.
• Style encoder $\mathcal{S}$: CLIP-ViT-Large, encodes style glyph images capturing font metrics, stroke textures, and color distributions.

Fusion DiT blocks interleave cross-attention between noisy latent tokens and projected embeddings of content and style, modulated with learned $\tanh$ gating residuals to control style injection strength. A 3D-RoPE positional encoding scheme with (glyph-index, x, y) indices enables rendering of arbitrary numbers of glyphs simultaneously.

FonTS leverages CLIP text encoders for prompt embeddings $c_{\mathrm{txt}}$, custom ETC-tokens to mark word-level typographic spans, and frozen DiT backbones except for joint text-attention layers. The SCA module performs decoupled joint attention, combining text-conditioned ($Q, K, V$) and image-conditioned ($Q', K', V'$) projections per transformer block. Only image-side adapter matrices $\{W'_k, W'_v\}$ are trainable, mitigating content leakage.

3. Diffusion Formulations and Loss Functions

Both UTDesign and FonTS implement continuous-time diffusion via rectified flow (RF-DiT), departing from classical DDPMs in favor of direct velocity prediction:

• Forward process: $x_t = t x_1 + (1-t) x_0$
• Instantaneous velocity: $v_t = x_1 - x_0$
• Parameterized objective:

$$\mathcal{L}_{\mathrm{rf}} = \mathbb{E}_{x_0,x_1,t,\mathcal{R}_c,\mathcal{R}_s} \Vert v_\theta(x_t, t, \mathcal{R}_c, \mathcal{R}_s) - v_t \Vert_2^2$$

(Zhao et al., 23 Dec 2025).

Flow-matching loss is used in FonTS (Eq. 2):

$$\mathcal{L}_{\mathrm{CFM}}(\theta) = \mathbb{E}_{t, \varepsilon, z_t} \Vert v_\theta(z_t, t) - u_t(z_t \mid \varepsilon) \Vert^2$$

where only selected parameters (e.g., joint text-attention layers) are updated via gradient masking:

$$\theta \leftarrow \theta - \alpha \cdot (\mathbf{M} \odot \nabla_\theta \mathcal{L}_{\mathrm{CFM}})$$

(Shi et al., 2024).

Auxiliary objectives include:

• Style consistency in CLIP-image space:

$$\mathcal{L}_{\mathrm{style}} = \mathbb{E}\left[1-\cos\left(E_{\mathrm{img}}(G), E_{\mathrm{img}}(x_{\mathrm{style}})\right)\right]$$

• Content preservation: $$\mathbb{E}[\Vert E_{\mathrm{txt}}(G) - E_{\mathrm{txt}}(x_{\mathrm{prompt}})\Vert^2]$$

Alignment losses and VAE objectives for RGBA rendering (UTDesign):

$$\mathcal{L}_{\mathrm{align}} = \mathbb{E}\Vert \mathcal{P}(\mathcal{M}(\mathcal{B}, \mathcal{D})) - \mathcal{S}(\mathcal{R}_s) \Vert_2^2$$

$$\mathcal{L}_{\mathrm{vae}} = \mathbb{E}\Vert \mathcal{D}_{\mathrm{vae}}(\mathcal{E}_{\mathrm{vae}}(\hat{G}_{\mathrm{rgb}})) - G_{\mathrm{rgba}} \Vert^2 + \lambda_{\mathrm{lpips}}\mathcal{L}_{\mathrm{lpips}}$$

(Zhao et al., 23 Dec 2025).

4. Dataset Construction and Pretraining Protocols

SynthGlyph (UTDesign) comprises 28.8 million triplets $(\mathcal{R}_c, \mathcal{R}_s, \mathcal{GT})$ synthesized from 4,194 TrueType fonts and 6,857 characters each. Content glyphs (white on gray) and style glyphs (white on noisy backgrounds) are rendered, and ground-truth images draw the content shape in the style’s color, producing RGBA crops. Style references are perturbed via blur, downsampling, and noise for robustness (Zhao et al., 23 Dec 2025).

FonTS constructs word-level typographic datasets using HTML-rendered pages:

• 625 text excerpts × 16 variants (attributes and positions) × 5 fonts ≈ ~50,000 images.
• SC-general: 580,000 general image/text pairs.
• SC-artext: 20,000 artistic text images with GPT-4V-refined captions.
• 100 style descriptions × 99 words = 9,900 prompts for style references (Shi et al., 2024).

5. Training Regimes and Evaluation Metrics

Typical training utilizes 8×A100 GPUs, DeepSpeed ZeRO, learning rates of $1e-5$, AdamW optimizer, and weight decay ($\lambda=1e-2$). FonTS requires 40,000 steps for TC-FT and over 100,000 for SCA (Shi et al., 2024). UTDesign implements three distinct training stages and freezes key components post Stage 1 for subsequent adaptation (Zhao et al., 23 Dec 2025).

Evaluation is conducted via:

• OCR-Acc: fraction of correctly recognized characters (PaddleOCR).
• Word-Acc: correct application of bold/italic/underline versus ground truth (GPT-4o/manual).
• Font-Con/Style-Con: user scoring for font and style agreement.
• CLIP similarity and MARIO-bench scene text rendering.

Table: Summary of Basic and Artistic Text Rendering Performance (FonTS, Table 1) | Method | OCR-Acc (%) | Font-Con (%) | Style-Con (%) | |------------------|-------------|--------------|---------------| | Glyph-ByT5 | 96.36 | 32.73 | — | | TextDiffuser-2 | 42.86 | 1.81 | — | | SD3-medium | 48.05 | 0.91 | 13.64 | | Flux.1-dev | 66.49 | 0.91 | 17.42 | | FonTS (Ours) | 82.85 | 63.64 | 74.43 |

Scene Text Rendering (STR) on MARIO-bench:

• Flux.1-dev: OCR-Acc 24.17%; CLIP 29.93%
• FonTS: OCR-Acc 53.57%; CLIP 31.66% (Shi et al., 2024).

6. Adaptation to New Scripts, Styles, and Use Cases

For new fonts or languages:

1. Create HTML-rendered datasets with ETC-tokens for the target script.
2. Fine-tune only joint text-attention parameters (20–40k steps).
3. Validate with OCR and GPT-derived annotation. Adapting to right-to-left scripts may necessitate adjustment of positional encoding.

Extending SCA to novel styles requires:

1. General style dataset (≥100k image/text pairs).
2. Fine-tune two adapter matrices per block (50–100k steps).
3. At inference, provide style image for CLIP encoding and SCA injection.

Current ETC-tokens handle only bold/italic/underline; expansion to color, size, stroke attributes would require new tokens and datasets. Multi-style mixing and interactive editing are highlighted as open research areas (Shi et al., 2024).

7. Synthesis and Domain Impact

DiT-based text style transfer establishes a technically rigorous approach for high-fidelity, controllable, and edit-friendly text rendering within graphic design. The fusion of deep transformer architectures, large-scale synthetic datasets, multi-modal conditioning, and efficient fine-tuning strategies facilitates versatile and scalable deployment. The demonstrated quantitative improvements over previous and commercial systems substantiate the practical utility of these models for academic, industrial, and creative applications (Zhao et al., 23 Dec 2025, Shi et al., 2024). An important implication is the convergence of text and vision foundation models toward unified design tooling operating robustly across languages, typographic traditions, and artistic domains.