Multi-Modal Typography

Updated 5 July 2026

Multi-modal typography is a multidisciplinary field that combines visual, linguistic, structural, and acoustic modalities to create dynamic and accessible text renderings.
It is applied in tasks like poster generation, animated captioning, cross-language design transfer, and integrated document layout to fuse creative design with computational control.
Recent systems leverage diffusion models, GANs, and hybrid architectures to balance stylized creativity, legibility, and robust multi-modal conditioning.

Multi-modal typography denotes typographic systems that condition, render, animate, edit, retrieve, or evaluate text by jointly using multiple modalities and representations, including background imagery, text semantics, glyph structure, motion, audio prosody, layout constraints, and multilingual character exemplars. In recent work, this includes poster text image generation from a clean background image, a target text string, and a bounding-box position (Gao et al., 2023), kinetic typography driven by separate static and dynamic captions plus explicit word guidance (Park et al., 2024), speech-modulated captions that map loudness, pitch, and duration into typographic variables (Pataca et al., 2022), and cross-language visual text design transfer that preserves style while changing writing systems (Choi et al., 2024). This suggests that multi-modal typography is less a single task than a family of tightly coupled design, generation, interaction, and evaluation problems.

1. Conceptual scope and problem formulations

A central characteristic of the field is that typography is not treated as text alone. In poster generation, the task can be formalized as Text Image Generation for Posters (TIGER): given a clean poster background image $I_{bg}$ , a target text string $c$ , and its bounding-box position $b$ , the goal is to generate a raster text image $O_t$ that exactly fills region $b$ on $I_{bg}$ (Gao et al., 2023). In kinetic typography, the object is a video $x^{0:T}$ , and conditioning is explicitly split between appearance and motion so that a denoiser predicts noise under dual textual guidance (Park et al., 2024). In multilingual style transfer, the MuST task takes cropped styled source characters $S=\{s_1,\dots,s_N\}$ and target-language characters $T=\{c_1,\dots,c_M\}$ , and learns a model that outputs target character images while preserving visual style (Choi et al., 2024).

A second formulation treats documents themselves as multi-modal objects. FlexDM represents a vector graphic document as a set of elements $X=\{X_1,\dots,X_S\}$ , where each element contains fields such as text content, font family/weight/style/color, position and size, element type, and image embedding (Inoue et al., 2023). In that setting, typography becomes one field group among several interdependent fields, and typographic prediction is performed jointly with layout, content, and image reasoning.

A third formulation extends typography beyond purely visual design. Speech-modulated captioning computes syllable-level loudness, pitch, and duration and maps them to font-weight, baseline-shift, and letter-spacing, respectively, embedding paralinguistic information directly into text form (Pataca et al., 2022). At the opposite end of the spectrum, cross-modal adversarial work uses visual text overlays, synthesized speech, and distractor prompt text as coordinated typographic perturbations to alter audio-visual MLLM predictions (Chen et al., 5 Apr 2026). Typography therefore appears both as a design substrate and as a control channel.

2. Modalities and representations

Recent systems differ primarily in how they encode the modalities to be fused. Some representations are explicitly textual, some visual, some structural, and some temporal or acoustic.

Modality	Representative representation	Example systems
Visual context	global-local background image, local background patch, style-only glyph map	TextPainter, SkyReels-Text
Linguistic control	sentence-level and word-level tokens, static and dynamic captions, use-case prompts	TextPainter, Kinetic Typography Diffusion Model, FontUse
Glyph structure	raster glyph encoder, Bézier control points, graph representation, glyph latent	TextPainter, Dynamic Typography, Font Completion, SIGIL
Temporal or acoustic signal	motion-module attention, per-frame displacement field, loudness/pitch/duration features	Kinetic Typography Diffusion Model, Dynamic Typography, speech-modulated typography

Visual-context encoding is especially prominent in poster and in-image text systems. TextPainter uses both a global style encoder over the full poster background and a local style encoder conditioned by a binary mask channel $c$ 0, while sentence-level and word-level Chinese-CLIP tokens inject semantics into AdaIN-controlled synthesis and cross-attention (Gao et al., 2023). SkyReels-Text separates content and style visually: $c$ 1 is a reference poster inpainted with the new text content, whereas $c$ 2 is a blank canvas populated with user-supplied glyph patches that encode the target font style (Yu et al., 17 Nov 2025).

Glyph representation is equally heterogeneous. Dynamic Typography starts from SVG outlines represented by cubic Bézier control points $c$ 3, then optimizes a shared base field and a per-frame motion field to produce animated control points $c$ 4 (Liu et al., 2024). Font completion by cycling between multi-modality representations inserts a graph constructor between an image encoder and a neural renderer, predicting node-selection and adjacency to recover a graph $c$ 5 that exposes stroke connectivity and supports local editing (Yuan et al., 2021). FontCLIP instead constructs a joint semantic typographic latent space in which image and text encoders share a $c$ 6-D embedding space for font samples and attribute descriptions (Tatsukawa et al., 2024).

Audio-aware typography uses a different representational logic. In the prosody study, loudness is computed with RMS, pitch with an autocorrelation algorithm restricted to the 50–350 Hz band, and duration from syllable boundaries; each feature is then normalized at utterance and local-window levels before being mapped into a continuous CSS or variable-font axis (Pataca et al., 2022). This suggests that “multi-modal” in typography includes not only multi-input generation, but also cross-domain mappings between speech, language, graphics, and layout.

3. Generation, conditioning, and control mechanisms

Many recent systems build on latent diffusion, video diffusion, or diffusion transformers, but the field is not exclusively diffusion-based. Kinetic Typography Diffusion Model uses a latent-diffusion video backbone in which $c$ 7 is injected into spatial cross-attention and $c$ 8 into temporal cross-attention. A third “word” branch encodes $c$ 9 “The word $b$ 0” and adds a zero-initialized $b$ 1 convolution over attention maps so that the network can learn per-letter focus only when it improves reconstruction (Park et al., 2024). Its final objective is

$b$ 2

with $b$ 3, where $b$ 4 up-weights denoising error on text pixels in the final frame to improve legibility (Park et al., 2024).

Word-level controllability in scene text rendering is also enforced directly in the training objective. WordCon introduces a Text-Image Alignment framework that uses grounding-model masks $b$ 5 to supervise both latent diffusion flows and joint attention maps. The total loss is

$b$ 6

while the trainable parameters are restricted to low-rank reparameterizations of key and value matrices in Double-DiT joint-attention layers (Shi et al., 26 Jun 2025). SkyReels-Text likewise emphasizes text regions through a text-region weighted reconstruction loss,

$b$ 7

with $b$ 8 in practice (Yu et al., 17 Nov 2025).

Training-free control mechanisms form a distinct line of work. VitaGlyph decomposes a glyph into Subject and Surrounding regions, refines only the Subject geometry through SDEdit, and fuses two ControlNet branches by

$b$ 9

thereby preserving readability while allowing localized semantic deformation (Feng et al., 2024). WordCraft inserts binary regional masks into multi-modal multi-head attention so that only queries within region $O_t$ 0 attend to prompt $O_t$ 1, and continuous refinement is achieved through masked noise blending across denoising steps (Wang et al., 13 Jul 2025).

Not all multimodal typography systems are diffusion-native. TextPainter uses a StyleGAN-based generator with AdaIN, global-local style encoders, a text comprehension module, and a ResNet-34 glyph encoder with U-Net skip connections (Gao et al., 2023). GLDesigner instead relies on a VLM backbone built from CLIP ViT-L/336 and LLaMA, using Early Feature Fusion and Adaptive Average Pooling to generate layout JSON under user constraints (He et al., 2024). FlexDM uses masked prediction over document fields rather than synthesis from noise (Inoue et al., 2023). A common misconception is that multi-modal typography is synonymous with diffusion; the literature instead includes GANs, VLMs, masked-transformer document models, and human-in-the-loop optimization.

4. Major application regimes

One large application regime is poster and scene-text generation or editing. TextPainter targets visually harmonious and semantically emphasized poster text (Gao et al., 2023). SkyReels-Text performs simultaneous editing of multiple text regions in distinct typographic styles without font labels or inference-time fine-tuning, using user-provided cropped glyph patches as style references (Yu et al., 17 Nov 2025). FontUse adopts a data-centric strategy in which automatically annotated style and use-case prompts are used to fine-tune existing generators without architectural modification (Xin et al., 6 Mar 2026). WordCon focuses on precise word-level typography control so that different words in an image can receive independently bound styles (Shi et al., 26 Jun 2025).

A second regime is artistic typography and WordArt. DS-Fusion stylizes a rasterized input glyph according to a style prompt while preserving legibility via a CNN-based discriminator in latent space (Tanveer et al., 2023). MetaDesigner organizes the pipeline into Pipeline, Glyph, Texture, and Q&A agents, using GPT-4, ControlNet, LoRA fusion, and multimodal feedback for iterative hyperparameter updates (He et al., 2024). VitaGlyph frames artistic typography as a compositional scene with Subject and Surrounding regions (Feng et al., 2024), while WordCraft emphasizes localized edits, iterative refinement, multi-character composition, and open-ended prompt interpretation (Wang et al., 13 Jul 2025).

A third regime concerns fonts, layout, and document structure. FontCLIP supports multilingual and cross-lingual font retrieval as well as vector-glyph optimization from text or image prompts (Tatsukawa et al., 2024). FontCraft uses preferential Bayesian optimization over a learned font-style latent space, integrates multimodal references through FontCLIP, and propagates a selected style from one character to a full glyph set in OpenType format (Tatsukawa et al., 17 Feb 2025). GLDesigner predicts aesthetic text-glyph layouts from glyph images, text tokens, and user instructions (He et al., 2024). Font completion via graph cycling offers an explicit manipulation interface at the graph level (Yuan et al., 2021), whereas FlexDM treats typographic attributes as one set of fields inside a broader multi-task document model (Inoue et al., 2023).

A fourth regime is dynamic, animated, and cross-lingual typography. Dynamic Typography deforms letters semantically and animates them through vector graphics and a video diffusion prior (Liu et al., 2024). Kinetic Typography Diffusion Model adds explicit appearance-motion disentanglement, a word branch, and glyph-weighted denoising for readable animated text (Park et al., 2024). SIGIL addresses visual text design transfer across languages by combining glyph latent guidance, pretrained VAEs for style guidance, and OCR-based reinforcement learning (Choi et al., 2024). Speech-modulated captioning adds an accessibility-oriented regime in which prosodic nuance is embedded into typographic form rather than image style (Pataca et al., 2022).

5. Data resources, benchmarks, and empirical tendencies

The recent literature is strongly data-driven. Kinetic Typography Diffusion Model constructs a dataset of 600K three-second clips rendered at $O_t$ 2, downsampled to $O_t$ 3 at $O_t$ 4 fps, from 584 professionally designed After Effects templates; for evaluation, 26 real words across all templates yield 15,184 ground-truth videos (Park et al., 2024). TextPainter introduces PosterT80K, comprising about 80K real-world e-commerce posters with 342K sentence-level text boxes (Gao et al., 2023). WordCon builds a 28K-image word-level controlled scene-text dataset with word-level segmentation masks (Shi et al., 26 Jun 2025). GLDesigner constructs SynTextLogo and GenTextLogo, both substantially larger than the legacy TextLogo3K benchmark (He et al., 2024). FontUse assembles about 70K typography-focused images annotated through a segmentation-plus-OCR-plus-MLLM pipeline (Xin et al., 6 Mar 2026). MuST-Bench contains over 1,100 poster images spanning English-centered transfer into Chinese, Korean, Thai, Russian, and Arabic (Choi et al., 2024).

Evaluation protocols also vary by task. Kinetic typography work reports FVD for motion fidelity, Inception/FID on frames for image quality, CLIPScore for semantic alignment, and OCR-F1 for legibility (Park et al., 2024). Poster generation uses FID, SSIM, PSNR, and user preference (Gao et al., 2023). Cross-language style transfer uses CLIP-I, OCR accuracy, MLLM Likert scoring, and human ranking (Choi et al., 2024). FontUse adds a fine-tuned Long-CLIP evaluator and human-transcription CER (Xin et al., 6 Mar 2026). Scene-text editing and rendering often combine textual correctness, geometric localization, style similarity, and background preservation, as in the SkyReels-Text benchmark and AnyText benchmark (Yu et al., 17 Nov 2025).

Several empirical trends recur across papers. In kinetic typography, separating static and dynamic captions yields a $O_t$ 5 FVD improvement; adding the word branch increases CLIP and OCR by $O_t$ 6– $O_t$ 7; and glyph loss improves OCR by $O_t$ 8– $O_t$ 9 with little FVD penalty (Park et al., 2024). TextPainter achieves FID $b$ 0, SSIM $b$ 1, and PSNR $b$ 2 dB, while its ablations show that removing style encoding, masking, or text encoding harms harmony or semantic emphasis (Gao et al., 2023). WordCon+TIA reports type controllability $b$ 3, word controllability $b$ 4, total controllability $b$ 5, OCR precision/recall $b$ 6, and an overall user-study score of $b$ 7 (Shi et al., 26 Jun 2025). On MuST-Bench, SIGIL reaches OCR $b$ 8 and CLIP-I $b$ 9 on English/Chinese/Korean averages, outperforming description-based baselines in legibility while remaining competitive in style fidelity (Choi et al., 2024).

6. Limitations, recurring trade-offs, and the attack surface

The dominant trade-off is creativity or stylistic richness versus legibility. DS-Fusion was explicitly designed around the conflict between artistic stylization and readability (Tanveer et al., 2023). Dynamic Typography relies on shape-preservation and perceptual regularization because, without them, deformation can dissolve or self-intersect glyphs (Liu et al., 2024). VitaGlyph controls deformation through SDEdit noise strength $I_{bg}$ 0 and ControlNet guidance scale $I_{bg}$ 1, with larger $I_{bg}$ 2 producing greater deformation (Feng et al., 2024). FontUse reports that extremely ornate prompts can push models to prioritize style over legibility (Xin et al., 6 Mar 2026). In multilingual transfer, SIGIL’s ablations show that removing glyph guidance improves style drift metrics but collapses OCR, while removing the OCR-based Corrector harms legibility without affecting CLIP-I substantially (Choi et al., 2024).

Another recurring issue is that stronger multimodal conditioning does not automatically imply broader generality. TextPainter handles only single-line text and notes residual inpainting artifacts and occasional imperfect edges (Gao et al., 2023). SkyReels-Text depends on OCR quality, disallows overlapping text boxes, and can lose subtle color or lighting cues for very exotic fonts (Yu et al., 17 Nov 2025). GLDesigner currently outputs axis-aligned boxes rather than rotated glyph layouts (He et al., 2024). Speech-modulated typography showed no significant difference between animated and static presentation, and open-ended comments indicated that participants’ mental models varied widely; baseline-shift was less intuitive, and letter-spacing could impair word-shape recognition (Pataca et al., 2022). A plausible implication is that multimodal enrichment is only useful when the mapping between modalities remains interpretable to readers or robust to downstream recognition.

The field also has an explicit security dimension. Cross-modal typographic attacks on audio-visual MLLMs use synchronized visual overlays, TTS-generated speech, and prompt injection to steer predictions toward malicious targets. Across tasks and models, coordinated multi-modal attack yields attack success rate $I_{bg}$ 3 versus $I_{bg}$ 4 for single-modality attacks (Chen et al., 5 Apr 2026). On MetaHarm content moderation with Qwen2.5-Omni-7B, clean harmful-detection accuracy drops from $I_{bg}$ 5 to $I_{bg}$ 6 under keyword attack and to $I_{bg}$ 7 under prompt-style attack, while benign TTS prompts increase unsafe-to-safe errors from $I_{bg}$ 8 to $I_{bg}$ 9 (Chen et al., 5 Apr 2026). This establishes multi-modal typography not only as a design technology but also as a mechanism for cross-modal prompt injection and model hijacking.

Overall, the literature presents multi-modal typography as a convergence zone for generative modeling, document intelligence, motion design, font engineering, accessibility, multilingual transfer, and robustness research. The most successful systems typically isolate appearance from motion, content from style, or subject from surrounding; impose explicit glyph-aware or region-aware supervision; and evaluate readability alongside aesthetics rather than treating it as a secondary attribute (Park et al., 2024).