UniCalli: Unified Diffusion for Chinese Calligraphy

• UniCalli is a unified diffusion framework for column-level calligraphy that jointly performs recognition and generation to maintain both glyph correctness and page aesthetics.
• It leverages a shared latent diffusion backbone with asymmetric noising across content, image, and box-map streams to enforce layout fidelity and style priors.
• Empirical results demonstrate improved metrics and expert evaluations, confirming enhanced ligature continuity, spatial consistency, and competitive recognition accuracy.

Searching arXiv for the UniCalli paper to ground the article in the cited source. UniCalli is a unified diffusion framework for column-level recognition and generation of Chinese calligraphy. It is designed to address a specific bifurcation in prior computational calligraphy systems: some methods produce high-quality isolated characters while disregarding page-level aesthetics such as ligatures and spacing, whereas others attempt page synthesis but degrade calligraphic correctness. UniCalli formulates recognition and generation as a single jointly trained problem, using shared latent diffusion machinery so that recognition constrains structural fidelity and generation supplies style and layout priors. The framework was introduced in "UniCalli: A Unified Diffusion Framework for Column-Level Generation and Recognition of Chinese Calligraphy" (Xu et al., 15 Oct 2025).

1. Problem setting and design rationale

The central problem addressed by UniCalli is computational replication of Chinese calligraphy under column-level constraints rather than isolated glyph synthesis. In this setting, correctness is not limited to individual character shape; it also includes inter-character spacing, column alignment, and ligature continuity. The paper positions these as page-level aesthetics that existing methods handle inadequately when they focus only on single-character rendering, while methods targeting larger compositions often sacrifice calligraphic correctness (Xu et al., 15 Oct 2025).

UniCalli therefore treats generation and recognition as mutually constraining tasks rather than as separate pipelines. In generation, the model must produce calligraphy conditioned on textual content and spatial priors. In recognition, it must infer textual content from calligraphy while also accounting for layout. The stated rationale is that joint training is deliberate: recognition constrains the generator to preserve character structure, while generation provides style and layout priors. This suggests a representation-learning objective at the level of abstract calligraphic concepts rather than a narrow mapping between rendered fonts and handwritten forms.

A notable aspect of the formulation is its unit of synthesis and analysis. The model operates on column-level input, specifically 5-char vertical crops resized to $128 \times 640$ and patchified, with content, calligraphy, and box-map crops concatenated (Xu et al., 15 Oct 2025). This preserves contextual dependencies that would be inaccessible in character-isolated settings, especially for ligatures and context-aware sizing.

2. Core architecture and latent representations

UniCalli builds on a latent-space diffusion backbone identified as the FLUX/MMDiT transformer, and it uses a single shared network for both generation and recognition by swapping which latent branch is noised (Xu et al., 15 Oct 2025). The architecture ingests three latent inputs:

• content latent $z_c = \mathcal{E}(I_c)$, described as a clean raster of the standard-font rendering of the target text;
• image latent $z_i = \mathcal{E}(I_i)$, the calligraphy image latent, possibly noisy;
• box-map latent $z_m = \mathcal{E}(I_m)$, a rasterized per-character bounding-box map.

These three streams are projected into a common transformer through Duplicate RoPE and Modulated Embedding. The positional construction is given as

$$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$

The paper states that this arrangement ensures a single shared 2D positional frame while preserving modality identity through learnable offsets $E_{\mathrm{mod},c}$, $E_{\mathrm{mod},i}$, and $E_{\mathrm{mod},m}$ (Xu et al., 15 Oct 2025).

The tokens and their diffusion timestep embeddings, with $T_c$ and $T_i = T_m$, are concatenated and passed through the MMDiT block. Its bidirectional self-attention is explicitly described as enabling holistic planning of full columns rather than auto-regressive glyph filling. In contextual terms, this architectural choice is aligned with the paper’s emphasis on layout fidelity and ligature continuity, because such phenomena depend on nonlocal interactions across the column.

The box map is not an auxiliary metadata channel in a weak sense; it is a structured latent input derived from a three-channel rasterization of each character’s 2D bounding box, including position and scale. The paper states that this forces the model to learn realistic inter-character spacing, column alignment, and ligature continuity, since in generation it must predict both $z_c = \mathcal{E}(I_c)$0 and $z_c = \mathcal{E}(I_c)$1, and in recognition it must reconstruct the box map as an auxiliary task (Xu et al., 15 Oct 2025).

3. Asymmetric diffusion formulation

UniCalli adopts standard DDPM parameterization for its forward and reverse processes, but specializes it through an asymmetric noising scheme that switches according to task mode (Xu et al., 15 Oct 2025). The forward process is written as

$z_c = \mathcal{E}(I_c)$2

with cumulative $z_c = \mathcal{E}(I_c)$3 and the equivalent marginal

$z_c = \mathcal{E}(I_c)$4

The learned reverse process is

$z_c = \mathcal{E}(I_c)$5

with simplified diffusion loss

$z_c = \mathcal{E}(I_c)$6

The task-specific asymmetry is the key modification. In generation mode, selected with probability $z_c = \mathcal{E}(I_c)$7 at each step, the model fixes $z_c = \mathcal{E}(I_c)$8 so content is not noised, samples $z_c = \mathcal{E}(I_c)$9, and applies the same noise to image and box. The network therefore denoises image and box conditioned on clean content. In recognition mode, the roles are reversed: $z_i = \mathcal{E}(I_i)$0 and $z_i = \mathcal{E}(I_i)$1, so the network denoises content given clean image and box (Xu et al., 15 Oct 2025).

The implementation is stated to use a flow-matching style noising rule,

$z_i = \mathcal{E}(I_i)$2

This asymmetric treatment is central to the framework’s bidirectionality. Recognition mode pressures the shared backbone to preserve sufficient glyph structure for content recovery, while generation mode pressures it to internalize layout and style priors that support recognition on real calligraphy. A plausible implication is that the latent space must encode both semantic identity and visual organization in a more entangled but task-useful form than a unidirectional generator or OCR system would require.

4. Training objectives and data mixture

UniCalli combines three diffusion-matching losses, one for each latent branch, with a recognition cross-entropy term (Xu et al., 15 Oct 2025). For branch $z_i = \mathcal{E}(I_i)$3, the denoising objective is

$z_i = \mathcal{E}(I_i)$4

In recognition mode, a final content decoder or small head produces logits $z_i = \mathcal{E}(I_i)$5 and incurs

$z_i = \mathcal{E}(I_i)$6

The combined multi-task objective uses the notation $z_i = \mathcal{E}(I_i)$7, $z_i = \mathcal{E}(I_i)$8, $z_i = \mathcal{E}(I_i)$9, and $z_m = \mathcal{E}(I_m)$0. In generation mode,

$z_m = \mathcal{E}(I_m)$1

whereas in recognition mode,

$z_m = \mathcal{E}(I_m)$2

The paper states that Table 4 confirms this balancing yields the best joint generation and recognition outcome (Xu et al., 15 Oct 2025).

The training corpus is heterogeneous and explicitly divided into labeled real, unlabeled real, and synthetic data. Labeled real data consists of 4K images with approximately 150K boxes and character labels, used in standard dual-mode training. Unlabeled real data consists of 4K images and is incorporated by always setting $z_m = \mathcal{E}(I_m)$3, so the model performs unconditional generation or denoising on image and box, thereby exposing it to more style variation. Synthetic TTF-rendered data is sampled at $z_m = \mathcal{E}(I_m)$4 and supplies unlimited, noise-free coverage of all glyphs for learning basic radical and stroke structure (Xu et al., 15 Oct 2025).

The paper interprets the result of this regimen as the emergence of “concept-level” abstractions of radicals and stroke-group patterns that are shared across scripts and styles. This suggests that the framework’s sample efficiency in limited-data regimes is not merely due to data augmentation, but to a shared latent organization encouraged by dual-mode denoising over mixed supervision.

5. Dataset and empirical performance

The dataset described for Chinese calligraphy contains more than 8,000 digitized works from 93 masters across five script families, with more than 4,000 works annotated by per-character boxes and modern transcriptions, totaling approximately 150K instances (Xu et al., 15 Oct 2025). The scale is significant because the task couples structural recognition with stylistic generation at column level rather than treating annotation as simple text-image pairing.

On generative evaluation, UniCalli is compared against FontDiffuser, ChatGPT-5, Ernie-4.5, and Doubao using $z_m = \mathcal{E}(I_m)$5, $z_m = \mathcal{E}(I_m)$6, $z_m = \mathcal{E}(I_m)$7, and $z_m = \mathcal{E}(I_m)$8. The reported values are:

• $z_m = \mathcal{E}(I_m)$9 versus $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$0–$$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$1 for baselines;
• $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$2 versus $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$3–$$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$4;
• $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$5;
• $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$6 (Xu et al., 15 Oct 2025).

The user study involved 20 calligraphy experts using a 1–5 Likert scale. UniCalli is reported to outperform all baselines in Style Fidelity $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$7, Glyph Accuracy $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$8, Naturalness $$\mathrm{RoPE}_i = \mathrm{CalcRoPE}(z_i), \quad \mathrm{RoPE}_k = \mathrm{RoPE}_i + E_{\mathrm{mod},k} \quad (k \in \{c,m\}).$$9, and Overall Preference $E_{\mathrm{mod},c}$0 (Xu et al., 15 Oct 2025). Because these dimensions separate structural fidelity from perceptual plausibility, the result is consistent with the paper’s claim that joint training helps prevent a drift toward “style-only” outputs.

For recognition, evaluation is reported as character-level accuracy on a held-out 100-image test set. UniCalli achieves a total accuracy of $E_{\mathrm{mod},c}$1 versus the best baseline, Doubao-1.5*, at $E_{\mathrm{mod},c}$2, with especially strong gains on Regular script, where it reaches $E_{\mathrm{mod},c}$3 versus $E_{\mathrm{mod},c}$4 (Xu et al., 15 Oct 2025). The paper therefore frames the framework not as a tradeoff in which generative richness harms OCR-like decoding, but as a joint system in which each task improves the other.

Ablation results isolate three architectural and training contributions. Joint training alone increases SSIM from $E_{\mathrm{mod},c}$5 to $E_{\mathrm{mod},c}$6. Adding RoPE duplication improves FID from $E_{\mathrm{mod},c}$7 to $E_{\mathrm{mod},c}$8. Adding Conditional Dropout with $E_{\mathrm{mod},c}$9 further improves FID from $E_{\mathrm{mod},i}$0 to $E_{\mathrm{mod},i}$1 (Xu et al., 15 Oct 2025). Qualitative analysis is reported to show substantially better ligature continuity and context-aware sizing than competitors.

6. Generalization beyond Chinese calligraphy

UniCalli is also evaluated beyond Chinese calligraphy, specifically on Oracle Bone Script and Egyptian hieroglyphs (Xu et al., 15 Oct 2025). These extensions are important because they test whether the learned abstractions are tied to one script family or whether the framework can transfer to other ancient writing systems.

For Oracle Bone Script, the model is trained on HUST-OBC. Generation is evaluated by experts and yields $E_{\mathrm{mod},i}$2 Completely Correct, $E_{\mathrm{mod},i}$3 Largely Correct, and $E_{\mathrm{mod},i}$4 Incorrect. Recognition reaches $E_{\mathrm{mod},i}$5 character accuracy, compared with OracleNet at $E_{\mathrm{mod},i}$6 (Xu et al., 15 Oct 2025). The reported qualitative result is faithful structural and stylistic emulation of real oracle glyphs. The asymmetry between strong generative evaluation and lower recognition relative to OracleNet suggests that UniCalli’s unification does not automatically dominate specialized recognizers in every low-resource script, though it remains effective as a single framework spanning both tasks.

For Egyptian hieroglyphs, the paper describes a two-step pipeline via Chinese as a semantic bridge. The qualitative outputs are illustrated in the paper, and recognition accuracy is reported as $E_{\mathrm{mod},i}$7 (Xu et al., 15 Oct 2025). Because the details are summarized more briefly than for Chinese calligraphy, the safest interpretation is that the framework can be adapted with minimal conceptual changes, rather than that it establishes an identical task formulation across all scripts.

Taken together, these experiments support the paper’s claim that the shared backbone, asymmetric noising, and explicit box-map prior can generalize to other ancient scripts. A plausible implication is that the framework’s principal transferable object is not a language-specific decoder, but a latent alignment between abstract content, visual form, and spatial layout.

7. Significance, scope, and common misunderstandings

UniCalli’s principal contribution is not merely a stronger image generator for stylized text. Its defining property is the unification of column-level generation and recognition within a single diffusion framework, with shared weights and task-dependent noising (Xu et al., 15 Oct 2025). This distinguishes it from systems that treat OCR and synthesis as sequential modules or as unrelated benchmarks.

A common misunderstanding would be to treat the framework as a page-generation model that ignores glyph correctness in favor of global aesthetics. The paper states the opposite motivation: recognition is used specifically to constrain generation so that character structure is preserved. Another misunderstanding would be to reduce the box map to a simple conditioning signal comparable to generic layout metadata. In UniCalli, the rasterized box map is a learned latent branch with denoising and reconstruction roles, and the paper explicitly connects it to spacing, column alignment, and ligature continuity (Xu et al., 15 Oct 2025).

The work also should not be read as claiming uniform superiority on every downstream task in every script. For Oracle Bone Script recognition, the reported $E_{\mathrm{mod},i}$8 character accuracy remains below OracleNet’s $E_{\mathrm{mod},i}$9 (Xu et al., 15 Oct 2025). The more precise claim is that a single model can achieve state-of-the-art generative quality for Chinese calligraphy while also delivering stronger recognition in the Chinese setting and extending successfully to other ancient scripts.

In sum, UniCalli defines a bidirectional latent-diffusion paradigm for calligraphic documents in which content, image, and spatial layout are modeled jointly. Its shared MMDiT backbone, asymmetric diffusion noising, rasterized box-map prior, and mixed real, unlabeled, and synthetic training regime are presented as the mechanisms by which it attains superior ligature continuity, layout fidelity, and competitive recognition within one unified model (Xu et al., 15 Oct 2025).