DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generation

Abstract: Handwritten Mathematical Expression Generation (HMEG) is challenging due to the complex two-dimensional layouts and long-range structural dependencies of mathematical expressions. Existing methods typically rely on explicit spatial supervision, such as symbol-level bounding boxes, which incurs high annotation costs and limits scalability. In this work, we propose DiffMath, a symbol- and graph-aware latent diffusion framework that leverages the hierarchical structure inherent in LaTeX as a structural prior, eliminating the need for positional supervision. First, we design a Relational Abstract Syntax Tree (RelAST), a generation-oriented representation that distills MathML trees into compact triplet sequences [S, R, D], where each token directly encodes a symbol identity, spatial relation, or nesting depth. Second, we introduce MathVAE, which learns structure-preserving latent representations through symbol-aware and relation-aware perceptual regularization, ensuring that the latent space captures both character semantics and spatial topology. Third, MathDiT performs conditional denoising in this structured latent space, further guided by a global symbol-count prior via Adaptive Layer Normalization (AdaLN) to improve structural coherence. Experiments show that DiffMath produces structurally consistent handwritten expressions, achieves superior performance over existing methods, and improves the accuracy of downstream OCR models through synthetic data augmentation.

• The paper introduces an end-to-end latent diffusion framework leveraging RelAST to map LaTeX to formula pixels without dense spatial annotations.
• The proposed MathVAE and MathDiT modules incorporate symbol- and relation-aware regularization, achieving 70.70% ExpRate and FID 5.43.
• DiffMath significantly augments HMER and recognition models by producing high-fidelity, structurally coherent handwritten expressions.

DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generation

Motivation and Problem Formulation

Handwritten Mathematical Expression Generation (HMEG) fundamentally differs from conventional handwritten text synthesis. The spatial and relational complexity inherent in mathematical expressions—structural hierarchy, nested 2D layouts, symbol dependencies—poses significant challenges for generative models. Traditional methods frequently rely on explicit spatial supervision, such as symbol-level bounding boxes, which is prohibitively expensive and scales poorly. Recent advances using two-stage paradigms decouple glyph generation from layout prediction but remain limited by data dependency and inconsistent structural restitution.

DiffMath introduces an end-to-end latent diffusion framework leveraging relational and graph-aware structural priors extracted from LaTeX, bypassing requirements for dense spatial annotation and directly mapping LaTeX inputs to formula pixels. This approach markedly improves global structural consistency and scalability for HMEG.

Figure 1: DiffMath versus two-stage approaches: end-to-end mapping from LaTeX to formula pixels enhances consistency and data efficiency.

Architecture: Structural Priors, Latent Space, and Diffusion

RelAST: Prescriptive Structural Representation

The core innovation is the Relational Abstract Syntax Tree (RelAST), a generation-oriented representation distilled from standardized MathML trees. RelAST linearizes formula structure into sequential triplets $[S, R, D]$—symbol identity, explicit spatial relation (e.g., RIGHT, ABOVE), and hierarchical nesting depth—thus preserving both local content and global topological dependencies.

Figure 2: RelAST construction: LaTeX is parsed to MathML and serialized into triplets encoding content, spatial relation, and nesting.

MathVAE: Structure-Preserving Latent Compression

MathVAE encodes online trajectories into a compact latent space, enhancing traditional VAE objectives with symbol-aware ($\mathcal{L}_{\text{sym}}$) and relation-aware ($\mathcal{L}_{\text{rel}}$) perceptual regularization. This ensures latent codes capture the semantics of character shapes and their structural positioning, facilitating downstream synthesis fidelity and convergence. Joint supervision via CTC-loss on the interleaved sequence of symbol and relation tokens further enforces topological consistency.

Figure 3: DiffMath pipeline: LaTeX-to-RelAST parsing, MathVAE latent compression, and topology-conditioned Diffusion Transformer for generation.

MathDiT: Conditional Latent Diffusion

MathDiT refines noisy latents to clean trajectory samples using a Transformer backbone conditioned on structured RelAST embeddings and a global symbol-count prior modulated via AdaLN. Symbol count impacts the time embedding during diffusion, particularly in later denoising steps, to regulate formula complexity and completeness. The conditional sequence modeling in latent space enables fine-grained control of spatial topology and symbol allocation.

Quantitative and Qualitative Evaluation

DiffMath is evaluated on MathWriting, using edit distance, expression recognition rate (ExpRate), BLEU, FID, and user study metrics. DiffMath achieves 70.70% ExpRate and FID 5.43, outperforming all SOTA baselines, including GAN-based (FormulaGAN), text-to-image transformers (SDXL, Qwen-Image, FLUX.1), and prior latent diffusion models (DiffInk). DiffMath consistently yields more visually authentic handwriting samples, greater structural fidelity in symbol placement, and higher user preference ranking.

Figure 4: Qualitative comparison: DiffMath maintains symbol integrity and structural cohesion, surpassing SOTA methods under complex scenarios.

Ablation studies confirm the critical role of symbol- and relation-aware regularization in MathVAE and the additive benefit of symbol, relation, depth, and global priors in MathDiT. Removing any aspect leads to significant drops in fidelity and structural correctness.

Failure Cases and Data Augmentation Utility

Analysis reveals DiffMath's limitations in dense layouts, rare symbol handling, and deep nested structures—errors manifest as missing symbols, misplacement, or confusion among semantically similar glyphs. These complexities form the next frontiers for generative modeling research.

Figure 5: Typical failure modes: rare symbols, deep nesting, and locally dense regions remain challenging for accurate generation.

DiffMath-generated samples substantially improve the performance of multiple HMER models (e.g., Uni-MuMER, Qwen-VL-8B-Instruct) across MathWriting and CROHME benchmarks, providing valuable augmentation for recognition training and reducing annotation overhead.

Structural Generalization and Expressive Capability

DiffMath effectively handles both routine and highly complex expressions, including multi-row matrices, integrals, and densely packed formulas, demonstrating robustness and extensibility in symbol and graph-aware synthesis.

Figure 6: Simple/common expression comparison: DiffMath preserves atomic fidelity in both basic and dense layouts.

Figure 7: Additional DiffMath samples: precise reproduction of matrix arrangements and long formulas with intricate local structures.

Implications and Theoretical Impact

The architectural advances in DiffMath—structural prior extraction via RelAST, symbol- and graph-aware latent regularization, and complexity-conditioned diffusion—represent a shift towards explicit structural modeling in generative tasks. This methodology bridges the gap between heuristic layout decoupling and principled end-to-end synthesis. For HMEG, it eliminates expensive spatial annotation, generalizes to arbitrary algebraic structures, and produces high-fidelity samples suitable for recognition and educational applications.

Practically, DiffMath offers a scalable foundation for automated content generation, training data augmentation, and interactive formula rendering. Theoretically, the framework sets a template for symbolic graph-conditioned diffusion over structured latent spaces, with broad applicability in knowledge-driven generation beyond mathematics—chemistry, engineering schematics, and structured language artifacts.

Conclusion

DiffMath establishes a robust, structure-aware latent diffusion paradigm for handwritten mathematical expression generation, leveraging prescriptive structural priors and symbol-count modulation to drive superior symbolic and relational fidelity. Its integration of topological constraints into deep generative modeling enables unannotated, scalable HMEG and enhances downstream recognition through synthetic data augmentation. Future directions include fine-grained handling of rare glyphs, deeper layout representations, and cross-domain extension of structural priors in generative diffusion architectures.