UM-Designer: Visual UML Code Generation Model

• The model leverages a ResNet/ViT-based CLIP encoder and a Vicuna transformer with a learnable vision-language connector to fuse visual and textual data.
• It employs LoRA fine-tuning to enhance efficiency, achieving high BLEU and SSIM scores on large-scale synthetic UML datasets.
• UM-Designer automates the translation of activity and sequence diagrams into syntax-valid PlantUML, streamlining software engineering workflows.

The UM-Designer visual-LLM is a multimodal system that generates executable Unified Modeling Language (UML) code from diagram images, targeting the automation of software engineering workflows by bridging visual representations and code artifacts. UM-Designer is architected on top of recent advances in large multimodal LLMs (MM-LLMs) and delivers end-to-end generation of syntax-valid PlantUML from activity and sequence diagram images, with quantitative benchmarks demonstrating high syntactic and structural fidelity (Bates et al., 15 Mar 2025).

1. Model Architecture

UM-Designer is based on LLaVA-1.5, a multimodal foundation model that integrates a ResNet/ViT-based CLIP visual encoder with a Vicuna transformer LLM via a learnable vision-language connector. The architecture can be decomposed as follows:

• Visual Encoder: The CLIP backbone processes diagram images into per-patch embeddings $V \in \mathbb{R}^{m \times d_v}$.
• Connector (Vision-Language Projector): A multilayer perceptron (MLP) or linear projection $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$ maps the visual tokens into the LM embedding space.
• LLM (Vicuna): Text token embeddings $X \in \mathbb{R}^{\ell \times d_\text{model}}$ are concatenated with the projected visual tokens for cross-modal fusion via multi-head self-attention.

Fusion is achieved via cross-modal multi-head attention. For each attention head: $$Q = X W_Q,\quad K = [X; V'] W_K,\quad V = [X; V'] W_V$$

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $V' = V W_\text{proj}$.

LoRA Adaptation: To support efficient fine-tuning, UM-Designer incorporates Low-Rank Adaptation (LoRA) in each multi-head attention block: $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r} B A$$ Here $A \in \mathbb{R}^{r \times d}$, $B \in \mathbb{R}^{d \times r}$, LoRA rank $r=128$, $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$0, and $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$1 is the frozen base weight (Bates et al., 15 Mar 2025).

2. Dataset Construction

UM-Designer was trained on a synthetic dataset constructed entirely via PlantUML. Major dataset characteristics:

• Diagram Types: Activity diagrams (control-flow, decisions, swimlanes) and sequence diagrams (lifelines, messages, activations).
• Data Generation: Randomized, syntactically correct PlantUML code was generated, rendered to PNG/SVG images, and paired with the source code in a JSON structure with fixed prompts.
• Scale and Splits: Four dataset sizes (train/test) per diagram type—Small (6,000/1,500), Medium (12,000/3,000), Large (24,000/6,000), and X-Large (120,000/30,000).

This fully synthetic approach ensures exhaustive coverage of UML constructs but is limited to clean, machine-generated diagrams (Bates et al., 15 Mar 2025).

3. Training Paradigms and Implementation

Two fine-tuning strategies were systematically compared:

• Standard Full Fine-Tuning: All model parameters were updated (LR$$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$2, batch size $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$3, grad accum $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$4; run on $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$5A100 GPUs).
• LoRA Fine-Tuning: Only LoRA matrices and the connector are trained (LoRA rank $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$6, $$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$7, connector LR$$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$8, LoRA LR$$W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$$9, batch size $X \in \mathbb{R}^{\ell \times d_\text{model}}$0, grad accum $X \in \mathbb{R}^{\ell \times d_\text{model}}$1; run on $X \in \mathbb{R}^{\ell \times d_\text{model}}$2A100 GPUs).

The objective is standard token-level cross-entropy: $X \in \mathbb{R}^{\ell \times d_\text{model}}$3 where $X \in \mathbb{R}^{\ell \times d_\text{model}}$4 is the one-hot encoding of the ground-truth token $X \in \mathbb{R}^{\ell \times d_\text{model}}$5 at timestep $X \in \mathbb{R}^{\ell \times d_\text{model}}$6 (Bates et al., 15 Mar 2025).

4. Evaluation Metrics and Empirical Results

UM-Designer models are evaluated on both code and perceptual fidelity:

• BLEU: $X \in \mathbb{R}^{\ell \times d_\text{model}}$7-gram overlap between generated and reference PlantUML code, incorporating brevity penalty.
• SSIM: Structural Simplify Index, quantifying perceptual similarity between images rendered from generated and reference UML code.
• Error Rates: Syntax errors (PlantUML render failures), no output, and diagram-type mismatches.

Key results on the X-Large test set:

Model/Setting	Diagram	BLEU	SSIM
13B LoRA	Sequence	0.779	0.942
13B LoRA	Activity	0.350	0.891
13B Full Fine-Tune	Sequence	0.769	0.936
7B LoRA	Sequence	~0.70	~0.90

Ablation studies reveal larger models correlate with higher BLEU/SSIM and reduced syntax errors; LoRA matches full fine-tuning in quality while using half the GPU resources. Sequence diagrams, being structurally simpler, yield higher scores than activity diagrams, and dataset scale is crucial for accuracy (small sets BLEU < 0.15, X-Large BLEU > 0.35/0.77 for activity/sequence) (Bates et al., 15 Mar 2025).

5. Workflow and System Pipeline

The end-to-end UM-Designer pipeline encompasses:

1. Image Input: Clean UML diagrams rendered in PNG/SVG format.
2. Vision-Language Preprocessing: Images are encoded via CLIP, visual features projected and fused via cross-modal attention with text prompt.
3. Code Generation: The transformer decoder autoregressively emits PlantUML code.
4. Post-Processing: Code is subject to syntactic checks and rendered for SSIM-based comparison against the reference.

The strict requirement for rendered, synthetic diagrams (without OCR or noise tolerance) delimits application to controlled scenarios. Absence of direct handling for class diagrams and handwritten or photographic inputs is a current limitation (Bates et al., 15 Mar 2025).

6. Use Cases, Limitations, and Future Directions

Applications:

• Automated conversion of legacy UML artifacts (e.g., static PDFs, scans) to editable PlantUML code.
• IDE plugins (e.g., VS Code, IntelliJ) enabling UML-to-code workflows.
• Automated synchronization between visual documentation and software codebases.

Limitations:

• Only supports cleanly-rendered, synthetic diagram images—photographs, hand-drawn sketches, and diagrams with noisy backgrounds are not handled.
• Focus restricted to activity and sequence diagrams; does not cover class, component, or state-chart diagrams.
• Does not incorporate OCR for embedded text; relies on PlantUML syntax adherence.

Future Work:

• Incorporation of integrated OCR and geometric layout analysis (e.g., LLaVA-OneVision, LLaVA-NeXT) for real-world diagram robustness.
• Extension to additional UML diagam types and broader model families.
• Adoption of more comprehensive metrics such as CodeBLEU (AST-aware) and execution-based (“pass@k”) correctness.
• Exploration of domain adaptation and semi-supervised paradigms for generalization to hand-drawn and noisy images (Bates et al., 15 Mar 2025).

7. Context and Relationship to Broader Multimodal UML Modeling Research

UM-Designer represents a focused instance within the broader landscape of multimodal code generation:

• MM-Coder and other models stress the necessity of harmonizing visual workflows and textual prompts, employing instruction-tuning datasets and execution-based benchmarks (e.g., Pass@1) (Chai et al., 11 Jul 2025).
• The current generation of MM-LLMs achieves 25–50% accuracy on complex code-gen benchmarks involving UML/flowchart images, indicating persistent difficulties in visual-semantic alignment and instruction compliance.
• The systematic use of synthetic datasets, cross-modal pretraining, and domain-adapted finetuning is affirmed as essential for effective image-to-code translation in UML tasks.
• Integration of human-in-the-loop workflows and post-generation error detection remains central for validating structural and semantic correctness, especially when migrating towards real-world deployments (Conrardy et al., 2024).

UM-Designer’s approach, combining vision-language foundation models, large-scale synthetic corpora, and LoRA-efficient adaptation, demonstrates that MM-LLMs can achieve practical UML diagram-to-code translation with measurable fidelity. However, generalization to non-synthetic settings and broadened UML coverage are necessary next steps for industrial applicability.