---
title: 'UM-Designer: Visual UML Code Generation Model'
url: https://www.emergentmind.com/topics/um-designer-visual-language-model
type: topic
---

# UM-Designer: Visual UML Code Generation Model

The UM-Designer visual-language model is a multimodal system that generates executable Unified Modeling Language (UML) code from diagram images, targeting the automation of software engineering workflows by bridging visual representations and code artifacts. UM-Designer is architected on top of recent advances in large multimodal language models (MM-LLMs) and delivers end-to-end generation of syntax-valid PlantUML from activity and sequence diagram images, with quantitative benchmarks demonstrating high syntactic and structural fidelity [2503.12293].

## 1. Model Architecture

UM-Designer is based on LLaVA-1.5, a multimodal foundation model that integrates a ResNet/ViT-based CLIP visual encoder with a Vicuna transformer language model via a learnable vision-language connector. The architecture can be decomposed as follows:

- **Visual Encoder**: The CLIP backbone processes diagram images into per-patch embeddings $V \in \mathbb{R}^{m \times d_v}$.
- **Connector (Vision-Language Projector)**: A multilayer perceptron (MLP) or linear projection $W_\text{proj} \in \mathbb{R}^{d_v \times d_\text{model}}$ maps the visual tokens into the LM embedding space.
- **Language Model (Vicuna)**: Text token embeddings $X \in \mathbb{R}^{\ell \times d_\text{model}}$ are concatenated with the projected visual tokens for cross-modal fusion via multi-head self-attention.

Fusion is achieved via cross-modal multi-head attention. For each attention head:
\[
Q = X W_Q,\quad K = [X; V'] W_K,\quad V = [X; V'] W_V
\]
\[
\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
\]
where $V' = V W_\text{proj}$.

**LoRA Adaptation:** To support efficient fine-tuning, UM-Designer incorporates Low-Rank Adaptation (LoRA) in each multi-head attention block:
\[
W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r} B A
\]
Here $A \in \mathbb{R}^{r \times d}$, $B \in \mathbb{R}^{d \times r}$, LoRA rank $r=128$, $\alpha=256$, and $W_0$ is the frozen base weight [2503.12293].

## 2. Dataset Construction

UM-Designer was trained on a synthetic dataset constructed entirely via PlantUML. Major dataset characteristics:

- **Diagram Types:** Activity diagrams (control-flow, decisions, swimlanes) and sequence diagrams (lifelines, messages, activations).
- **Data Generation:** Randomized, syntactically correct PlantUML code was generated, rendered to PNG/SVG images, and paired with the source code in a JSON structure with fixed prompts.
- **Scale and Splits:** Four dataset sizes (train/test) per diagram type—Small (6,000/1,500), Medium (12,000/3,000), Large (24,000/6,000), and X-Large (120,000/30,000).

This fully synthetic approach ensures exhaustive coverage of UML constructs but is limited to clean, machine-generated diagrams [2503.12293].

## 3. Training Paradigms and Implementation

Two fine-tuning strategies were systematically compared:

- **Standard Full Fine-Tuning:** All model parameters were updated (LR$=2\times 10^{-5}$, batch size $8$, grad accum $4$; run on $4\times$A100 GPUs).
- **LoRA Fine-Tuning:** Only LoRA matrices and the connector are trained (LoRA rank $128$, $\alpha=256$, connector LR$=2\times 10^{-5}$, LoRA LR$=2\times 10^{-4}$, batch size $16$, grad accum $4$; run on $2\times$A100 GPUs).

The objective is standard token-level cross-entropy:
\[
\mathcal{L} = -\sum_{t=1}^T \sum_{v \in \mathcal{V}} y_{t,v} \log \hat{y}_{t,v}
\]
where $y_{t,v}$ is the one-hot encoding of the ground-truth token $v$ at timestep $t$ [2503.12293].

## 4. Evaluation Metrics and Empirical Results

UM-Designer models are evaluated on both code and perceptual fidelity:

- **BLEU:** $n$-gram overlap between generated and reference PlantUML code, incorporating brevity penalty.
- **SSIM:** Structural Simplify Index, quantifying perceptual similarity between images rendered from generated and reference UML code.
- **Error Rates:** Syntax errors (PlantUML render failures), no output, and diagram-type mismatches.

Key results on the X-Large test set:

| Model/Setting              | Diagram      | BLEU  | SSIM   |
|---------------------------|--------------|-------|--------|
| 13B LoRA                  | Sequence     | 0.779 | 0.942  |
| 13B LoRA                  | Activity     | 0.350 | 0.891  |
| 13B Full Fine-Tune        | Sequence     | 0.769 | 0.936  |
| 7B LoRA                   | Sequence     | ~0.70 | ~0.90  |

Ablation studies reveal larger models correlate with higher BLEU/SSIM and reduced syntax errors; LoRA matches full fine-tuning in quality while using half the GPU resources. Sequence diagrams, being structurally simpler, yield higher scores than activity diagrams, and dataset scale is crucial for accuracy (small sets BLEU < 0.15, X-Large BLEU > 0.35/0.77 for activity/sequence) [2503.12293].

## 5. Workflow and System Pipeline

The end-to-end UM-Designer pipeline encompasses:

1. **Image Input:** Clean UML diagrams rendered in PNG/SVG format.
2. **Vision-Language Preprocessing:** Images are encoded via CLIP, visual features projected and fused via cross-modal attention with text prompt.
3. **Code Generation:** The transformer decoder autoregressively emits PlantUML code.
4. **Post-Processing:** Code is subject to syntactic checks and rendered for SSIM-based comparison against the reference.

The strict requirement for rendered, synthetic diagrams (without OCR or noise tolerance) delimits application to controlled scenarios. Absence of direct handling for class diagrams and handwritten or photographic inputs is a current limitation [2503.12293].

## 6. Use Cases, Limitations, and Future Directions

**Applications:**
- Automated conversion of legacy UML artifacts (e.g., static PDFs, scans) to editable PlantUML code.
- IDE plugins (e.g., VS Code, IntelliJ) enabling UML-to-code workflows.
- Automated synchronization between visual documentation and software codebases.

**Limitations:**
- Only supports cleanly-rendered, synthetic diagram images—photographs, hand-drawn sketches, and diagrams with noisy backgrounds are not handled.
- Focus restricted to activity and sequence diagrams; does not cover class, component, or state-chart diagrams.
- Does not incorporate OCR for embedded text; relies on PlantUML syntax adherence.

**Future Work:**
- Incorporation of integrated OCR and geometric layout analysis (e.g., LLaVA-OneVision, LLaVA-NeXT) for real-world diagram robustness.
- Extension to additional UML diagam types and broader model families.
- Adoption of more comprehensive metrics such as CodeBLEU (AST-aware) and execution-based (“pass@k”) correctness.
- Exploration of domain adaptation and semi-supervised paradigms for generalization to hand-drawn and noisy images [2503.12293].

## 7. Context and Relationship to Broader Multimodal UML Modeling Research

UM-Designer represents a focused instance within the broader landscape of multimodal code generation:

- MM-Coder and other models stress the necessity of harmonizing visual workflows and textual prompts, employing instruction-tuning datasets and execution-based benchmarks (e.g., Pass@1) [2507.08719].
- The current generation of MM-LLMs achieves 25–50% accuracy on complex code-gen benchmarks involving UML/flowchart images, indicating persistent difficulties in visual-semantic alignment and instruction compliance.
- The systematic use of synthetic datasets, cross-modal pretraining, and domain-adapted finetuning is affirmed as essential for effective image-to-code translation in UML tasks.
- Integration of human-in-the-loop workflows and post-generation error detection remains central for validating structural and semantic correctness, especially when migrating towards real-world deployments [2404.11376].

UM-Designer’s approach, combining vision-language foundation models, large-scale synthetic corpora, and LoRA-efficient adaptation, demonstrates that MM-LLMs can achieve practical UML diagram-to-code translation with measurable fidelity. However, generalization to non-synthetic settings and broadened UML coverage are necessary next steps for industrial applicability.

Source: https://www.emergentmind.com/topics/um-designer-visual-language-model