---
title: 'SysMBench: Formal SysML v2 Benchmark'
url: https://www.emergentmind.com/topics/sysmbench
type: topic
---

# SysMBench: Formal SysML v2 Benchmark

SysMBench is a benchmark for evaluating whether large language models can generate formal system models in **OMG SysML v2 textual notation** from **natural-language requirements**. Each scenario pairs an English requirements description with a human-curated SysML model, a rendered graphical diagram, and metadata including a domain label, a key grammar label, and a difficulty level. Introduced in 2025 together with the semantic-aware evaluation framework **SysMEval**, SysMBench targets end-to-end **natural-language-to-system-model generation** rather than entity extraction or toy modeling tasks, and it is presented as the first public benchmark for **SysML-based system model generation from natural-language requirements** [2508.03215].

## 1. Definition, scope, and modeling task

SysMBench addresses a specific problem in model-based systems engineering and software development: system models are valuable artifacts for requirements analysis and architecture design, but constructing them is difficult because SysML syntax is specialized, public examples are relatively scarce, and manual model development is slow and error-prone. The benchmark asks whether LLMs can translate natural-language requirements into formal system models in a real modeling language, rather than merely recover isolated entities or relations. In this formulation, each benchmark instance provides a requirements description as input and expects a complete **SysML v2 textual model** as output [2508.03215].

The benchmark is deliberately built around declarative model generation. The authors contrast it with prior work that often reduced system modeling to information extraction on very small handcrafted datasets and did not require generation in a concrete modeling language. SysMBench instead evaluates full-text generation in SysML v2 textual notation. The benchmark overview identifies six components per sample: **Requirements**, **System model with textual notation**, **System model with graphical notation**, **Domain label**, **Key grammar**, and **Difficulty level**. This organization makes the benchmark simultaneously an input-output corpus, a grammar coverage resource, and a basis for qualitative inspection through rendered diagrams [2508.03215].

## 2. Corpus construction and validation pipeline

The corpus construction process begins with **161 unique system models** crawled from publicly available SysML teaching materials and example repositories, especially the **SysML-v2-Release** repository from the Systems Modeling Community. These raw examples exhibited three recurring problems: some lacked practical scenarios, some depended on cross-file references and were incomplete in isolation, and some used semantically uninformative names such as “example1” or “Items Example.” To make the corpus suitable for generation and evaluation, the first author manually removed models without practical scenarios, manually inlined referenced external elements so that each model became self-contained, and renamed models so their names reflected their semantics. After this preprocessing, the final benchmark contains **151 scenarios** [2508.03215].

The natural-language requirements were not scraped from external sources but **human-authored from the system models**. Two graduate-level software engineering students, each with over four years of software engineering experience and familiarity with system modeling concepts, carried out a dual-annotation process in which one annotator drafted a requirement and the other carefully double-checked it. The annotation criteria were **Naturalness**, meaning that the requirement should read like a real stakeholder’s description, and **Completeness**, meaning that all structural and behavioral aspects of the system model must be reflected in the natural-language requirement. This design places the benchmark closer to controlled translation from prose requirements into system models than to noisy web-scale alignment [2508.03215].

Validation operated on both sides of the benchmark pair. On the model side, each textual SysML model was loaded into a **SysML interpreter** to ensure that it compiled correctly and could produce a valid graphical notation diagram. On the requirements side, the authors checked that all elements in the system model were traceable to at least one sentence in the requirement and that the requirement did not contain extra content beyond what was in the model. This establishes a tight traceability relation between prose and formal artifact, which is especially important because later evaluation includes semantic decomposition of generated and reference models [2508.03215].

## 3. Dataset composition and annotation schema

The final benchmark contains **151 human-curated scenarios**. Its domain taxonomy was first derived from the official SysML tutorial, from which the top 10 domains were identified, and later extended when five samples did not fit any initial category. The result is a **13-domain** annotation scheme: **Vehicle Traffic**, **Photography Management / Photography Technique**, **Information Management**, **Simulation Calculation**, **Energy Materials**, **Network Communication**, **Fault Diagnosis**, **Aerospace**, **Confidentiality and Security**, **Systems Engineering**, **Embedded Device**, **Medical Health**, and **Water Resource Transportation**. The detailed domain counts are highly imbalanced: **Vehicle Traffic** has **108** samples; **Photography Technique** 12; **Information Management** 8; **Simulation Calculation** 4; **Energy materials**, **Network Communication**, **Fault diagnosis**, and **Aerospace** 3 each; **Confidentiality and security** and **Systems Engineering** 2 each; and **Embedded device**, **Medical Health**, and **Water resource transportation** 1 each [2508.03215].

Benchmark instances are also labeled by **key grammar**, using a taxonomy inherited from the source teaching materials. The grammar categories include constructs such as **Attribute**, **Generalization**, **Subsetting**, **Redefinition**, **Enumeration**, **Part**, **Item**, **Connection**, **Port**, **Function-based Behavior**, **Interface**, **State-based Behavior**, **Binding Connector**, **Requirement**, **Verification**, **Expression**, **Sequence Modeling**, **Action Definition**, **Conditional Succession**, **Assignment Action**, **Message**, **State Definition**, **Transition**, **Constraint**, **Use Case**, **Functional Allocation**, **Metadata**, **View**, and **Package**. This annotation scheme turns SysMBench into a grammar-stratified evaluation resource rather than a simple pool of undifferentiated scenarios [2508.03215].

Difficulty is approximated by **LoM (Lines of Model)**, defined as the number of non-blank, non-comment lines in the SysML file after preprocessing. The benchmark uses five bins: **Level 1** for $\text{LoM} < 30$ with **63** samples, **Level 2** for $30 \le \text{LoM} < 60$ with **64** samples, **Level 3** for $60 \le \text{LoM} < 90$ with **12** samples, **Level 4** for $90 \le \text{LoM} < 120$ with **6** samples, and **Level 5** for $\text{LoM} \ge 120$ with **6** samples. Corpus-level size statistics further report an **average requirement length of 127 tokens**, a minimum of **67**, and a maximum of **229**, together with an **average system model length of 41 lines**, a minimum of **3**, and a maximum of **158** [2508.03215].

## 4. SysML v2 representation and task interface

SysMBench is built entirely around **OMG SysML v2 textual notation**. The task prompt used in the benchmark states: “**You are a senior MBSE engineer. Task: Given the following natural-language requirements, create an OMG SysML v2 textual model. Return only valid SysML v2 code, no explanations or commentary.**” The expected output is therefore not an explanation of the model, but executable-form textual SysML code. This framing is significant because it couples natural-language understanding with formal syntax control and modeling-language competence [2508.03215].

The paper’s examples show that the benchmark exercises a broad subset of SysML constructs. Illustrative snippets include `package`, `private import`, `enum def`, `part def`, `attribute`, `specializes`, and `redefines`; other examples use `item def`, `ref item`, nested `part`, `interface def`, `end`, `connect`, directional feature references such as `::>`, multiplicities like `[1]`, `[4]`, and `[*]`, and relations such as `subsets`. A representative example defines a traffic-light model with an enumeration `TrafficLightColor`, a `TrafficLight` part with an attribute `currentColor`, and a specialized `TrafficLightGo` that `redefines currentColor = TrafficLightColor::green`. Another example shows a vehicle model with `item def Fuel`, `item def Person`, a `Vehicle` part containing `attribute mass : Real`, a `ref item driver : Person`, and nested parts for the fuel tank. These examples make clear that SysMBench targets both structural and behavioral constructs, typed references, specialization, and declarative relations [2508.03215].

The benchmark’s representation philosophy is therefore not limited to flat component lists. Its metadata and examples indicate evaluation over atomic structural and behavioral elements such as parts, items, attributes, references, ports, interfaces, actions, states, transitions, constraints, allocations, use cases, and view-related elements. A plausible implication is that benchmark performance depends not only on surface fluency in SysML syntax but also on the ability to preserve higher-level declarative relations such as specialization, redefinition, and subsetting, which later figures prominently in the error analysis [2508.03215].

## 5. Evaluation methodology and SysMEval

SysMBench is evaluated with three traditional text metrics—**BLEU**, **ROUGE-L (F1)**, and **BERTScore F1** using `bert-base-uncased`—and with **SysMEval**, a semantic-aware metric introduced specifically for system-model generation. The motivation for SysMEval is that token-overlap metrics are poorly aligned with declarative model semantics: semantically equivalent models can differ substantially in naming or ordering, while superficially similar strings may still encode incorrect structure. SysMEval is described as inspired by **Fact-QA / SyllabusQA** and **FActScore**, and it evaluates generated models by decomposing them into **atomic modeling claims** and checking whether those claims are semantically supported by the reference model [2508.03215].

Let $M_p$ be the generated system model and $M_t$ the human-written ground-truth system model, with claim sets $S_p$ and $S_t$ obtained by decomposition. SysMEval Precision and Recall are defined as:
$$
\text{SysMEval-P} = \frac{1}{|S_p|} \sum_{a \in S_p} \mathbb{I}[a \text{ is supported by } S_t]
$$
and
$$
\text{SysMEval-R} = \frac{1}{|S_t|} \sum_{a \in S_t} \mathbb{I}[a \text{ is covered by } S_p].
$$
The paper then computes **SysMEval-F1** from SysMEval-P and SysMEval-R. Conceptually, SysMEval-P measures semantic correctness of generated claims, whereas SysMEval-R measures semantic completeness relative to the reference [2508.03215].

Operationally, SysMEval is implemented using **GPT-4**, specifically **`gpt-4.1-2025-04-14`**, to perform both claim decomposition and semantic alignment. For precision, the evaluator lists all atomic modeling claims in the generated model and determines whether each is supported by the reference model; for recall, it lists all atomic claims in the reference model and determines whether each is covered by the generated model. The prompts explicitly instruct the evaluator to ignore formatting or identifier naming differences if the structure and semantics match. The paper also states that SysMEval uses a **chain-of-thought prompting strategy** internally, although detailed generation hyperparameters such as temperature, top-$p$, max tokens, stop sequences, number of samples, or random seeds are not reported in the visible benchmark description [2508.03215].

## 6. Experimental results, failure modes, and limitations

The benchmark evaluates **17 LLMs**, divided into **10 general LLMs** and **7 code LLMs**, under zero-shot direct prompting and three enhancement strategies: **few-shot prompting**, **chain-of-thought prompting**, and **grammar prompting** with a SysML BNF subset. The main zero-shot results are uniformly modest. **DeepSeek R1** achieves the best surface metrics with **BLEU 4.0**, **ROUGE 47**, and **BERTScore 69**. **GPT-4.1** achieves the highest **SysM-P = 71**, **DeepSeek R1** the highest **SysM-R = 60**, and **Qwen3-32B** the highest **SysM-F1 = 62**. The headline conclusion is that all evaluated models perform poorly on SysMBench, with the best BLEU only **4%** and the best SysMEval-F1 only **62%** [2508.03215].

Prompt engineering does not remove this gap. **Few-shot prompting** generally raises surface similarity metrics such as BLEU but often lowers semantic metrics; for example, **Claude 3 Opus** increases from **2.6%** to **6.1%** BLEU under few-shot prompting while **SysM-R** falls from **56%** to **47%**. **Chain-of-thought prompting** frequently hurts performance, which the authors attribute to redundant constructs, partially formed constructs, and error propagation into the final model. **Grammar prompting** strongly improves surface-form conformity—for **Claude 3**, BLEU rises from **2.6%** to **10.2%**—but often reduces semantic precision because models over-generate optional syntax, with **SysM-P** for Claude 3 dropping from **70%** to **57%**. The pattern suggests that syntax conditioning alone is insufficient and can even exacerbate semantic drift toward over-complete but unfaithful models [2508.03215].

The paper’s subgroup analyses indicate substantial heterogeneity by domain, difficulty, and grammar category. For **Qwen3-32B**, the best overall model by SysM-F1, the highest **SysM-P** appears in **calculation** and **network**, and **Aerospace** is also strong in precision; the highest **SysM-R** appears in **security** and is also strong in **calculation**. By contrast, **transportation** and **medical** are reported as much lower on both metrics. Across difficulty levels, surface metrics decline roughly monotonically: BLEU falls from **2.8** at Level 1 to **0.4** at Level 4 and **0.5** at Level 5, while SysMEval behaves non-monotonically, with **SysMEval-P** values of **67**, **68**, **58**, **64**, **51** and **SysMEval-R** values of **52**, **62**, **61**, **76**, **51** from Levels 1 through 5. This suggests that model size and formal fluency do not straightforwardly determine semantic completeness on larger models [2508.03215].

The qualitative error analysis emphasizes grammar-sensitive failures rather than mere lexical mismatch. A highlighted Claude 3 failure on a traffic-light scenario correctly handles enumeration and attribute rules but omits a required package import and misuses specialization grammar by collapsing a type-level specialization into an instance-level construct. Another case replaces an aggregation-plus-subsetting design with a flatter vehicle structure, omitting relations such as `subsets` even though the result remains superficially plausible. Grammar-level results for Qwen3-32B reinforce this pattern: **Generalization** achieves **SysM-P 1.000** and **SysM-R 1.000**; **Enumeration** reaches **SysM-P 1.000** and **SysM-R 0.813**; **Variant Configuration** reaches **SysM-P 0.951** and **SysM-R 0.905**; but **Interface** falls to **SysM-P 0.250** and **SysM-R 0.571**, **Binding Connector** to **0.188/0.283**, **View** to **0.221/0.241**, and **Requirement** to **0.328/0.453**. This suggests that some canonical structural forms are substantially easier for current models than specialized systems-engineering constructs [2508.03215].

Several limitations are explicit. The dataset is heavily imbalanced toward **Vehicle Traffic** with **108 of 151** samples. The source corpus comes from **public SysML teaching materials and examples**, which introduces tutorial bias relative to industrial repositories. Difficulty is approximated by **LoM**, which captures size but not necessarily conceptual complexity. SysMEval depends on **GPT-4.1** as an evaluator, which improves semantic sensitivity but introduces possible judge-model dependence. The benchmark covers only **SysML v2 textual notation**, not other modeling languages such as AADL or UML, and the visible description does not fully report decoding hyperparameters. Within those constraints, SysMBench establishes a concrete evaluation setting for a task on which current LLMs remain weak: faithful generation of formal system models from requirements in a real modeling language [2508.03215].

Source: https://www.emergentmind.com/topics/sysmbench