SysMBench: Formal SysML v2 Benchmark

• SysMBench is a benchmark for natural-language-to-SysML v2 model generation, combining realistic requirements with formal modeling artifacts.
• It features 151 human-curated scenarios annotated with requirements, SysML code, diagrams, domain and grammar labels, and difficulty levels for comprehensive evaluation.
• Evaluation leverages traditional text metrics and the semantic-aware SysMEval metric, assessing both surface fluency and structural accuracy of generated models.

SysMBench is a benchmark for evaluating whether LLMs can generate formal system models in OMG SysML v2 textual notation from natural-language requirements. Each scenario pairs an English requirements description with a human-curated SysML model, a rendered graphical diagram, and metadata including a domain label, a key grammar label, and a difficulty level. Introduced in 2025 together with the semantic-aware evaluation framework SysMEval, SysMBench targets end-to-end natural-language-to-system-model generation rather than entity extraction or toy modeling tasks, and it is presented as the first public benchmark for SysML-based system model generation from natural-language requirements (Jin et al., 5 Aug 2025).

1. Definition, scope, and modeling task

SysMBench addresses a specific problem in model-based systems engineering and software development: system models are valuable artifacts for requirements analysis and architecture design, but constructing them is difficult because SysML syntax is specialized, public examples are relatively scarce, and manual model development is slow and error-prone. The benchmark asks whether LLMs can translate natural-language requirements into formal system models in a real modeling language, rather than merely recover isolated entities or relations. In this formulation, each benchmark instance provides a requirements description as input and expects a complete SysML v2 textual model as output (Jin et al., 5 Aug 2025).

The benchmark is deliberately built around declarative model generation. The authors contrast it with prior work that often reduced system modeling to information extraction on very small handcrafted datasets and did not require generation in a concrete modeling language. SysMBench instead evaluates full-text generation in SysML v2 textual notation. The benchmark overview identifies six components per sample: Requirements, System model with textual notation, System model with graphical notation, Domain label, Key grammar, and Difficulty level. This organization makes the benchmark simultaneously an input-output corpus, a grammar coverage resource, and a basis for qualitative inspection through rendered diagrams (Jin et al., 5 Aug 2025).

2. Corpus construction and validation pipeline

The corpus construction process begins with 161 unique system models crawled from publicly available SysML teaching materials and example repositories, especially the SysML-v2-Release repository from the Systems Modeling Community. These raw examples exhibited three recurring problems: some lacked practical scenarios, some depended on cross-file references and were incomplete in isolation, and some used semantically uninformative names such as “example1” or “Items Example.” To make the corpus suitable for generation and evaluation, the first author manually removed models without practical scenarios, manually inlined referenced external elements so that each model became self-contained, and renamed models so their names reflected their semantics. After this preprocessing, the final benchmark contains 151 scenarios (Jin et al., 5 Aug 2025).

The natural-language requirements were not scraped from external sources but human-authored from the system models. Two graduate-level software engineering students, each with over four years of software engineering experience and familiarity with system modeling concepts, carried out a dual-annotation process in which one annotator drafted a requirement and the other carefully double-checked it. The annotation criteria were Naturalness, meaning that the requirement should read like a real stakeholder’s description, and Completeness, meaning that all structural and behavioral aspects of the system model must be reflected in the natural-language requirement. This design places the benchmark closer to controlled translation from prose requirements into system models than to noisy web-scale alignment (Jin et al., 5 Aug 2025).

Validation operated on both sides of the benchmark pair. On the model side, each textual SysML model was loaded into a SysML interpreter to ensure that it compiled correctly and could produce a valid graphical notation diagram. On the requirements side, the authors checked that all elements in the system model were traceable to at least one sentence in the requirement and that the requirement did not contain extra content beyond what was in the model. This establishes a tight traceability relation between prose and formal artifact, which is especially important because later evaluation includes semantic decomposition of generated and reference models (Jin et al., 5 Aug 2025).

3. Dataset composition and annotation schema

The final benchmark contains 151 human-curated scenarios. Its domain taxonomy was first derived from the official SysML tutorial, from which the top 10 domains were identified, and later extended when five samples did not fit any initial category. The result is a 13-domain annotation scheme: Vehicle Traffic, Photography Management / Photography Technique, Information Management, Simulation Calculation, Energy Materials, Network Communication, Fault Diagnosis, Aerospace, Confidentiality and Security, Systems Engineering, Embedded Device, Medical Health, and Water Resource Transportation. The detailed domain counts are highly imbalanced: Vehicle Traffic has 108 samples; Photography Technique 12; Information Management 8; Simulation Calculation 4; Energy materials, Network Communication, Fault diagnosis, and Aerospace 3 each; Confidentiality and security and Systems Engineering 2 each; and Embedded device, Medical Health, and Water resource transportation 1 each (Jin et al., 5 Aug 2025).

Benchmark instances are also labeled by key grammar, using a taxonomy inherited from the source teaching materials. The grammar categories include constructs such as Attribute, Generalization, Subsetting, Redefinition, Enumeration, Part, Item, Connection, Port, Function-based Behavior, Interface, State-based Behavior, Binding Connector, Requirement, Verification, Expression, Sequence Modeling, Action Definition, Conditional Succession, Assignment Action, Message, State Definition, Transition, Constraint, Use Case, Functional Allocation, Metadata, View, and Package. This annotation scheme turns SysMBench into a grammar-stratified evaluation resource rather than a simple pool of undifferentiated scenarios (Jin et al., 5 Aug 2025).

Difficulty is approximated by LoM (Lines of Model), defined as the number of non-blank, non-comment lines in the SysML file after preprocessing. The benchmark uses five bins: Level 1 for $\text{LoM} < 30$ with 63 samples, Level 2 for $30 \le \text{LoM} < 60$ with 64 samples, Level 3 for $60 \le \text{LoM} < 90$ with 12 samples, Level 4 for $90 \le \text{LoM} < 120$ with 6 samples, and Level 5 for $\text{LoM} \ge 120$ with 6 samples. Corpus-level size statistics further report an average requirement length of 127 tokens, a minimum of 67, and a maximum of 229, together with an average system model length of 41 lines, a minimum of 3, and a maximum of 158 (Jin et al., 5 Aug 2025).

4. SysML v2 representation and task interface

SysMBench is built entirely around OMG SysML v2 textual notation. The task prompt used in the benchmark states: “You are a senior MBSE engineer. Task: Given the following natural-language requirements, create an OMG SysML v2 textual model. Return only valid SysML v2 code, no explanations or commentary.” The expected output is therefore not an explanation of the model, but executable-form textual SysML code. This framing is significant because it couples natural-language understanding with formal syntax control and modeling-language competence (Jin et al., 5 Aug 2025).

The paper’s examples show that the benchmark exercises a broad subset of SysML constructs. Illustrative snippets include package, private import, enum def, part def, attribute, specializes, and redefines; other examples use item def, ref item, nested part, interface def, end, connect, directional feature references such as ::>, multiplicities like [1], [4], and [*], and relations such as subsets. A representative example defines a traffic-light model with an enumeration TrafficLightColor, a TrafficLight part with an attribute currentColor, and a specialized TrafficLightGo that redefines currentColor = TrafficLightColor::green. Another example shows a vehicle model with item def Fuel, item def Person, a Vehicle part containing attribute mass : Real, a ref item driver : Person, and nested parts for the fuel tank. These examples make clear that SysMBench targets both structural and behavioral constructs, typed references, specialization, and declarative relations (Jin et al., 5 Aug 2025).

The benchmark’s representation philosophy is therefore not limited to flat component lists. Its metadata and examples indicate evaluation over atomic structural and behavioral elements such as parts, items, attributes, references, ports, interfaces, actions, states, transitions, constraints, allocations, use cases, and view-related elements. A plausible implication is that benchmark performance depends not only on surface fluency in SysML syntax but also on the ability to preserve higher-level declarative relations such as specialization, redefinition, and subsetting, which later figures prominently in the error analysis (Jin et al., 5 Aug 2025).

5. Evaluation methodology and SysMEval

SysMBench is evaluated with three traditional text metrics—BLEU, ROUGE-L (F1), and BERTScore F1 using bert-base-uncased—and with SysMEval, a semantic-aware metric introduced specifically for system-model generation. The motivation for SysMEval is that token-overlap metrics are poorly aligned with declarative model semantics: semantically equivalent models can differ substantially in naming or ordering, while superficially similar strings may still encode incorrect structure. SysMEval is described as inspired by Fact-QA / SyllabusQA and FActScore, and it evaluates generated models by decomposing them into atomic modeling claims and checking whether those claims are semantically supported by the reference model (Jin et al., 5 Aug 2025).

Let $M_p$ be the generated system model and $M_t$ the human-written ground-truth system model, with claim sets $S_p$ and $S_t$ obtained by decomposition. SysMEval Precision and Recall are defined as:

$$\text{SysMEval-P} = \frac{1}{|S_p|} \sum_{a \in S_p} \mathbb{I}[a \text{ is supported by } S_t]$$

and

$30 \le \text{LoM} < 60$0

The paper then computes SysMEval-F1 from SysMEval-P and SysMEval-R. Conceptually, SysMEval-P measures semantic correctness of generated claims, whereas SysMEval-R measures semantic completeness relative to the reference (Jin et al., 5 Aug 2025).

Operationally, SysMEval is implemented using GPT-4, specifically gpt-4.1-2025-04-14, to perform both claim decomposition and semantic alignment. For precision, the evaluator lists all atomic modeling claims in the generated model and determines whether each is supported by the reference model; for recall, it lists all atomic claims in the reference model and determines whether each is covered by the generated model. The prompts explicitly instruct the evaluator to ignore formatting or identifier naming differences if the structure and semantics match. The paper also states that SysMEval uses a chain-of-thought prompting strategy internally, although detailed generation hyperparameters such as temperature, top-$30 \le \text{LoM} < 60$1, max tokens, stop sequences, number of samples, or random seeds are not reported in the visible benchmark description (Jin et al., 5 Aug 2025).

6. Experimental results, failure modes, and limitations

The benchmark evaluates 17 LLMs, divided into 10 general LLMs and 7 code LLMs, under zero-shot direct prompting and three enhancement strategies: few-shot prompting, chain-of-thought prompting, and grammar prompting with a SysML BNF subset. The main zero-shot results are uniformly modest. DeepSeek R1 achieves the best surface metrics with BLEU 4.0, ROUGE 47, and BERTScore 69. GPT-4.1 achieves the highest SysM-P = 71, DeepSeek R1 the highest SysM-R = 60, and Qwen3-32B the highest SysM-F1 = 62. The headline conclusion is that all evaluated models perform poorly on SysMBench, with the best BLEU only 4% and the best SysMEval-F1 only 62% (Jin et al., 5 Aug 2025).

Prompt engineering does not remove this gap. Few-shot prompting generally raises surface similarity metrics such as BLEU but often lowers semantic metrics; for example, Claude 3 Opus increases from 2.6% to 6.1% BLEU under few-shot prompting while SysM-R falls from 56% to 47%. Chain-of-thought prompting frequently hurts performance, which the authors attribute to redundant constructs, partially formed constructs, and error propagation into the final model. Grammar prompting strongly improves surface-form conformity—for Claude 3, BLEU rises from 2.6% to 10.2%—but often reduces semantic precision because models over-generate optional syntax, with SysM-P for Claude 3 dropping from 70% to 57%. The pattern suggests that syntax conditioning alone is insufficient and can even exacerbate semantic drift toward over-complete but unfaithful models (Jin et al., 5 Aug 2025).

The paper’s subgroup analyses indicate substantial heterogeneity by domain, difficulty, and grammar category. For Qwen3-32B, the best overall model by SysM-F1, the highest SysM-P appears in calculation and network, and Aerospace is also strong in precision; the highest SysM-R appears in security and is also strong in calculation. By contrast, transportation and medical are reported as much lower on both metrics. Across difficulty levels, surface metrics decline roughly monotonically: BLEU falls from 2.8 at Level 1 to 0.4 at Level 4 and 0.5 at Level 5, while SysMEval behaves non-monotonically, with SysMEval-P values of 67, 68, 58, 64, 51 and SysMEval-R values of 52, 62, 61, 76, 51 from Levels 1 through 5. This suggests that model size and formal fluency do not straightforwardly determine semantic completeness on larger models (Jin et al., 5 Aug 2025).

The qualitative error analysis emphasizes grammar-sensitive failures rather than mere lexical mismatch. A highlighted Claude 3 failure on a traffic-light scenario correctly handles enumeration and attribute rules but omits a required package import and misuses specialization grammar by collapsing a type-level specialization into an instance-level construct. Another case replaces an aggregation-plus-subsetting design with a flatter vehicle structure, omitting relations such as subsets even though the result remains superficially plausible. Grammar-level results for Qwen3-32B reinforce this pattern: Generalization achieves SysM-P 1.000 and SysM-R 1.000; Enumeration reaches SysM-P 1.000 and SysM-R 0.813; Variant Configuration reaches SysM-P 0.951 and SysM-R 0.905; but Interface falls to SysM-P 0.250 and SysM-R 0.571, Binding Connector to 0.188/0.283, View to 0.221/0.241, and Requirement to 0.328/0.453. This suggests that some canonical structural forms are substantially easier for current models than specialized systems-engineering constructs (Jin et al., 5 Aug 2025).

Several limitations are explicit. The dataset is heavily imbalanced toward Vehicle Traffic with 108 of 151 samples. The source corpus comes from public SysML teaching materials and examples, which introduces tutorial bias relative to industrial repositories. Difficulty is approximated by LoM, which captures size but not necessarily conceptual complexity. SysMEval depends on GPT-4.1 as an evaluator, which improves semantic sensitivity but introduces possible judge-model dependence. The benchmark covers only SysML v2 textual notation, not other modeling languages such as AADL or UML, and the visible description does not fully report decoding hyperparameters. Within those constraints, SysMBench establishes a concrete evaluation setting for a task on which current LLMs remain weak: faithful generation of formal system models from requirements in a real modeling language (Jin et al., 5 Aug 2025).