SysMEval: Multi-Faceted Evaluation

• SysMEval is a multifaceted term that encapsulates diverse evaluation approaches, from meta-programming compilation techniques to system-level and semantic assessment frameworks.
• It systematically reifies hidden evaluation processes, allowing explicit metric aggregation, statistical analysis, and semantic claim matching in complex system models.
• The framework extends to interactive, human-in-the-loop evaluations, enabling real-time assessments in tasks like text simplification and multimodal reasoning.

SysMEval is a label that appears with multiple meanings in recent arXiv literature. In its earliest and most technically specific use, it denotes the evaluation-centric view of compilation introduced for the Sysmel language, in which compiling a program is redefined as interpreting and evaluating its definition as a script inside a reflective environment $\Sigma$ (Salgado, 2023). In later work, the same label is used for system-level evaluation of multi-agent and foundation-model systems, for a semantic-aware metric over SysML v2 models, and for interactive human-in-the-loop evaluation settings in text simplification and multimodal reasoning (Emde et al., 9 Mar 2026, Ackerman et al., 30 Jan 2025, Jin et al., 5 Aug 2025, Roscan et al., 10 Apr 2026, Tang et al., 25 Aug 2025). The recurring idea is not a single shared implementation, but a family of approaches that move evaluation or compilation from a hidden subsystem to an explicit, inspectable, and programmable object of study.

1. Terminological scope

The literature uses “SysMEval” for several distinct constructs. The table summarizes the main senses explicitly described in the supplied papers.

Sense of SysMEval	Core object	Representative source
Evaluation-centric compilation	Program definition evaluated as a script in $\Sigma$	(Salgado, 2023)
System-level evaluation of multi-agent LLM systems	Entire agentic system as unit of analysis	(Emde et al., 9 Mar 2026)
Statistical multi-metric evaluation	Pairwise testing, metric aggregation, visualization	(Ackerman et al., 30 Jan 2025)
Semantic-aware metric for system models	Atomic-claim precision, recall, and F1	(Jin et al., 5 Aug 2025)
Interactive simplification evaluation	Prompt–model matrix, alignment, human scoring	(Roscan et al., 10 Apr 2026)
Controlled multimodal system evaluation	Semantically equivalent text–image inputs	(Tang et al., 25 Aug 2025)

This multiplicity is central to understanding the term. A common source of confusion is to treat SysMEval as a single benchmark, library, or metric. The papers instead use the same label for different technical artifacts: a compilation model, an evaluation methodology, a statistical framework, and a semantic scoring procedure.

2. SysMEval in Sysmel: compilation as evaluation

In “The Design and Implementation of an Extensible System Meta-Programming Language,” SysMEval is the evaluation-centric view of compilation for Sysmel. A “program definition” is an executable script in Sysmel whose evaluation directly orchestrates parsing, semantic analysis, construction of closures and function definitions, IR generation, optimization, and final emission of machine code and program entities. The environment $\Sigma$ exposes the parser, the AST, the semantic analyzer, optimizers, code generators, and backends via a meta-object protocol, making the compilation pipeline first-class and programmable (Salgado, 2023).

The core semantic picture is expressed as an evaluation relation

$E\llbracket P \rrbracket(\Sigma) \to \Sigma'$

where $P$ is a program definition and $\Sigma'$ is the environment after the program entity graph and compilation artifacts have been produced. The traditional pipeline is still present, but it is reified as composable transformations:

$$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$

Macros introduce additional pre-analysis rewrites $M: AST_0 \to AST_0$, and the environment chooses and composes these transformations under user control.

The reflective interface is organized around AST nodes, types, closures, passes, and backend emitters. AST nodes expose meta-level messages including #analyzeAndEvaluateWithEnvironment:, #analyzeWithEnvironment:, #evaluateWithEnvironment:, #compileBytecodesDirectlyWith:, and #generateSSAValueWith:. Function application analysis proceeds in two phases: macro expansion when the callee is a macro over unanalyzed argument nodes, followed by type-directed analysis. Message sends analyze the receiver first and then delegate analysis to the receiver’s type. The type system itself belongs to the MOP: types are instances of Type, support method lookup through #methodDictionary and #lookupSelector:, and provide application typing and message-send analysis. The same MOP also supports #ref and #pointer, enabling C-like references and pointers, assignment through #:=, address operations, and conversion operations.

The compilation substrate is stratified into a high-level IR, a middle-level IR, and a low-level IR. The high-level IR is SSA-inspired by LLVM and represents function applications, message sends, allocas, loads and stores, and primitive intrinsics as calls; its optimizations include constant propagation, control-flow simplification, and inlining. The middle-level IR is a mostly portable three-address SSA code with single machine primitives per instruction, plus branch-combine, register allocation, stack frame layout, and debug info preparation. The low-level IR produces assembly and object file emission for ELF, COFF, and Mach-O, with relocation annotations for serialized program entity graphs.

SysMEval is also meta-circular. The compiler’s semantic analysis, macro system, type system, and code-generation entry points are defined in Sysmel itself; AST nodes and types are ordinary Sysmel classes; lambda nodes evaluate to closures carrying a capture vector and a function definition; and IR generation is invoked through AST-node messages. The self-hosting fixed point is described as bootstrapping iterations

$L_{n+1} = \mathrm{compile}_L(L_n, source_L),$

seeking a stable $L^*$. The bootstrap begins with a minimal C implementation containing parsing, base semantic analysis, a LISP2 compacting GC, a register-based bytecode, and a simple x86_64 JIT. The paper reports feasibility by achieving three full self-compilation cycles.

Several limitations are explicit. Since self-compilation starts from analyzed in-memory AST nodes after Phase 0, parsing and semantic analysis are not fully validated by later cycles. Serializing a copy of the fully analyzed AST and meta-objects increases memory footprint. The paper does not specify a dedicated sandbox for meta-programming effects, so extensions operate inside the image or environment and assume trusted code or careful curation of extension points.

3. SysMEval as system-level evaluation infrastructure

A second major usage appears in work on LLM systems. In “MASEval: Extending Multi-Agent Evaluation from Models to Systems,” SysMEval denotes system-level evaluation in which the entire agentic system—agents, framework implementation, orchestration logic, topology, memory, error handling, tools, user simulation, evaluators, and model adapters—is treated as the unit of analysis. MASEval operationalizes this with a framework-agnostic library organized around the abstractions Task, Benchmark, Environment, AgentAdapter, User, Evaluator, and ModelAdapter, and a five-phase lifecycle Setup → Execute → Collect → Evaluate → Report (Emde et al., 9 Mar 2026).

MASEval’s empirical claim is that framework choice can matter as much as model choice. Across three benchmarks, three models, and three frameworks, averaging over six domains yields mean ranges of $\Sigma$0 percentage points across models and $\Sigma$1 percentage points across frameworks, with mean standard deviations of $\Sigma$2 and $\Sigma$3 percentage points respectively. The paper reports, for example, a $\Sigma$4 percentage-point framework gap on MACS Travel for Haiku-4.5, where smolagents scores $\Sigma$5 and LlamaIndex scores $\Sigma$6. The framework is explicitly BYO, allows swapping frameworks and providers, uses structured error attribution, supports repetitions through n_task_repeats, and records per-agent traces, token usage, and latency.

In “TaskEval: Synthesised Evaluation for Foundation-Model Tasks,” a related SysMEval perspective is realized by synthesizing task-specific evaluators when neither a metric nor a dataset exists. The approach is implemented in the prototype GenValidator and is organized around a task-agnostic meta-model, a human-in-the-loop interaction protocol, and an eval synthesiser that selects or generates evaluators from the strategy classes summarize, visualize, judge, and logic program (Widanapathiranage et al., 4 Dec 2025). The minimal formalization given in the paper is

$\Sigma$7

with $\Sigma$8 input space, $\Sigma$9 output space, $\Sigma$0 constraints, $\Sigma$1 evidence requirements, $\Sigma$2 evaluation objectives, and $\Sigma$3 strategy templates. Preliminary experiments on chart data extraction and document question answering report evaluator accuracies of $\Sigma$4 and $\Sigma$5 respectively.

In “S-Eval: Towards Automated and Comprehensive Safety Evaluation for LLMs,” SysMEval is instantiated as a system-level safety evaluation pipeline. S-Eval combines an expert testing LLM $\Sigma$6 with a safety critique LLM $\Sigma$7, and grounds generation in a four-level taxonomy spanning $\Sigma$8 dimensions, $\Sigma$9 categories, $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$0 subcategories, and $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$1 sub-subcategories (Yuan et al., 2024). Base risk prompts and attack prompts are generated and filtered by quality control, while $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$2 returns a binary decision $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$3, a safety confidence score $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$4, and an explanation. The evaluation aggregates per-category Safety Score

$E\llbracket P \rrbracket(\Sigma) \to \Sigma'$5

and per-attack Attack Success Rate

$E\llbracket P \rrbracket(\Sigma) \to \Sigma'$6

The released benchmark contains $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$7 prompts, comprising $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$8 base risk prompts and $E\llbracket P \rrbracket(\Sigma) \to \Sigma'$9 attack prompts.

4. Statistical and multimodal SysMEval formulations

In “Statistical multi-metric evaluation and visualization of LLM system predictive performance,” SysMEval is a general-purpose framework for statistically sound evaluation across multiple metrics and datasets. It operates on score samples $P$0, score-sample lists $P$1 for one metric on one dataset, and collections $P$2 spanning multiple metrics or datasets. Test selection is automatic: Welch’s two-sample $P$3-test for unpaired numeric data, the independent proportions $P$4-test for unpaired binary data, the paired $P$5-test for paired numeric data, and McNemar’s test for paired binary data. Within paired datasets, metrics are standardized and aligned so “higher is better,” then aggregated element-wise as

$P$6

Across metrics or datasets, p-values are combined using Wilson’s harmonic mean p-value, and effect sizes are aggregated by inverse-variance weighting (Ackerman et al., 30 Jan 2025).

This formulation is explicitly designed to avoid common evaluation pathologies: unprincipled averaging, failure to respect pairing, and repeated aggregation of already adjusted p-values. Multiple comparisons within a metric or dataset are handled by Holm–Šidák step-down adjustment, while HMP aggregation is used across metrics or datasets for a fixed pair of systems. In the CrossCodeEval demonstration, the framework ranks systems by aggregated standardized means and reports that all system pairs are statistically significantly different from each other under the HMP-aggregated analysis across the four language datasets.

A modality-controlled variant appears in “SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-LLMs.” Here SysMEval refers to controlled, system-level multimodal evaluation in which semantically equivalent text and image inputs are constructed across chess, chemistry, music, and graphs or networks. The benchmark measures modality-specific accuracies,

$P$7

cross-modal agreement,

$P$8

and modality imbalance,

$P$9

Across $\Sigma'$0 contemporary VLMs, the paper reports systematic modality imbalance, with vision frequently lagging language and cross-modal agreement remaining relatively low; robustness experiments indicate that results are largely stable under semantics-preserving visual transformations (Tang et al., 25 Aug 2025).

5. SysMEval as a semantic-aware metric for system model generation

In “A System Model Generation Benchmark from Natural Language Requirements,” SysMEval is not a system-level protocol but a semantic-aware evaluation metric for LLM-generated system models in SysML v2 textual notation. The metric converts the generated model $\Sigma'$1 and human reference $\Sigma'$2 into atomic claim sets

$\Sigma'$3

where claims are minimal, verifiable statements about both structure and behavior. Precision and recall are defined at the claim level:

$\Sigma'$4

$\Sigma'$5

with

$\Sigma'$6

All claims have equal weight, and “supported” or “covered” is judged semantically rather than by string identity (Jin et al., 5 Aug 2025).

The claim inventory spans structural aspects such as package, part, item, attribute, multiplicity, containment, associations, generalization, subsetting, redefinition, ports, interfaces, connectors, dependency, and imports, and behavioral aspects such as action definitions, parameters, succession, assignments, states, transitions, guards, messages, and sequence modeling. The implementation does not parse models into graphs or ASTs; instead, GPT-4.1 is used as a semantic judge to extract claims and assess claim-level alignment, returning SysMEval-Precision, SysMEval-Recall, and SysMEval-F1.

The main empirical result is that LLMs perform poorly on SysMBench under both traditional and semantic-aware metrics. The highest BLEU reported is approximately $\Sigma'$7, while the highest SysMEval-F1 is approximately $\Sigma'$8. The contrast is part of the paper’s argument: low token overlap does not imply low semantic fidelity, and conversely syntactically similar outputs may still contain meaning-breaking errors such as wrong types, ports, or state transitions. The metric therefore penalizes hallucinations through precision and omissions through recall.

The paper is equally explicit about limitations. SysMEval depends on GPT-4.1 as an LLM judge, does not build a formal AST or graph, does not formally check behavior-level equivalence such as bisimulation of state machines, and inherits the ambiguity of natural-language semantic adjudication for cases such as subsetting versus equivalent explicit feature sets.

6. Interactive and human-centered variants

In “MuTSE: A Human-in-the-Loop Multi-use Text Simplification Evaluator,” SysMEval is described as systematic multi-dimensional evaluation for text simplification. MuTSE executes arbitrary $\Sigma'$9 prompt–model permutations concurrently, produces a real-time comparison matrix, computes readability diagnostics inline, and supports customizable human scoring with exportable JSON and CSV artifacts (Roscan et al., 10 Apr 2026). The system is a decoupled client–server platform with a FastAPI/Python backend and a Vue.js 3 frontend.

Its central technical component is a tiered semantic alignment engine. Tier 1 uses the sentence-transformers model “paraphrase-multilingual-MiniLM-L12-v2” to compute cosine similarities between source and simplified sentence embeddings; Tier 2 falls back to TF–IDF cosine similarity; Tier 3 uses a positional scaffold. Alignment is regularized by a linearity bias heuristic $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$0, with default $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$1, through

$$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$2

Each simplified sentence $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$3 is mapped to the source sentence $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$4 that maximizes this score. The paper’s interpretation is that higher $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$5 values favor the near-linear, monotonic progressions typical of monolingual simplification and reduce false positives from semantically similar but positionally distant matches.

MuTSE also integrates readability and human scoring. It reports Flesch–Kincaid Grade Level, Flesch Reading Ease, compression ratio

$$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$6

and user-defined weighted evaluation:

$$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$7

where $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$8 is the normalized score for dimension $$C = T_{\text{codegen}} \circ (\cdots \circ T_{\text{opt}}^2 \circ T_{\text{opt}}^1) \circ T_{\text{sem}} \circ T_{\text{parse}}.$$9. The framework is therefore not only a metric suite but an annotation environment in which prompt design, model comparison, alignment, and human judgment are co-located.

Taken together, these uses show that “SysMEval” is best understood as a recurrent design orientation rather than a single canonical artifact. In one line of work it redefines compilation itself as evaluation inside a reflective environment; in another it expands evaluation from models to full systems; in another it introduces semantics-aware claim matching; and in another it couples automated metrics with interactive human inspection. The shared emphasis is first-class treatment of structure, semantics, and system context, but the concrete object denoted by the term changes substantially across papers.