- The paper introduces a verification-gated agent harness that grounds generated Structured Text in complete PLC projects and permits delivery only after logged specification, compilation, and runtime checks pass.
- SemaPLC achieved a 72.6% mean verified pass rate on 117 function tasks, compared with 63.9% for Agents4PLC, while improving integrated compilation to 89.4% and dynamic behavior to 52.2% on 65 project tasks.
- The results show that live execution exposes timer, state, interlock, and output-timing defects missed by static checks, although runtime reliability remains limited and requires substantially higher interaction costs.
Motivation and problem
Programmable logic controllers (PLCs) programmed in IEC 61131-3 Structured Text (ST) govern most industrial automation, and prior work has established that LLMs can generate individual program organization units (POUs) with compiler-in-the-loop repair, formal property checking via PLCverif, multi-agent iteration, and vendor IDE integration. The authors argue that this literature measures the wrong thing for deployment: in production, control logic is rarely an isolated POU, and two requirements go unmeasured. Project grounding requires generated logic to integrate into an established project, reusing its function blocks, variables, types, and build conventions. Correct runtime behavior requires that the integrated program, even when it compiles and passes static checks, exhibits correct timer, state-transition, interlock, and output-timing behavior on a live controller. Prior systems execute generated code to demonstrate that it can run, not to measure how reliably it runs; benchmark units remain predominantly independent POUs with limited runtime tests.
SemaPLC addresses both the method and the measurement gap. The method is an agent harness whose defining feature is not its tool set but a completion discipline: the agent may not terminate on its own judgment, and every claimed pass must be corroborated by a tool log. The measurement is a two-track evaluation that scores integrated compilation, static behavior, and dynamic behavior separately at benchmark scale across seven backbone models.
Harness design
SemaPLC runs on a generic, PLC-agnostic event-driven tool-use core and organizes five components: an agent core for planning, editing, and result interpretation; project/task grounding; a PLC skill library; verification processes; and a verification gate. All components act through a single PLC MCP tool server (with a CLI equivalent) exposing syntax checking, compilation, deployment, runtime status, live variable reading and forcing, trace sampling, and scripted behavior checks. Domain knowledge is externalized as Markdown documents — a rules file, a curated wiki of function-block signatures and compiler pitfalls, and procedural skills — none containing benchmark answers.
Three verification sources feed the gate uniformly. A structured specification audit checks the candidate clause-by-clause against the natural-language requirement, targeting defects a compiler accepts but the requirement forbids (missing drivers for published variables, threshold polarity, interlock violations, scan-cycle state semantics). Compilation returns line-anchored diagnostics, of which only the first is fed back to keep repair local. Live runtime validation builds and deploys the program, injects agent-derived scenarios, samples external variables, and compares against requirement-derived runtime assertions; failure stages prescribe repair targets (a flat trace indicates wiring, a wrong trace indicates block logic, a late transition indicates a timer or edge detector).
The gate enforces three invariants that together yield a delivery-integrity guarantee. Bounded retries: at most two repair rounds per check. Edit invalidation: any modification voids all prior verdicts, so verdicts attach to exact bytes. Earned claims: each outcome is a machine-readable sentinel cross-validated against the tool-call log; an unlogged self-report is downgraded to unchecked. The delivered file is additionally hash-checked against the last successful compilation and must contain no located address literals, since input injection occurs on a separate test copy.
Evaluation design
The function track reuses the 117 independent-POU Agents4PLC tasks under a strict verified-pass criterion: a held-out judge (RuSTy compilation plus PLCverif/nuXmv model checking) counts a task as passed only when at least 80% of requirement-derived properties are verified satisfied, with inconclusive verdicts (unsupported constructs, translation failures, timeouts) counted as failures. Notably, the authors audited the released Agents4PLC oracle and found defects — wrong constants, tautological assertions, contradictory properties, invented thresholds — in 43 of 117 tasks, which PLC engineers repaired before evaluation. The project-context track derives 65 tasks over ten industrial plants from Spec2Control: given a section narrative, a function-block catalog, and an empty entry harness, generated logic must compile and run inside a full ST project. Dynamic behavior is scored by deploying candidate and hidden reference to a live runtime under up to six scenarios and comparing traces on core output ports; any build, deployment, or timeout failure on either side scores zero. A structural separation between what methods see during generation and what scores them is documented, with a leakage audit finding no verbatim copying — though the paper concedes that agent-derived and scoring scenarios can align through the shared requirement, so dynamic scores measure behavior on the benchmark scenarios rather than generalization.
Seven backbone models across five vendors and two capability tiers (MiniMax-M2.7/M3, Qwen3.5-Plus, DeepSeek-V4-Flash/Pro, GLM-5.2, GPT-5.5) are compared against LLM4PLC, AutoPLC, and Agents4PLC, plus a "bare" SemaPLC configuration with the harness stripped to measure the harness's contribution.
Function-track results
SemaPLC attains the highest strict verified pass rate on all seven models, with a mean of 72.6% versus 63.9% for the strongest baseline (Agents4PLC) and 30.2% for LLM4PLC. It wins even on GPT-5.5 (82.1% vs 79.5%), and its worst model (67.5%) exceeds every baseline's mean. The cross-model spread shrinks from 25–31 points for baselines to 14.6 for SemaPLC, consistent with the interpretation that external, model-independent checks keep weaker backbones from losing much.
| Model |
LLM4PLC |
AutoPLC |
Agents4PLC |
SemaPLC bare |
SemaPLC full |
| MiniMax-M2.7 |
22.2 |
49.6 |
53.8 |
39.3 |
69.2 |
| MiniMax-M3 |
15.4 |
65.0 |
55.6 |
60.7 |
69.2 |
| Qwen3.5-Plus |
13.7 |
67.5 |
67.5 |
62.4 |
75.2 |
| DeepSeek-V4-Flash |
41.0 |
54.7 |
54.7 |
34.2 |
67.5 |
| DeepSeek-V4-Pro |
43.6 |
61.5 |
62.4 |
55.6 |
69.2 |
| GLM-5.2 |
30.8 |
59.0 |
74.4 |
63.2 |
76.1 |
| GPT-5.5 |
44.4 |
79.5 |
78.6 |
71.8 |
82.1 |
| Mean |
30.2 |
62.4 |
63.9 |
55.3 |
72.6 |
The bare-to-full comparison shows gains of 8.5 to 33.3 points per model, largest on the weakest backbones (DeepSeek-V4-Flash +33.3), with the cross-model spread contracting from 37.6 to 14.6 points and bare compile rates rising from 85.5% to 99.2%. The authors are careful to note that bare-versus-full conflates declarative knowledge (skills, wiki) with the verification loop, so the gain is attributed to the harness as a whole.
Project-track results and the runtime gap
The project track is where the paper's central empirical claim is established. On integrated compilation, SemaPLC averages 89.4% against 58.7–81.5 for baselines. On static behavior it leads on five of seven models (81.6 mean vs 71.7–75.7). The decisive layer is dynamic behavior: SemaPLC averages 52.2 versus at most 31.4 (AutoPLC), is best on every model, and never drops below 31.3, while fixed pipelines fall to single digits on their weakest backbones (e.g., LLM4PLC at 3.0 on MiniMax-M2.7).
| Method |
Integrated compile (mean) |
Static behavior (mean) |
Dynamic behavior (mean) |
| LLM4PLC |
58.7 |
75.7 |
22.4 |
| AutoPLC |
81.5 |
74.0 |
31.4 |
| Agents4PLC |
71.2 |
71.7 |
30.3 |
| SemaPLC |
89.4 |
81.6 |
52.2 |
The layer profile supports the paper's strongest claim: the three baselines sit within 4.0 static points of one another but spread 9.0 dynamic points, and SemaPLC shows the smallest static-to-dynamic drop (29.4 points vs 41.4–53.3). Similar static scores therefore do not imply similar runtime behavior, and an evaluation that stops before execution cannot separate reliable methods from unreliable ones. Even so, the best dynamic score (65.4) remains far below its static counterpart, which the authors state plainly: runtime PLC generation is far from solved. The advantage also narrows at the frontier — on GPT-5.5 the dynamic lead over Agents4PLC is only 1.8 points, and SemaPLC trails baselines on static behavior there (84.1 vs 88.8).
A case study from the coking-refinery plant illustrates the mechanism: two compiling candidates mishandle competing conditions on one setpoint (a low-flow write overwriting a fault default), a defect invisible to compilation and static analysis; runtime forcing of the low-flow input produces a concrete 500-versus-2500 mismatch, localizing a cause-specific repair that re-verification then confirms.
A cumulative ablation on DeepSeek-V4-Flash adds the three verification layers one at a time. Dynamic behavior rises monotonically from 23.1 (generation only) to 30.3, 43.7, and 54.1 with spec, compilation, and runtime layers, while static behavior moves far less (71.5 to 78.0). Cost climbs correspondingly from 34k tokens and 8.9 requests per task to 129k and 47.8, with runtime validation the most expensive stage. Decomposing the 3,590 individual scenario–port checks shows that each layer primarily converts structurally failing checks (not built, missing ports: 74.7% down to 21.5%) into measurable ones; the share of observed wrong values rises (2.1% to 24.4%), which the authors correctly frame as a coverage-expansion byproduct rather than degradation — a check must run before it can be wrong. Residual wrong values concentrate in limit-breach scenarios (17.3% vs 7.1% under normal operation).
The function track motivates runtime validation from the formal side: across 1,293 delivered programs, the PLCverif/nuXmv pipeline reaches conclusive verdicts on 75.7% of properties for programs without REAL types or timers and 87.0% for REAL-bearing programs, but 0 of 174 properties across the 32 timer-bearing (TON) programs obtain a conclusive verdict. The stateful timing constructs that formal checking cannot cover in this pipeline are precisely what runtime validation exercises directly.
Interaction cost
On the function track, SemaPLC matches Agents4PLC's request count (6.5 vs 6.3 per task) and is substantially faster in wall-clock (71 s vs 454 s median), because Agents4PLC's per-iteration model checking dominates its runtime. On the project track, wall-clock is comparable (347 vs 344 s) but SemaPLC issues 34.1 requests per task against 6.9 — its dynamic lead is paid for in model interactions. The authors attribute this to architecture rather than inefficiency: the baseline's fixed multi-agent workflow bounds iterations nearly constant across models, whereas SemaPLC's open-ended gate-driven loop scales requests with the backbone's competence (16.4 to 60.4 per model).
Limitations and open questions
The paper is explicit about its boundaries. Dynamic scoring exercises a bounded scenario set (up to six per task) derived from the hidden reference, so behavior under unseen operating conditions is unmeasured — a consequence of the agent-derived and scoring scenarios sharing a common requirement source. The harness's advantage narrows sharply on the strongest model, raising the question of how much of the dynamic gain persists as backbones improve. The layer ablation is single-model, so the decomposition of gains across verification layers may not transfer across backbones. Bare-versus-full attribution cannot separate declarative knowledge from the verification loop. The formal-coverage result is specific to the PLCverif/nuXmv configuration; timer properties are not unverifiable in principle, and the paper leaves open whether alternative encodings could bring scan-cycle-spanning timing constructs within conclusive model-checking coverage.
Conclusion
SemaPLC demonstrates that a verification-gated completion discipline — termination only on logged, externally corroborated evidence, with edit invalidation and earned claims — improves strict function-level correctness on every tested backbone and, more consequentially, that live runtime evaluation is the layer that actually discriminates among PLC code-generation methods. The finding that statically indistinguishable methods diverge by tens of dynamic points is a methodological result with force beyond the specific harness: benchmark design for industrial code generation must execute what it scores. The remaining open questions — generalization beyond benchmark scenarios, frontier-model convergence, and formal coverage of timing constructs — define the immediate agenda for this line of work.