Most PLC code generation work judges success by static metrics or compilation alone; execution on a live runtime is the ultimate test. This paper shows that an agentic harness that demands external, logged checks at every completion gate—specification, compile, and runtime behavior—finds and fixes many errors that static scoring misses, substantially improving verified behavior when code actually runs.
Key Findings
- Verification-gated loop: the system closes the generation loop with automated compilation, deployment to a live OpenPLC runtime, input forcing, and trace-based behavioral checks, and it iteratively repairs until external checks pass. This shifts completion criteria from model self‑assessment to mechanized evidence.
- Measured improvements: on 117 independent-POU tasks the harness attains a 72.6% mean strict verified pass rate across seven models. On a 65-task project-context track, dynamic behavior scores separate methods sharply (baselines 22.4–31.4 vs. 52.2 for the harness), showing execution-level checks reveal issues static metrics miss.
- Usable components: the work bundles a web IDE and a PLC tool suite (compiler/upload/run/read/force/trace tools plus an OpenPLC Docker runtime) so the agent can compile, deploy, force inputs, and trace variables end-to-end and present ladder diagrams and process simulations.
Who it's for and tradeoffs
Great fit if you need generated PLC logic to not only compile but demonstrably behave inside an existing project and runtime—automation engineers, researchers building code-generating agents, and teams integrating LLMs into industrial automation workflows. Look elsewhere if your goal is lightweight code sketching or craft-level ST snippets without live verification: the harness adds complexity (tooling, Docker runtime, scenario design) and focuses evaluation on execution rather than only on static or human-review metrics.
