- The paper presents a hybrid model combining fine-tuned LLMs with a grammar-driven parser to achieve over 48% Exact Match and 88% JSON similarity.
- The approach incorporates a self-correcting feedback loop that iteratively refines LLM outputs to enforce structural conformance through a validated AST.
- Experimental results demonstrate that integrating semantic inference with strict grammatical rules enhances safety-critical command interpretation in industrial human-robot collaboration.
Grammar-Constrained LLMs for Robust Robotic Command Understanding in Industrial Environments
Motivation and Problem Setting
Industrial human-robot collaboration critically depends on reliable interpretation of free-form natural language instructions into actions that are both safe and precisely executable. While recent LLMs offer advancements in open-domain language understanding, their probabilistic and unconstrained nature poses substantial challenges in safety-critical contexts such as smart manufacturing. In these domains, high structural rigor must coexist with flexible interpretation to support robust operation and adaptive instructions.
This work systematically investigates the tension between open-domain generalization of LLMs and the deterministic structural integrity required for robot command interpretation. It identifies the weaknesses of both unconstrained LLM outputs (which may fail to yield executable, valid commands) and purely grammar-driven NLU systems (which often lack semantic flexibility for complex or novel phrasings). The approach targets the gap by integrating both advances.
Methodology
The paper introduces a hybrid architecture that decomposes robot command interpretation into two modular processing stages: a fine-tuned LLM for high-level semantic inference, and a grammar-driven canonicalization/validation pipeline for structural enforcement.
LLM Semantic Front-End: The Meta-Llama-3-8B-Instruct model is selected, enabling both prompt-based control and parameter-efficient fine-tuning via LoRA. The system rigorously applies prompt engineering to enforce system/user role separation and standardizes hyperparameters influencing decoding determinism. Fine-tuning is performed using the HuRIC dataset, targeting robust extraction of action frames and elements expressed in industrial contexts.
Grammar-Based NLU Back-End: A bespoke grammar is implemented using the Lark parser framework (EBNF formalism), encoding allowable robot action frames and argument schemas. The grammar parser enforces that only actions and slots matching an ontological list can be generated, yielding a canonical AST and ultimate translation into a deterministic robot-executable JSON schema.
Hybrid Two-Layer Integration: The central contribution is the hybrid pipeline. Initial LLM outputs are passed to the grammar parser, which strictly validates the structure, element mapping, and ontology compliance. Any structural incongruities detected trigger an automatic feedback loop—regenerating corrective prompts for the LLM (informing it of rejection reasons with explicit error messages and expecting a reformulated response). The feedback loop iteratively enforces convergence to a valid, grammar-consistent output, leveraging the complementary strengths of both modules.
Experimental Protocol: All models (API-based LLM, fine-tuned LLM, grammar-based NLU, and the hybrid system) are evaluated on the HuRIC dataset, with metrics focusing on (1) JSON Exact Match (EM) against gold references, and (2) JSON Similarity (Jaccard index over key-value pairs) to quantify partial correctness. Failures and format errors are explicitly analyzed.
Results
API-Based LLM (No Fine-Tuning): Yields negligible executable output (EM=0%, JSON Similarity=0%), highlighting the inability of a base LLM to map diverse natural commands to precise robot targets without adaptation.
API-Based LLM with LoRA Fine-Tuning: Achieves moderate accuracy (EM=36.4%, JSON Similarity=72.3%), capturing a substantial portion of semantic roles but exhibiting frequent structural/formatting errors and failures (about 26% failed JSON, tendency for unconstrained generation).
Grammar-Based NLU: While effective at imposing validity, grammatical rigidity constrains semantic extraction (EM=21.2%, JSON Similarity=42.3%). Strong syntactic performance is offset by an inability to robustly map linguistically diverse user inputs to canonical robot frames, especially for out-of-vocabulary or structurally divergent instructions.
Hybrid Grammar-Based LLM: Demonstrates robust command understanding and execution readiness (EM=48.5%, JSON Similarity=88.1%), outperforming both standalone approaches in semantic coverage, exact formation, and overall consistency. The self-correcting feedback mechanism effectively reduces invalid/failed output rates (~24%), with empty outputs in strict rejection cases. Notably, median performance reaches 95% JSON Similarity, signifying reliable convergence on reference annotations.
Theoretical and Practical Implications
The paper provides empirical evidence that unconstrained LLMs—even when fine-tuned—are insufficient for safety-critical robotics contexts due to lack of structural determinism. Conversely, grammar-only models, though robust to structure, fail for linguistically varied instructions typical of naturalistic human-robot interaction.
By modularizing reasoning (LLM) and structure (grammar/validation) with an interposed feedback loop, the hybrid system achieves strongly complementary benefits: improved semantic recall, high EM and structural rigor, and superior robustness to instruction variation. The approach aligns with the need for task-specific symbolic scaffolding when deploying LLMs for robotics, highlighting that grammar-constrained decoding—implemented post hoc, rather than during token generation—is both practical and effective.
The methodology scales to new action ontologies or industrial task suites by updating grammar definitions and fine-tuning datasets. The feedback-driven corrective prompting paradigm could generalize to other structured-output tasks where LLMs are integrated with rigorous schema conformance requirements (e.g., code generation, data extraction).
Future Directions
The study identifies several critical limitations and next steps:
- Dataset Scope and Composition: The modest size and linguistic diversity (including interrogatives and polite requests) of HuRIC constrains generalization. Larger, domain-specific corpora of imperative, multi-action commands would facilitate training more capable hybrid systems.
- Grammar Engineering: Unified high-coverage grammars introduce structural ambiguity. Hierarchical grammar design or probabilistic parsing may provide more fine-grained control, enabling prioritization of likely frames/elements.
- Model Variance: The proposed framework is centered on Meta-Llama-3-8B-Instruct. Systematic benchmarking across foundation models (e.g., GPT-4, alternative LLMs with different decoding behaviors) is necessary to test robustness and attribute observed gains to architecture rather than model selection.
- Industrial Deployment: Custom datasets reflecting real industrial tasks and failure modes, combined with continual grammar evolution, would be required for production-readiness in smart manufacturing.
Conclusion
This work establishes that hybrid grammar-constrained LLM architectures—combining deep contextual reasoning with strict symbolic structure enforcement and a self-correcting feedback loop—are essential for robust robot command understanding in industrial HRC settings. The approach significantly outperforms both standalone LLMs and grammar-parsers, achieving strong structural consistency and high semantic fidelity. Its modular design and demonstrated efficacy offer a concrete foundation for deployment and extension to broader structured interpretation tasks in human-robot collaboration and beyond.
