LLM-Human-in-the-Loop Pipeline

• LLM-HITL pipelines are integrated systems where LLM agents perform simulation and control tasks while human feedback at key checkpoints guides decision-making.
• They enable adaptive policy tuning and closed-loop optimization, demonstrated in applications such as digital twin control and multi-robot coordination.
• Key challenges include managing non-determinism and ensuring data fidelity through advanced prompt engineering and structured human intervention.

A LLM–Human-in-the-Loop (LLM-HITL) pipeline refers to any computational architecture or workflow that systematically embeds human decision, judgment, or feedback at well-defined stages within an otherwise automated, LLM-centric process. This paradigm enables new forms of closed-loop optimization, interactive supervision, and adaptive policy learning by leveraging LLMs as simulation, reasoning, or control agents while preserving critical points of human intervention to address ambiguity, ensure safety, or maximize utility in complex environments.

1. Foundational Architectures and Representative Systems

Several instantiations of the LLM-HITL paradigm have been operationalized across both cyber-physical and software systems.

• Digital Twin + RL Control: A prime example is the digital twin pipeline for human-in-the-loop building control, where LLM agents (e.g., GPT-4 as static map initializer, GPT-3.5 as population simulator) serve as a population model and preference synthesizer for a physical environment (e.g., a shopping mall). The system incorporates LLM-generated group profiles, real-time synthetic feedback on human comfort, and feeds these signals into an offline-trained DQN RL agent for HVAC setpoint optimization (Yang et al., 2024).
• Human-in-the-Loop Empirical Research: HLER decomposes empirical economic research into multi-agent LLM orchestration modules for data auditing, profiling, hypothesis generation, econometric analysis, and manuscript drafting, but explicitly embeds lightweight human decision gates for hypothesis selection and final publication approval—structuring the process as dual iterative loops (Zhu et al., 8 Mar 2026).
• Multi-Robot Coordination: HMCF integrates human supervision for safety and reallocation decisions into a multi-agent LLM-controlled robotic system, with human-in-the-loop triggers for feasibility or exception handling (Li et al., 1 May 2025).
• Software Development Pipelines: The HULA framework places human review at key handoff points (plan acceptance and code review) in a multi-agent LLM coding workflow, facilitating direct corrective feedback and user guidance (Takerngsaksiri et al., 2024).

A generic LLM-HITL pipeline thus typically comprises: (1) one or more LLM agents for generative/simulation tasks, (2) an explicit mechanism for incorporating human feedback, (3) a central controller or orchestrator, and (4) a closed data/control loop.

2. Pipeline Topology and Data/Control Flow

A canonical LLM-HITL pipeline operates as a staged composition:

1. Environment/Task Initialization: Static context, environmental parameters, or task-specific data are encoded—potentially by LLMs (e.g., generating a floor plan or robot spec from prompts).
2. State Simulation/Reasoning: At runtime, LLMs generate or update critical state variables (e.g., population distributions, thermal comfort votes, robot plans, research hypotheses) on a defined schedule, typically at discrete time steps.
3. Human Interaction Checkpoints: At pre-specified stages (e.g., before policy deployment or manuscript submission), humans approve, edit, or reject LLM outputs, inject new constraints, or select among candidate proposals.
4. Adaptive Policy Selection/Update: The pipeline either retrains or updates downstream policies based on joint human-LLM signals, as in RL training (e.g., DQN for setpoint adjustment) or plan resynthesis (e.g., via LLM commonsense reasoning incorporating human instructions).
5. Deployment/Actuation and Logging: The chosen actions or outputs are applied to a physical or digital environment; states, decisions, and feedback are logged for traceability and future learning.

For example, the Agent-in-the-Loop RL (AitL-RL) implementation (Yang et al., 2024) follows a recurrent loop per time step:

• Query LLM twin for current state $s_t$
• RL agent selects action $a_t$
• Apply $a_t$ to simulated environment
• Compute reward $R_t$ and next state $s_{t+1}$
• Store experience tuple $(s_t, a_t, R_t, s_{t+1})$ and periodically update RL policy via DQN
• Human intervention can occur in tuning reward weights or updating group profiles

3. Prompt Engineering and Human Feedback Integration

Explicit, carefully engineered prompt templates play a pivotal role in LLM-HITL systems:

• Zero-stochasticity in Digital Twins: All LLM calls are made at temperature $=0$ to reduce non-determinism (Yang et al., 2024).
• Structured, Multi-stage Prompts: Multiple prompt types orchestrate the flow (e.g., Category/Population/Distribution prompts for simulating group behavior in malls).
• Human-in-the-Loop Editing: Where LLM outputs are ambiguous or suboptimal, human agents directly edit planning artifacts (e.g., code diff, plan nodes), issue re-generation requests, or supply corrective constraints (e.g., in robot plan refinement (Merlo et al., 28 Jul 2025)).
• Adaptive Policy Tuning: Human feedback not only corrects immediate errors but can steer system evolution—refining reward weights, RL policy architectures, or input parameters.

For example, in the robot replanning pipeline, users iteratively inspect an LLM-generated semantic plan, issue natural-language corrections, and accept or roll back plan updates in a loop until an executable, robust plan emerges (Merlo et al., 28 Jul 2025).

4. Formal Algorithmic and Reward Structures

LLM-HITL pipelines are typically formalized via a combination of:

• RL State, Action, and Reward Definitions: States incorporate both current environment status and explicit synthetic human feedback (e.g., $$s_t = [v_t^↑, v_t^↓, v_t^↔; t; T_{out}(t); T_{in}(t); O_t]$$ in AitL-RL), and the action space may be scalar (centralized) or vectorized (distributed) setpoints.
• Reward Functions Balancing Multicriteria: A weighted sum reward, $R_t = w_c \cdot U_t - w_e \cdot E_t$, proportionally incorporates user comfort and energy usage, parametrized by human-tuned weights.
• Policy Update and Experience Replay: Q-learning and DQN losses formalize the RL agent’s updates; batch sampling and soft target updates regulate learning dynamics.
• Cost/Risk Minimization in Plan Generation: For robotics, explicit cost functions $$C(P) = w_{coll} C_{coll}(P) + w_{contact} C_{contact}(P) + w_{time} C_{time}(P)$$ are minimized by the LLM under human-imposed constraints (Merlo et al., 28 Jul 2025).

These structures formalize the “in-the-loop” integration, ensuring human signals and preferences are not merely post-hoc corrections but active factors in optimization.

5. Evaluation Metrics and Empirical Results

LLM-HITL pipelines are empirically assessed via multifaceted metrics, reflecting both operational and human-centered objectives:

Metric	Description	Paper/Context
Comfort score	$\sum_t U_t$ over daily episodes	(Yang et al., 2024)
Energy score	Negated total energy cost ($-\sum_t E_t$)	(Yang et al., 2024)
Total score	Weighted reward sum, i.e., $w_c \sum U_t - w_e \sum E_t$	(Yang et al., 2024)
Convergence speed	Number of episodes to stable reward	(Yang et al., 2024)
Success Rate (robotics)	$\#$(tasks completed)/$\#$(tasks attempted)	(Li et al., 1 May 2025)
Average steps (robotics)	Mean steps to complete task	(Li et al., 1 May 2025)
Iteration-to-success (HITL replanning)	Iterations to satisfactory plan	(Merlo et al., 28 Jul 2025)
Usability (SUS)	System Usability Scale by users	(Merlo et al., 28 Jul 2025)

In empirical studies:

• AitL-RL (balanced) produced $15$–$25\%$ higher total score than set-point control
• Distributed RL policies (per-store setpoints) delivered $8$–$12\%$ higher scores than a single centralized setpoint (Yang et al., 2024)
• Human-in-the-loop robot plan refinement achieved $100\%$ success on benchmark tasks with an average of $2.4$ iterations per user; usability scored $83/100$ (Merlo et al., 28 Jul 2025)

6. Observed Limitations and Practical Considerations

Critical limitations and design nuances emerge:

• Compute/Deployment: LLM inference and lightweight DQN training are practical for edge/cloud hybrid deployment, but not real-time LLM training.
• Data Fidelity: Fully synthetic human data may not capture all edge cases; future iterations must consider real-world human feedback or retrieval-augmented prompt strategies.
• Generalizability: The described pipelines are largely domain-agnostic—CPS, traffic control, and resource planning all admit similar loops with different simulators and prompts.
• Residual Non-determinism: Even at temperature $=0$, LLM outputs can vary slightly; this stochasticity is argued to mirror genuine human unpredictability (Yang et al., 2024).
• Online Adaptation: While offline LLM-generated data supports training, real deployments will require live re-calibration as occupant preferences drift or context changes (Yang et al., 2024).

7. Broader Impact and Prospects

LLM-HITL pipelines represent an overview of foundation model simulation fidelity and targeted, scalable human feedback. Key advantages include:

• Personalization: Allows adaptive, group- or location-specific control without requiring impractical data collection from real users.
• Closed-Loop Optimization: RL agents leverage simulated and synthetic “votes” for efficient policy learning, which would be infeasible with purely observational human data.
• System Robustness: By embedding human feedback at the loop’s most impactful decision points, the system guards against LLM hallucinations and reward hacking.

In summary, LLM-HITL pipelines establish a template for distributed, interactive control and optimization in complex, human-centered environments—balancing the strengths of foundation models with critical, targeted human insight (Yang et al., 2024, Merlo et al., 28 Jul 2025, Li et al., 1 May 2025, Takerngsaksiri et al., 2024, Zhu et al., 8 Mar 2026).