HILL Game: Human-in-the-Loop RL

• Human-in-the-Loop Learning (HILL) Game is a paradigm that combines human intervention with RL, enabling safety and improved efficiency through protocol programs.
• It leverages techniques like action pruning, reward shaping, and simulation switching to guide agents without accessing internal policy details.
• Empirical studies in environments such as Catcher and Taxi demonstrate faster learning rates, safer exploration, and enhanced convergence in RL applications.

A Human-in-the-Loop Learning (HILL) game is a paradigm in which human expertise is systematically incorporated into the reinforcement learning (RL) process to improve safety, efficiency, and adaptability of learning agents. Unlike conventional RL frameworks that treat the agent as fully autonomous—or require detailed assumptions about the agent's internals—HILL games introduce protocol-based mechanisms allowing humans to intervene by manipulating states, actions, and rewards, often in a modular and agent-agnostic fashion. This approach has significant implications for the design, generalization, and real-world deployment of RL agents, particularly in complex or safety-critical settings.

1. Agent-Agnostic Protocol Program Schema

The agent-agnostic HILL schema is centered on the concept of protocol programs acting as intermediaries between the agent, the environment, and the human teacher (Abel et al., 2017). Unlike prior methods that embed assumptions about the learner's structure (e.g., requiring access to internal Q-values or policy parameters), this schema operates by intercepting and potentially modifying the information flow at the agent-environment interface. The primary protocol program maintains a consistent input–output interface—receiving states and rewards, and outputting actions—while may internally execute various forms of human intervention or advice. This results in a black-box treatment of the learning agent, facilitating ease of deployment across different classes of RL algorithms and agent representations.

The modularity achieved through this agent-agnostic interface enables portability and reusability: the same human-guidance protocol can be applied to Q-learning, R-max, policy gradient agents, or others, without modification of their internal logic.

2. Integration of Human Teacher: Mechanisms and Objectives

Within protocol programs, the human teacher intervenes by manipulating state, action, or reward channels:

• Action Pruning: The human (via a function Δ(s, a)) can block actions deemed catastrophic in a given state. This guides safe exploration by preventing the agent from entering dangerous or irreversible states.
• Reward Shaping: Rewards returned to the agent can be altered by augmenting the environmental reward R(s, a) with a potential-based term $F(s, a, s') = \gamma\phi(s') - \phi(s)$, biasing learning towards preferred behaviors.
• State Manipulation: The agent's observations may be pre-processed, abstracted, or mapped onto alternative representations φ(s), facilitating transfer learning or accentuating task-relevant features.
• Training in Simulation: Protocols may initially redirect the agent's experience to a simulated environment $\mathcal{M}^*$ under human monitoring, allowing safe pre-training before moving to the real task environment.

These interventions serve key objectives in HILL games: enhancing safety (e.g., by action pruning), accelerating learning (e.g., by reward shaping), and reducing trial-and-error costs (e.g., by simulation-based bootstrapping). Notably, these mechanisms can be combined or interleaved, and do not depend on the specifics of the agent's learning algorithm.

3. Formalization and Algorithmic Instantiations

Protocol programs encapsulate human-guided interventions algorithmically. The paper details concrete pseudocode instances:

• Action Pruning:

def act(state): action = agent.act(state) while Delta(state, action): # Human can prune action and modify reward/state (state, reward) = human.prune(state, action) action = agent.act(state) return action

• Reward Shaping: Given an environmental transition (s, a, s'), the reward can be dynamically modified:

$r' = R(s, a) + F(s, a, s')$

where $F(s, a, s')$ is computed by human-designed shaping functions.

• Simulation Switching: Protocol mediates between simulated and real environments, switched by a human.

These examples formalize the information-theoretic role of protocol programs without embedding learning-specific details. Potential-based shaping formulas such as $F(s, a, s') = \gamma\phi(s') - \phi(s)$ (and its time-indexed extension) establish how reward transformations can be structured to guarantee convergence properties.

4. Unified Framework: Special-Case Representation

The protocol program schema is shown to subsume various classic human-in-the-loop techniques as special cases:

Mechanism	Protocol Program Manifestation	Human Role
Action Pruning	Δ(s, a) function in action selection loop	Approves/blocks potentially bad actions
Reward Shaping	F(s, a, s') function modifies reward channel	Designs shaping function F(·)
Training in Simulation	Switches environment from simulator $\mathcal{M}^*$ to real	Signals readiness for transition

By demonstrating that disparate prior methods (action pruning, reward shaping, simulation bootstrapping) are all instances of protocol programs, the schema provides a theoretical unification and opens the door to new hybrid strategies that blend multiple modes of human intervention.

5. Experimental Evidence in RL Domains

Validation experiments in the paper span both a Pong-like "Catcher" environment and the classical Taxi domain:

• Catcher: Agents using protocol-mediated action pruning (blocking actions that could induce catastrophic events) learned faster and exhibited higher early performance compared to unassisted RL agents.
• Taxi: Both Q-Learning and R-max, when augmented with action-pruning protocols (e.g., preventing drop-offs in illegal states), achieved greater cumulative rewards and faster convergence. Particularly, R-max benefited from reduced exploration of non-optimal state-action pairs.

These experiments empirically support the claims that protocol-based HILL frameworks improve not just safety but also learning efficiency, especially in sparse- or delayed-reward environments.

6. Implications and Future Developments for HILL Games

Key implications of the presented agent-agnostic HILL schema include:

• Modularity and Transferability: The absence of agent-specific assumptions means human-guidance mechanisms can be transferred across RL architectures with minimal engineering effort—a crucial property for maintainable and scalable HILL games.
• Safety and Sample Efficiency: Protocols such as action pruning enable safe real-world deployment (e.g., robotics, autonomous agents) by restricting high-risk actions during early learning phases.
• Centaur Systems: Hybrid systems can emerge where human intuition provides high-level constraints or guidance, while RL agents supply high-frequency policy adaptation, yielding superadditive performance.
• Generalization of Human Guidance: Framing action pruning, reward shaping, and simulation-based training as protocol programs suggests a compositional approach, where new forms of intervention (e.g., dynamic state abstraction, online curriculum design) can be prototyped and validated within the same schema.
• Directions for Research: The schema’s flexibility facilitates research into dynamic and personalized interventions (e.g., adjusting protocol strategies based on agent competency or user preferences), as well as its adaptation to multi-agent or cooperative game contexts.

7. Summary and Theoretical Significance

The agent-agnostic protocol program schema for HILL games establishes a modular, theoretically unified platform for integrating human expertise at the interface of RL agent-environment interaction (Abel et al., 2017). By decoupling intervention logic from agent internals, the framework supports safe, efficient, and widely applicable human-guided RL. The conceptual and practical contributions include formal algorithmic constructs for protocol-mediated learning, the unification of prior HILL methods as special cases, and substantiating empirical evidence for enhanced performance and risk mitigation in complex domains. This provides a foundational cornerstone for future research and development of flexible, safe, and effective HILL game systems.