RL-DiL-piKL: Regularized Imitation in Deep RL

• RL-DiL-piKL is a reinforcement learning framework that integrates human imitation via distributional D_KL regularization to balance reward maximization with conservative strategies.
• It employs a novel lambda sampling approach from a preset distribution during both RL training and search-time planning to model diverse agent behaviors.
• Empirical evaluations on games like Diplomacy and control tasks demonstrate enhanced performance, human compatibility, and formal convergence guarantees.

The RL-DiL-piKL framework is a family of reinforcement learning (RL) algorithms that incorporate policy regularization via a distribution over regularization strengths, with applications to multi-agent games and interpretable symbolic reasoning. This approach enables deep RL agents to blend reward maximization with the imitation of human strategies, offering both theoretical guarantees and empirical performance gains across domains such as no-press Diplomacy and relational RL with differentiable inductive logic programming. The framework unifies policy improvement via $D_\mathrm{KL}$-regularization (with behavioral cloning as anchor) and distributional regularization weights, and supports both RL training and search-time planning procedures (Bakhtin et al., 2022, Bueff et al., 2023).

1. Problem Formulation and Core Principles

RL-DiL-piKL addresses reinforcement learning in sequential multi-agent or single-agent decision problems under imperfect information and large action spaces. States $S$ are complex (e.g., Diplomacy board positions; continuous state vectors in control); actions $A_i$ can be extremely large (order tuples in Diplomacy, discrete control choices in RL benchmarks). The RL agent is trained to maximize expected reward, but must also coordinate with or emulate human behavior, especially in mixed-human-agent environments.

A key ingredient is access to a behavioral cloning (BC) policy $\tau_i(s) \in \Delta(A_i)$ trained from human demonstration data, providing a human-plausible anchor distribution. RL-DiL-piKL regularizes the agent's policy $\pi_i(s)$ toward $\tau_i(s)$ by incorporating a $D_\mathrm{KL}(\pi_i\,\|\,\tau_i)$ penalty, with the weighting parameter $\lambda$ chosen randomly per iteration from a finite population distribution $\beta_i$. This formalizes a population of agent types, interpolating from conservative (large $\lambda$) to reward-seeking (small $\lambda$).

2. The DiL-piKL Planning Algorithm

The DiL-piKL algorithm extends piKL-hedge by replacing a fixed regularization strength $\lambda$ with a sampled value from a distribution $\beta_i$ at each planning step. Given running payoffs $Q^{t-1}_i(a_i)$ (average action values from previous episodes, typically from a neural critic), the planning update at each iteration $t$ is: $$\pi^t_{i,\lambda}(a_i) \propto \exp\left( \frac{Q^{t-1}_i(a_i) + \lambda\,\log\tau_i(a_i)}{\eta_{t-1} + \lambda} \right)$$ where $\eta_{t-1}$ represents a decaying temperature parameter. The population belief $\beta_i$ is usually supported on values such as $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$.

Each search iteration samples $\lambda \sim \beta_i$, computes $\pi^t_{i,\lambda}$, samples $a_i^t\sim \pi^t_{i,\lambda}$, observes the joint action outcome, and updates $Q^t_i(a_i)$ accordingly. This approach models a spectrum of policies reflecting the diversity of human strategies.

3. RL-DiL-piKL Self-Play and Joint Training

For RL agent training, DiL-piKL is integrated into a multi-network deep RL loop analogous to the DORA/DNVI architectures. The agent maintains:

• A value network $V(s; \theta_v)$ estimating state value under equilibrium play.
• A policy proposal network $\pi_i(s,a_i;\theta_\pi)$ providing action likelihoods.

During data generation, candidate actions are sampled, and a 1-ply equilibrium is computed using the DiL-piKL procedure (over sampled joint action sets), yielding equilibrium policies $\sigma$ and value tuples $u$. Experience tuples $(s, \sigma, u)$ are stored in a buffer used to train both $V$ and $\pi$ via combined value and policy losses: $$\mathcal{L}(\theta_v, \theta_\pi) = \frac{1}{2} \left\| V(s;\theta_v) - \left[ r(s) + \gamma \mathbb{E}_{a'\sim\sigma} V(f(s,a');\hat\theta_v) \right] \right\|^2 - \sum_{i}\sum_{a_i} \sigma_i(a_i) \log\pi_i(s, a_i;\theta_\pi)$$ Human-imitation regularization is enforced at both training and inference by replacing regret-matching with DiL-piKL anchored on $\tau$. At inference, $\lambda$ sampling for opponents is retained, but the agent typically plays with a low fixed $\lambda$ to maximize reward while modeling opponents as more human-like.

4. Theoretical Guarantees

DiL-piKL carries formal no-regret and convergence guarantees under a regularized utility: $$\tilde u_{i,\lambda}(\pi_i, a_{-i}) = u_i(\pi_i, a_{-i}) - \lambda \, D_{\mathrm{KL}}(\pi_i\,\|\tau_i)$$ Over horizon $T$, the regret is logarithmic: $$\max_{\pi_i'}\sum_{t=1}^T\left[\tilde u_{i,\lambda}(\pi_i',a_{-i}^t) - \tilde u_{i,\lambda}(\pi^t_{i,\lambda},a_{-i}^t)\right] \leq O\left(\frac{\log T}{\lambda}\right) + O\left(\frac{1}{\eta}\right)$$ In two-player zero-sum one-step games, last-iterate convergence to the Bayes-Nash equilibrium of the regularized game occurs at rate $O(\log T / (\lambda T))$ (Bakhtin et al., 2022).

5. Agent Architectures and Implementation

In large-combinatorial multi-agent domains, RL-DiL-piKL is instantiated with scalable deep architectures:

• Diplomacy: Per-location (81+ coasts; 38 channels) and per-player features are embedded, concatenated, and processed by a 10-layer transformer encoder (width 224, 8 heads). Policy outputs are decoded with a small LSTM, while value heads use attention-pooling and MLPs to predict payoffs per player.
• Hyperparameters for self-play include Adam optimizer (IL: lr 2e-3, RL: lr 1e-4), replay buffer (1.28M), transformer dropout, and DiL-piKL $\beta$ uniform over the preset $\lambda$ grid.

Regularization toward human BC is enforced throughout training. At inference, $\lambda$ is low for agent moves, higher for modeling human or diverse opponents.

6. Empirical Evaluation and Results

Empirical validation was conducted on both combinatorial games and continuous RL benchmarks.

Diplomacy (No-Press)

• Population evaluation (1000 matches): Diplodocus agents trained with RL-DiL-piKL achieved average SoS scores of 29% (Diplodocus-Low) and 28% (Diplodocus-High), compared to 13–20% for previous DNVI/DORA variants and 6–18% for other baselines.
• Human tournament (200 games, 62 human participants): Diplodocus variants ranked first and third in Elo among all players with $\geq$5 games, both achieving average SoS scores above 25%.
• Expert judgment: Diplodocus was rated strongest and most human-compatible by Diplomacy experts.
• Gains were attributed predominantly to training under DiL-piKL; inference-time regularization alone yielded marginal improvements.

Relational RL (CartPole, LunarLander)

• Agents embedding differentiable Neural Logic (dNL) with nonlinear predicates (sin-transforms, etc.) were trained with RL-DiL-piKL (SAC-discrete backbone).
• On CartPole-v1, only SAC variants with dNL solved the task (reaching 300±0 return).
• On LunarLander-v2, dNLRL-SAC variants achieved mean episodic returns of 162±110 (continuous-predicate agent) and 133±114 (with nonlinear transforms).
• Convergence time was higher for dNL agents compared to MLP baselines (interpretability/speed trade-off).
• Interpretability: Learned FOL rules can be extracted directly from logic layer weights, such as:

  left() ← (CartPos < 2.83 ∧ CartVel > −1.19 ∧ CartVel > 0.18 ∧ PoleAngle > −0.62 ∧ PoleAngleVel > −0.24) [weight=0.56] ∨ (…)

  This demonstrates the framework’s capacity for extracting symbolic rules from trained policies (Bueff et al., 2023).

7. Significance and Practical Implications

RL-DiL-piKL unifies policy optimization, human imitation, and symbolic interpretability within a theoretically justified framework. The distributional regularization approach retains flexibility to model both conservative (human-like) and competitive (reward-seeking) behaviors. Empirical results show substantial improvements in both agent-human coordination and task reward, especially when the regularization is integrated during learning rather than only at test time.

A plausible implication is that RL-DiL-piKL provides a foundation for safe, interpretable reinforcement learning agents capable of effective coordination with humans in high-stakes, multi-agent environments. Its generality enables both the explicit extraction of symbolic policy logic and robust integration with deep learning architectures for large combinatorial or continuous-control tasks (Bakhtin et al., 2022, Bueff et al., 2023).