LS-NFSP: Latent-Space Neural Fictitious Self-Play

• The paper introduces LS-NFSP, which constrains agent exploration to a compact latent skill manifold to balance physical feasibility with strategic diversity.
• It decouples low-level motor control from high-level tactical decisions using an encoder-decoder architecture optimized with KL and reconstruction losses.
• Empirical results in humanoid boxing benchmarks demonstrate LS-NFSP's superior stability, engagement, and performance compared to standard NFSP approaches.

Latent-Space Neural Fictitious Self-Play (LS-NFSP) is a multi-agent reinforcement learning (MARL) methodology that extends classic Neural Fictitious Self-Play (NFSP) by reparameterizing high-level agent interaction and strategic adaptation within a bounded, learned latent action manifold. LS-NFSP enables stable and high-performing self-play in high-dimensional, contact-rich domains by decoupling strategic exploration from direct actuation, notably in settings such as autonomous humanoid boxing, where physical feasibility and behavioral diversity must be simultaneously achieved (Yin et al., 30 Jan 2026).

1. Conceptual Basis and Motivation

NFSP, originally introduced by Heinrich and Silver, is a scalable, model-free approach for learning approximate Nash equilibria via a dual-policy system in extensive-form games, employing a mixture of best-response RL and average-policy imitation (Heinrich et al., 2016). While effective in discrete imperfect-information domains, standard NFSP suffers when transferred to continuous, high-dimensional motor spaces typical of robotics, due to the instability and non-stationarity of direct exploratory updates.

In LS-NFSP, the key refinement is to constrain agent exploration and policy optimization to a compact, pre-validated latent space of motor skills—specifically, a hyperspherical manifold distilled from human demonstration data. This restriction is motivated by two core challenges:

• Physical Feasibility vs. Strategy Diversity: Directly optimizing over 29-DoF actuators of a humanoid leads to catastrophic failures given the fragility of balance and contact; yet diverse, non-myopic exploration is critical for non-trivial competitive tactics.
• Strategy Evolution vs. System Stability: Self-play dynamics inherently introduce non-stationarity and adversarial co-adaptation, risking instability, especially when low-level policies are not robust to new behaviors generated during training.

By forcing agents to select actions as points on a unit hypersphere in latent space—where each point decodes to a physically plausible skill—LS-NFSP enables safe, rapid exploration with guaranteed physical validity. The theoretical underpinning relies on Glicksberg’s theorem, ensuring the existence of Nash equilibria in continuous, bounded action spaces (Yin et al., 30 Jan 2026).

2. Latent-Space Skill Manifold and Policy Parameterization

The LS-NFSP methodology constructs a latent skill manifold using a hierarchical representation:

• Encoder ($E_\phi$): Maps a privileged observation $o_t = (s_t^{prop}, s_t^{ref})$ to the mean and log-variance of a diagonal Gaussian in latent space $$\mathcal{N}(z \mid \mu_t^e, \operatorname{diag}(\sigma_t^e{}^2))$$.
• State-Conditioned Prior ($P_\xi$): Maps only the proprioceptive state $s_t^{prop}$ to a Gaussian prior $$\mathcal{N}(z \mid \mu_t^p, \operatorname{diag}(\sigma_t^p{}^2))$$.
• Decoder ($D_\psi$): Takes as input the normalized latent code $\hat z_t = z_t / \|z_t\|_2$ concatenated with $s_t^{prop}$, and outputs the 29-DoF motor targets for a PD controller.
• Latent Constraint: All sampled latent actions are normalized to reside on $S^{d-1}$ (the unit $d$-sphere), ensuring a compact and continuous strategy space for the RL agent.

Skill distillation is guided by the following losses:

• KL divergence between encoder and prior: $$L_{prior} = \mathbb{E}_{o_t}[\operatorname{D_{KL}}(E_\phi(z|o_t)\parallel P_\xi(z|s_t^{prop}))]$$
• Reconstruction loss: $$L_{rec} = \mathbb{E}_{o_t,a_t\sim\pi_{trk}} \| a_t - D_\psi(s_t^{prop}, z_t)\|^2$$
• Total loss: $L_{distill} = L_{rec} + \lambda_{prior} L_{prior}$, with $\lambda_{prior}\approx 10^{-3}$.

This construction is critical to filter low-level infeasibility from high-level tactical policy optimization (Yin et al., 30 Jan 2026).

3. LS-NFSP Policy System and Training Algorithm

Each agent in LS-NFSP maintains two neural policies:

• Best-Response Policy ($\pi_z^{RL,\theta^i}$): An on-policy actor (PPO-trained), inputting the agent’s proprioceptive and goal-oriented features, outputting a distribution on the latent sphere.
• Average-Policy Network ($\bar\pi_z^{\phi^i}$): Trained by supervised imitation on a reservoir buffer sampled from own past best responses, mapping the state to the mean latent action.

At each environment step, agent $i$ samples $z_t^i$ from a mixed policy:

$$z_t^i \sim \sigma = \eta \cdot \bar\pi_z^{\phi^i} + (1 - \eta) \cdot \pi_z^{RL,\theta^i}$$

where $\eta$ is the NFSP anticipatory mixing parameter (e.g., $\eta=0.1$).

The training loop follows:

1. For each timestep, select policy according to $\eta$.
2. If acting with the best-response policy, store the state-latent pair in the supervised buffer (reservoir) for policy averaging.
3. Execute latent actions, decode to motor targets, apply in physics simulation.
4. Store full transitions in the RL buffer.
5. Periodically update PPO using the RL buffer and update average policy via L2 loss on the reservoir.

This framework preserves the fictitious-play dynamics while bounding all strategy updates within the latent manifold (Yin et al., 30 Jan 2026).

4. Stability Properties and Theoretical Guarantees

LS-NFSP’s primary stability advance emerges from its construction of a bounded, continuous action space for MARL interaction:

• Compact Strategy Set: By normalizing latent actions to $S^{d-1}$, the system aligns with the prerequisites of Glicksberg’s generalization, ensuring the existence and accessibility of Nash equilibria.
• Mitigated Non-Stationarity: High-level policies cannot introduce physically catastrophic or previously unseen low-level commands, greatly reducing the effective non-stationarity observed by the environment and co-agents.
• Control-Tactic Decoupling: The decoder network absorbs the burden of contact stability and balance, freeing the high-level policy to consider only tactical diversity.

The theoretical insight is formalized: under regularity and boundedness, approximate best-response updates in this setting converge via fictitious play to an $\epsilon$-Nash equilibrium in the continuous zero-sum game limit.

5. Empirical Results and Comparative Analysis

Extensive simulation on the RoboStriker humanoid boxing benchmark demonstrates the efficacy of LS-NFSP relative to both standard self-play and alternative MARL protocols (Yin et al., 30 Jan 2026). Key findings include:

• Superior Engagement and Physical Metrics:

| Metric | 29-DoF SP | LS-NFSP (ours) | |-------------------------------|-------------|------------------| | Offensive Landing Rate $\eta_{hit}\uparrow$ | 0.142±0.05 | 0.685±0.03 | | Engagement Rate $ER\uparrow$ | 0.315±0.08 | 0.824±0.02 | | Base Orientation Stability $BOS\uparrow$ | 0.418±0.12 | 0.942±0.01 | | Torque Smoothness $TS_\tau\downarrow$ | 7.452±1.211| 0.930±0.150 |

• Cross-Play Strength: In win-rate matrices, LS-NFSP outperformed naive latent-space self-play and ablation variants, e.g., LS-NFSP wins 76.2% of box matches against naive SP, and 68.5% against standard fictitious SP. No-AMP or no-warmup variants further underperformed (win rate 82.4% for LS-NFSP over LS-NFSP w/o AMP).
• Tactical/Ablation Study:

| Method | $\eta_{hit}\uparrow$ | $ER\uparrow$ | |-------------------|---------------------|--------------| | PPO-Only | 0.231±0.03 | 0.495±0.02 | | Naive SP | 0.350±0.04 | 0.580±0.05 | | Fictitious SP | 0.420±0.03 | 0.650±0.04 | | LS-NFSP (ours) | 0.685±0.03 | 0.824±0.02 |

These results confirm that LS-NFSP produces more robust, tactically effective, and physically stable policies than approaches operating directly in the raw actuator domain or omitting motion priors.

6. Implementation Details and Practical Considerations

Key practical aspects for LS-NFSP in physical simulation and robotics include:

• Environment: NVIDIA Omniverse / Isaac Lab, 4,096 parallelized environments, simulation at 200 Hz, control loop at 50 Hz.
• Hardware: Unitree G1 humanoid (29 DoF).
• Network Architectures: Encoder, decoder, and prior as 3-layer MLPs (width 256); latent dimension $d=32$.
• Distillation Hyperparameters: $\lambda_{prior}=10^{-3}$.
• Training: PPO with learning rate $3 \times 10^{-4}$, clip $\epsilon=0.2$, 4 epochs, batch size 64.
• Reservoir/Replay: Buffer size $K=10^6$, mixing rate $\eta=0.1$.
• Domain Randomization: Friction randomized $[0.8,1.2]$, link mass and actuator gains ±10%.
• Early Termination: During distillation/warmup if base height drops below 0.25 m or tilt exceeds 0.8 rad.
• Warmup: Weighted rewards $w_{task}=0.8$, $w_{style}=0.2$, with AMP discriminator (gradient penalty $w_{gp}=10$).

The pipeline readily generalizes to other continuous-action multi-agent domains where control constraints and strategic learning must be jointly addressed (Yin et al., 30 Jan 2026).

7. Relation to Classic NFSP and Extensions

LS-NFSP builds directly on the NFSP framework proposed by Heinrich and Silver (Heinrich et al., 2016), preserving its core structure of RL-based best response, average-policy imitation, mixture policy execution, and the use of supervised reservoirs. The fundamental extension is the injection of a deep encoder-decoder module that maps from rich observations to a structured latent space, thereby permitting the application of fictitious self-play to domains previously inaccessible to standard NFSP due to high actuation dimensionality and fragile dynamics.

The methodology and architectures of LS-NFSP indicate its further extensibility via integration with more advanced representation learning, multi-agent credit assignment, and adaptive motion prior distillation. A plausible implication is the broad applicability of LS-NFSP across various multi-agent, contact-rich, continuous domains where policy stability and physical plausibility are paramount.

References:

• "RoboStriker: Hierarchical Decision-Making for Autonomous Humanoid Boxing" (Yin et al., 30 Jan 2026)
• "Deep Reinforcement Learning from Self-Play in Imperfect-Information Games" (Heinrich et al., 2016)