Linear Policy Net (LPN) in Deep RL

• Linear Policy Net (LPN) is a time-varying linear feedback architecture for deep reinforcement learning that employs an action Jacobian penalty to regularize policy smoothness and reduce computational overhead.
• It leverages an analytically available Jacobian from a compact MLP, enabling faster convergence and efficient implementation compared to standard fully connected architectures.
• Empirical evaluations on tasks like walking, backflips, and parkour show that LPN achieves superior control smoothness and robust real-world deployment, exemplified by applications on platforms such as Boston Dynamics Spot.

A Linear Policy Net (LPN) is a time-varying linear feedback architecture for deep reinforcement learning (RL), designed to generate smooth control policies for simulated and physical agents, especially in domains such as character animation and legged robotics. LPNs are distinguished by their ability to efficiently implement an action Jacobian norm penalty, serving as a global policy smoothness regularizer, while avoiding prohibitive computational costs endemic to generic fully connected architectures. Recent work has explored both the architectural details and learning-theoretic implications of linear or log-linear policy parametrization, highlighting their computational and convergence benefits (Xie et al., 20 Feb 2026, Alfano et al., 2022).

1. Motivation and Rationale

In RL-based motion imitation and control, policy networks often exploit high-frequency variations in the state–action mapping to maximize reward, producing jittery, unphysical signals that exceed actuator bandwidth and degrade performance on real hardware. Prior remedies include explicit action-difference penalties $\|a_t - a_{t-1}\|^2$ or local Lipschitz constraints (sampling-based direct penalization of $\|\partial \pi / \partial s\|$). These techniques require substantial per-task tuning and often present conflicts with the primary reward objectives.

The LPN framework introduces a principled alternative: penalizing the Frobenius norm of the state-action Jacobian $\|\partial \pi / \partial s\|_F^2$ directly. This quantity robustly captures the sensitivity of the policy to input perturbations, uniformly regularizing the policy’s smoothness across tasks without ad hoc hyperparameter adjustment. However, direct computation of the Jacobian penalty in standard multilayer architectures is computationally intensive—typically requiring an extra back-propagation sweep per sample (∼1.5× slowdown).

LPNs address this computational bottleneck by leveraging policy forms where the Jacobian is analytically available as an output of the architecture (Xie et al., 20 Feb 2026).

2. Mathematical Formulation

Consider a full state vector $s_t \in \mathbb{R}^n$ and an action vector $a_t \in \mathbb{R}^m$. The LPN parameterizes the policy at time $t$ as:

$$a_t = K_t(s_t, \hat s_t; \theta) s_t + k_t(s_t, \hat s_t; \theta) + \hat a_t$$

where $\hat s_t$ and $\hat a_t$ are reference motion state and action at $t$, and $K_t\in\mathbb{R}^{m\times n}$, $k_t\in\mathbb{R}^{m}$ are time-varying feedback gain and bias terms. In the instantiation of (Xie et al., 20 Feb 2026), a two-layer MLP with hidden width 256 and ReLU activations maps $\hat s_t \mapsto K_t, k_t$.

The RL training objective augments the standard cumulative reward with an action Jacobian penalty:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=1}^T r(s_t,a_t) - \lambda \|\partial \pi(s_t;\theta)/\partial s_t\|_F^2\right]$$

For PPO training, this objective yields a composite loss:

$L_{total} = L_{PPO} + w_{Jac} \cdot L_{Jac}$

with $L_{Jac} = \|K_t\|_F^2$, and a fixed weight $w_{Jac}=10$ suffices across tasks.

3. Efficient Implementation of the Action Jacobian Penalty

Autograd frameworks allow calculation of Jacobians for generic feedforward architectures, but this entails a full backward pass for each (state, action) pair. For LPNs, $K_t$ is produced by the MLP as an explicit output, and does not depend on $s_t$. Therefore,

$J(s_t) = \partial a_t/\partial s_t = K_t$

directly, and the penalty reduces to computing the Frobenius norm of $K_t$, which adds negligible overhead relative to the base PPO update.

For generic fully connected nets, per-sample Jacobian calculation via back-propagation imposes a typical training slowdown by a factor of 1.5. The LPN, in contrast, maintains training throughput nearly identical to standard PPO, enabling scalable penalty application with minimal compute cost.

4. Training Procedure and Computational Characteristics

Key training parameters include usage of PPO with 50 parallel MuJoCo environments at 120 Hz (policy at 30 Hz), 2,500 steps per iteration, and convergence typically achieved within 2,000 iterations (∼1 hour, 5 million samples on an RTX A6000 plus 12 CPU cores).

The high-level optimization loop follows:

• For each iteration, collect rollouts, at each $t$ compute $K_t,k_t$ from MLP, sample $a_t$ from $\mathcal{N}(K_t s_t + k_t + \hat a_t, \delta^2I)$, and step the simulator.
• Calculate PPO advantages and loss.
• Compute the Jacobian penalty as $L_{Jac} = \mathbb{E}_t[\|K_t\|_F^2]$.
• Compose total loss $L_{total}$ and update $\theta$ via Adam or SGD.

The LPN+Jacobian approach empirically converges in fewer iterations than either fully connected networks trained with action-change or Lipschitz constraints, while incurring lower or comparable computational cost per iteration.

5. Inference and Real-World Deployment

The LPN offers substantial inference-time efficiencies:

• The smoothing term’s weight $w_{Jac}$ is fixed; there is no need for per-task retuning over a broad class of motions (including static gaits and dynamic acrobatics).
• At deployment, only a compact 2-layer MLP need be evaluated at policy-rate (e.g., 30 Hz, or slower for some tasks), followed by matrix-vector application $K_t s_t$ at the fast inner loop (e.g., up to 500 Hz in PD-controlled actuators).
• On real hardware, all $K$ matrices for a typical reference cycle can be precomputed and replayed, enabling run-time execution without any on-board neural network inference.

LPNs have been successfully deployed on physical platforms such as the Boston Dynamics Spot with a custom arm, achieving complex coordination like pacing gaits combined with arm swings and table-tennis striking motions (Xie et al., 20 Feb 2026).

6. Empirical Evaluation and Comparisons

LPNs were assessed across a series of simulated motion imitation tasks: walking, running, backflips, sideflips, cartwheels, "table tennis footwork," and complex parkour maneuvers:

Task	Method	Action Smoothness ↓	High-Freq Ratio ↓	Motion Jerk ↓
Walking	LPN+Jac	0.0016	0.9	115.6
Walking	FF+Jac	0.0014	2.1	108.5
Walking	Lipschitz	0.0040	9.5	139.5
Walking	No Reg	0.0031	8.8	134.2
Backflip	LPN+Jac	0.061	8.7	140.8
Backflip	FF+Jac	0.042	4.0	109.6
Backflip	Lipschitz	0.195	33.5	170.8
Backflip	No Reg	0.148	26.6	168.5
Footwork	LPN+Jac	0.009	1.3	116.6
Footwork	FF+Jac	0.014	5.6	124.6
Footwork	Lipschitz	0.036	21.0	164.6
Footwork	No Reg	0.053	27.1	178.1

These results demonstrate that LPN+Jac achieves competitive or superior smoothness and frequency metrics compared to all baselines, while converging substantially faster and at lower computational cost. Although FF+Jac sometimes produces marginally better jerk scores on certain tasks, it requires twice as many iterations and higher per-iteration compute. Action-change and Lipschitz penalties offer limited improvement and may fail on dynamic maneuvers unless heavily tuned.

LPNs enabled robust real-hardware performance, including adaptive leg-arm coordination and high-frequency control under practical latency constraints (Xie et al., 20 Feb 2026).

7. Theoretical Context: Linear and Log-Linear Policy Parametrization

The LPN shares conceptual foundations with broader classes of linear and log-linear policy architectures. For instance, log-linear policies of the form:

$\pi_w(a|s) = \exp(w^\top \phi(s,a)) / Z_w(s)$

enable tractable updates and guarantee global linear convergence of natural policy gradient (NPG) algorithms under appropriate feature coverage and error conditions (Alfano et al., 2022). Specifically, if the $Q$-function can be approximated by a linear combination of features with bias $\delta$ and the feature covariance is well-conditioned ($\kappa<\infty$), then a geometrically increasing step size yields convergence rate $O((1 - 1/\nu_\mu)^t)$ up to error terms determined by $\delta$ and sampling accuracy:

$$\Delta_T \leq (1 - 1/\nu_\mu)^T \cdot \frac{2}{1-\gamma} + 2\nu_\mu \sqrt{\frac{2|A|\delta}{1-\gamma}}$$

where $\nu_\mu$ quantifies distribution mismatch (Alfano et al., 2022).

A plausible implication is that practical LPN design benefits from well-chosen feature maps and stable dynamics, exploiting both computational and statistical efficiency made possible by the linear parametrization.

---

References:

• "Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty" (Xie et al., 20 Feb 2026)
• "Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrization" (Alfano et al., 2022)