Reinforcement Learning with Physical Feedback

• RLPF is a reinforcement learning paradigm that integrates physically grounded signals—such as dynamics residuals, tactile feedback, and EEG responses—into the training process for improved safety and dynamical consistency.
• It employs diverse optimization routes including reward shaping, actor-loss regularization, and controller adjustments to incorporate explicit physics, ensuring actions are executable and energy-efficient.
• Empirical studies across domains, from humanoid motion to molecular generation, demonstrate significant gains in convergence speed, trajectory tracking, and system robustness.

Reinforcement Learning with Physical Feedback (RLPF) denotes a class of reinforcement-learning and RL-style post-training methods in which the optimization signal is derived from physically grounded evidence rather than, or in addition to, purely task-level rewards. In the current literature, “physical feedback” spans several distinct sources: analytic dynamics residuals, simulator-evaluated feasibility, tactile deviations, force-field evaluations, traffic-model advice, and physiological signals such as EEG error-related potentials. The shared objective is to bias learning toward policies or generators whose outputs are executable, dynamically consistent, energetically stable, physically plausible, or safety-assured under real or simulated embodiment (Yue et al., 15 Jun 2025, Chandra et al., 15 Mar 2026, Huang et al., 2024, Zhou et al., 22 Aug 2025). A terminological caveat is necessary: the acronym RLPF is also used in Meta’s advertising work to mean “reinforcement learning with performance feedback,” where “performance feedback” refers to aggregate business outcomes rather than physical or haptic feedback (Jiang et al., 29 Jul 2025).

1. Conceptual scope and definitional boundaries

Across the cited works, RLPF is not a single algorithmic template but a family of mechanisms for injecting physics-relevant information into learning. In humanoid motion generation, the feedback is a physics-aware motion evaluation obtained by retargeting generated trajectories, executing them in Isaac Gym or MuJoCo with a tracking policy, and converting success or failure into a reward for RL fine-tuning of a text-conditioned motion generator (Yue et al., 15 Jun 2025). In PIPER, the feedback is a differentiable residual derived from analytic inverse dynamics, computed from terms such as the mass matrix, bias forces, and external/contact forces, and inserted directly into the actor objective as a soft regularizer (Chandra et al., 15 Mar 2026). In contact-rich manipulation, Residual Feedback Learning modifies the feedback channels of a black-box Cartesian impedance controller, allowing the RL policy to act through virtual joint positions or end-effector pose rather than only through output torques (Ranjbar et al., 2021).

The scope broadens further outside classical robot control. PE-RLHF for autonomous driving treats actions from IDM and MOBIL as a continuously available physics-based policy, then arbitrates between human and physics actions through a value-based gate during intervention (Huang et al., 2024). PhysMaster uses pairwise human judgments of physical plausibility to optimize a physical representation for video generation through DPO, while molecular RLPF applies PPO-style fine-tuning to equivariant diffusion models with rewards computed from DFT or xTB forces (Ji et al., 15 Oct 2025, Zhou et al., 22 Aug 2025). EEG-based work extends the idea to physiological feedback by decoding ErrP signals and folding them into reward shaping for Bayesian DQN (Xu et al., 2020).

This breadth implies that “physical” in RLPF should not be restricted to haptics or rigid-body contact. In the cited literature it includes explicit mechanics, physically informed mentors, simulator-grounded feasibility, force-field evaluations, and physiological reactions to observed errors (Chandra et al., 15 Mar 2026, Huang et al., 2024, Zhou et al., 22 Aug 2025, Xu et al., 2020).

2. Feedback mechanisms and optimization routes

PIPER explicitly frames a generic RLPF paradigm in which physical feedback can be injected by reward shaping, hard constraints, or regularization. Its own contribution adopts the regularization route: the reward function is not altered, no hard safety layer is imposed, and instead the actor loss is augmented by a physically meaningful residual norm (Chandra et al., 15 Mar 2026). The broader literature instantiates all three ideas, plus a closely related route in which physical feedback changes action selection or controller inputs.

Instantiation	Feedback source	Optimization mechanism
Humanoid large motion models	Tracking success and alignment verification	GRPO sequence-level RL
PIPER	Analytic inverse-dynamics residual	Actor-loss regularization
Residual Feedback Learning / HRRL	Virtual controller feedback and residual actions	PPO over feedback/action residuals
Tactile reactive planning	Tactile deviation from expected traces	PMNN initialization plus PI$^2$-CMA
PE-RLHF driving	IDM+MOBIL physics actions and human intervention	Reward-free proxy value learning with gated execution
Molecular diffusion RLPF	DFT or xTB force evaluations	PPO-style DDPO fine-tuning
PhysMaster	Human pairwise preferences over physical plausibility	Flow-DPO on DiT and PhysEncoder

In reward-based variants, physics enters as a scalar or sequence-level signal. The humanoid framework uses a success indicator reward for tracking and penalizes text-motion and motion-motion embedding distances through an alignment verification module (Yue et al., 15 Jun 2025). Molecular RLPF defines the primary reward as the negative root-mean-square force of the generated molecule under a chosen evaluator, with failed evaluations receiving a fixed penalty (Zhou et al., 22 Aug 2025). In EEG-based RLPF, the reward is shaped by decoded ErrP events, either at every step or through an auxiliary reward learned from pre-labeled trajectories (Xu et al., 2020).

In regularization-based variants, the physical signal acts as an auxiliary loss. PIPER computes the discrepancy between the policy’s action and the action implied by inverse dynamics, using a PINN acceleration proxy to preserve differentiability; the residual does not replace task reward, but biases policy updates toward dynamically consistent actions (Chandra et al., 15 Mar 2026).

In controller-mediated variants, physical feedback affects what the low-level controller sees or what fallback action is executed. Residual Feedback Learning alters controller observations such as joint positions or end-effector pose, while PE-RLHF selects between human and physics actions during takeover, with the hybrid intervention then used to train a reward-free actor-critic system (Ranjbar et al., 2021, Huang et al., 2024).

3. Representative mathematical formulations

A central regularization-based formulation appears in PIPER. For generalized coordinates $q$, velocities $\dot{q}$, and actions interpreted as torques, inverse dynamics is written as

$M(q)\ddot{q} + b(q,\dot{q}) = \tau,$

with $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$. PIPER introduces a differentiable acceleration proxy $\Phi_\phi(s,a)$ and defines

$$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$

leading to

$$L_{phys} = \mathbb{E}\big[\|r(s,\pi_\theta(s))\|_2^2\big],$$

and an augmented actor objective

$$J_{actor}(\theta) = J_{RL}(\theta) - \lambda_{phys}\,\mathbb{E}\big[\|r(s,a)\|_2^2\big].$$

The critic and advantage estimation remain unchanged (Chandra et al., 15 Mar 2026).

The humanoid motion framework uses RL at the sequence level. For each instruction, candidate token sequences are decoded to motions, retargeted to robot morphology, and evaluated by a student tracking policy in simulation. The sequence reward is

$$r_i = w_{track}\cdot R_{tracking}^{m_i} - w_{align}\cdot \big(\alpha R_{TA}^{m_i} + (1-\alpha)R_{MA}^{m_i}\big),$$

with $q$0 and $q$1. The motion generator is then updated with a GRPO objective that uses clipped likelihood ratios, group-normalized advantages, and KL regularization to a reference policy (Yue et al., 15 Jun 2025).

The molecular diffusion formulation casts reverse diffusion as a finite-horizon MDP with state $q$2 and action $q$3. The terminal reward is computed only at $q$4, and the primary reward is

$q$5

where

$q$6

Fine-tuning uses the clipped PPO surrogate

$q$7

with normalized terminal rewards as advantages and $q$8 in the reported experiments (Zhou et al., 22 Aug 2025).

PE-RLHF uses a different mathematical route. During takeover, human and physics actions are compared through an ensemble proxy value estimator, and the higher-valued action is executed:

$q$9

The policy is then optimized without environmental rewards through a proxy-value objective and an intervention-value penalty (Huang et al., 2024).

These formulations show that RLPF does not require a uniform notion of reward. Depending on the domain, the physically grounded signal may appear as a residual norm, a terminal force-based scalar, a simulator-derived success indicator, or a value-based arbitration mechanism.

4. Robotic control and embodied execution

The most direct RLPF instantiations occur in embodied control. The humanoid framework in “RL from Physical Feedback: Aligning Large Motion Models with Humanoid Control” combines an LLaMA2-7B text-conditioned motion generator, a motion tokenizer trained on MotionX, a teacher tracking policy trained with PPO using privileged simulator information, and a student policy distilled by a DAgger-like scheme to remove privileged inputs (Yue et al., 15 Jun 2025). Generated motions are retargeted through SMPL shape and pose optimization and then executed in Isaac Gym or MuJoCo. Failure is defined when positional deviation exceeds $\dot{q}$0m at any timestep or the body loses balance by a pitch-angle threshold. The alignment verification module uses pretrained contrastive encoders to compute text-to-motion and motion-to-motion distances, preventing the generator from collapsing to trivial but trackable behaviors.

PIPER addresses a different embodied-control problem: actor-critic methods often rediscover physics from data alone, despite accurate dynamics models being available in simulators. Its Automated Dynamics Oracle extracts $\dot{q}$1 with CRBA, obtains $\dot{q}$2 via RNEA at zero acceleration, and queries $\dot{q}$3 from MuJoCo contact sensors (Chandra et al., 15 Mar 2026). The method is explicitly algorithm-agnostic across PPO, SAC, TD3, and TQC, requires no simulator modifications, and introduces no inference-time optimization. The essential idea is not to learn a separate world model, but to use analytic inverse dynamics as a differentiable soft constraint during actor updates.

Residual Feedback Learning and HRRL focus on the structure of closed-loop manipulation systems. Standard residual policy learning adds a perturbation to a controller’s output, but a high-frequency impedance controller interprets that perturbation as an error and actively counteracts it. RFL instead changes the physical feedback entering the controller. In joint-space form,

$\dot{q}$4

while the end-effector variant computes a pose residual through $\dot{q}$5 and $\dot{q}$6. HRRL combines output residuals and feedback residuals in a 14-dimensional action for a 7-DOF arm (Ranjbar et al., 2021). The paper treats the Cartesian impedance controller as a black box running at 1 kHz, with RL inference at 40 Hz, and reports that modifying the feedback channel is particularly effective for large spatial corrections under uncertainty.

The tactile-feedback framework in “Supervised Learning and Reinforcement Learning of Feedback Models for Reactive Behaviors” takes yet another route. The task is to scrape a board while maintaining flat contact under unknown board roll angle, using two BioTac tactile sensors with 38-dimensional electrode pressure signals and a 6-DoF force–torque sensor (Sutanto et al., 2020). Nominal behaviors are encoded with Quaternion DMPs, expected tactile traces are stored as Associative Skill Memories, and a Phase-Modulated Neural Network maps tactile deviation $\dot{q}$7 to a coupling term $\dot{q}$8. RL then fine-tunes a low-dimensional DMP-parameterized policy with PI$\dot{q}$9-CMA on hardware, while preserving DMP convergence properties through phase modulation.

5. Generative modeling, safe driving, and physiological feedback

RLPF has also become a post-training paradigm for generative systems whose outputs must satisfy extra-linguistic physical criteria. PhysMaster introduces a PhysEncoder, built on a DINOv2 backbone initialized from Depth Anything, to extract physical priors from a single initial frame and concatenate them with image conditioning for a DiT-based image-to-video model (Ji et al., 15 Oct 2025). The paper does not define an explicit physics reward function. Instead, physical feedback is supplied through human pairwise preferences over generated videos, and Flow-DPO is used first to align the DiT and then, crucially, to optimize only the physical head of PhysEncoder while the generator is frozen. This makes the representation itself the object of RL-style alignment.

Molecular generation provides a more explicit physics oracle. “Guiding Diffusion Models with Reinforcement Learning for Stable Molecule Generation” fine-tunes EGNN-based equivariant diffusion models with terminal rewards computed from B3LYP/6-31G(2df,p) DFT forces on QM9 or GFN2-xTB on GEOM-drug (Zhou et al., 22 Aug 2025). The reward targets energetic stability rather than likelihood alone, and the work emphasizes size-invariant masked log-probabilities so that PPO ratios remain stable across variable-size molecular graphs.

PE-RLHF extends the idea of physical feedback to safety-critical control with human oversight. The physics-based policy $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$0 is constructed from IDM for longitudinal control and MOBIL for lane changes, producing interpretable actions that serve as a safety-assured fallback (Huang et al., 2024). Training is reward-free: instead of hand-crafted environmental rewards, the system learns a proxy value function that assigns higher value to hybrid intervention actions and lower value to conflicting autonomous actions, while a cosine-similarity-based intervention cost discourages frequent takeover.

The EEG-based work shows that physical feedback may also be physiological. Here the signal is the human observer’s intrinsic ErrP response, recorded with a BIOPAC CAP-100C and OpenBCI Cyton, preprocessed with a Riemannian geometry pipeline, and decoded to shape the reward of a Bayesian DQN agent (Xu et al., 2020). The paper proposes both a full-access shaping baseline and a more robust two-stage method in which ErrP-labeled trajectories are used to learn an auxiliary reward through a soft-Q human model and a dynamics-consistency baseline.

Taken together, these studies broaden the semantics of RLPF beyond direct robot-environment contact. The feedback may come from force fields, traffic laws, simulated feasibility, or neural signatures of human error detection, provided that it encodes physically relevant structure and enters the optimization loop in a principled manner.

6. Empirical record across domains

The reported empirical record is heterogeneous in metrics but unusually consistent in direction: physically grounded feedback improves execution feasibility, stability, sample efficiency, or safety relative to baselines that optimize semantics, appearance, imitation, or task reward alone.

Domain	Reported result	Paper
Humanoid motion generation	RLPF-MA reaches Succ $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$1 on CMU IsaacGym/MuJoCo versus Base Model $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$2; RLPF-TA reaches Succ $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$3	(Yue et al., 15 Jun 2025)
Physics-informed robot RL	PIPER yields $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$4–$M(q)\ddot{q} + b(q,\dot{q}) = \tau,$5 faster convergence, precision gains up to $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$6, and training overhead $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$7–$M(q)\ddot{q} + b(q,\dot{q}) = \tau,$8	(Chandra et al., 15 Mar 2026)
Peg insertion under uncertainty	HRRL achieves success rate $M(q)\ddot{q} + b(q,\dot{q}) = \tau,$9 over $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$0 hardware trials at orientation uncertainty $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$1 rad	(Ranjbar et al., 2021)
Safe autonomous driving	Success rate improves from $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$2 to $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$3 and episodic return from $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$4 to $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$5 against IDM+MOBIL	(Huang et al., 2024)
EEG-based implicit feedback	Training accelerates by about $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$6 and query count falls by a $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$7 average on Maze	(Xu et al., 2020)
Tactile reactive planning	PI$$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$8-CMA converges within $$b(q,\dot{q}) \coloneqq C(q,\dot{q})\dot{q} + g(q) - \tau_{ext}$$9 iterations for each primitive; RL improves the initially unseen $\Phi_\phi(s,a)$0 setting without degrading seen settings	(Sutanto et al., 2020)
Molecular diffusion	On QM9, molecule stability rises from $\Phi_\phi(s,a)$1 to $\Phi_\phi(s,a)$2 and validity to $\Phi_\phi(s,a)$3; on GEOM-drug, atom stability rises from $\Phi_\phi(s,a)$4 to $\Phi_\phi(s,a)$5 and validity from $\Phi_\phi(s,a)$6 to $\Phi_\phi(s,a)$7	(Zhou et al., 22 Aug 2025)
Physics-aware video generation	PhysMaster final improves SA from $\Phi_\phi(s,a)$8 to $\Phi_\phi(s,a)$9 and PC from $$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$0 to $$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$1, with inference time $$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$2s per $$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$3s video	(Ji et al., 15 Oct 2025)

Several papers also provide mechanistic ablations rather than only end metrics. In the humanoid setting, removing the alignment verification causes the generator to degenerate to standing motions despite strong low-level tracking, while removing the tracking reward collapses success rates (Yue et al., 15 Jun 2025). In manipulation, increasing “buffer steps” worsens standard residual policy learning because the controller gets more time to cancel residual actions; feedback-side intervention avoids that failure mode (Ranjbar et al., 2021). In driving, removing the physics policy degrades episodic return, success, and safety, while increasing the gating threshold causes the system to rely more heavily on physics advice when mentor quality deteriorates (Huang et al., 2024). In molecular generation, continued supervised training on rejection-sampled outputs does not match the stability and validity gains of PPO with force-based rewards (Zhou et al., 22 Aug 2025).

7. Limitations, failure modes, and unresolved issues

Despite these results, the literature identifies several recurring constraints. Oracle fidelity is a central one. PIPER assumes accurate mass, inertia, gravity, and contact-force information; under sim-to-real mismatch, the physics weight $$\hat{\tau}_{phy}(s,a) = M(q)\Phi_\phi(s,a) + b(q,\dot{q}), \qquad r(s,a) = \hat{\tau}_{phy}(s,a) - a,$$4 must be tuned lower, and abrupt switching in hybrid dynamics remains challenging (Chandra et al., 15 Mar 2026). Molecular RLPF depends on expensive or approximate evaluators: DFT is accurate but costly, while xTB is cheaper but less faithful; terminal-only rewards also leave credit assignment sparse (Zhou et al., 22 Aug 2025). PE-RLHF’s performance floor relies on correct value-based arbitration, and the guarantee is stated in expectation rather than as a worst-case bound under arbitrary misestimation (Huang et al., 2024).

A second issue is objective balance. In the humanoid paper, tracking-only optimization produces trivial, highly trackable motions, so semantic alignment must be enforced explicitly through contrastive verification (Yue et al., 15 Jun 2025). PhysMaster reports a related dependence on feedback quality: human preference labels are costly, current AI evaluators have insufficient physics knowledge, and subtle conservation violations may be underrepresented because the method optimizes what annotators notice rather than an explicit mechanistic law (Ji et al., 15 Oct 2025).

A third issue is embodiment and generalization. The tactile-feedback scraping system adapts only roll orientation and targets low-frequency tactile behaviors, leaving full 6D control and richer contact regimes to future work (Sutanto et al., 2020). Residual Feedback Learning is demonstrated on a peg-in-hole task with a specific black-box impedance controller; extending the framework requires identifying stable and accessible feedback channels for other controllers and tasks (Ranjbar et al., 2021). EEG-based RLPF remains limited by noise, inter-subject variability, and the current reliance on discrete environments and pre-generated trajectories for label efficiency (Xu et al., 2020).

Finally, the acronym itself is ambiguous. In Meta’s ad-generation deployment, RLPF denotes “reinforcement learning with performance feedback,” not physical feedback, and the reward is historical click-through rate rather than any physical signal (Jiang et al., 29 Jul 2025). That collision has little methodological overlap with the physical-feedback literature beyond the general idea of post-training from a learned reward model. For the physically grounded line of work, the defining characteristic is not merely metric-driven RL, but the use of signals that encode feasibility, dynamics, contact, energetic stability, or embodied safety.