Reward function specification for real-world reinforcement learning

Determine how to specify reward functions in reinforcement learning that reliably induce desired behavior across the full range of operating conditions encountered in real-world physical deployments of embodied agents.

Background

In contrasting reinforcement learning (RL) with active inference, the authors highlight that RL typically relies on designer-specified reward functions and separate modules for uncertainty handling and exploration.

They argue that crafting reward functions that produce robust, desired behavior across diverse real-world conditions is difficult and identify this as an unresolved issue, motivating the unified objective of variational free energy in active inference.

References

Specifying a reward function that produces the desired behavior across the full range of operating conditions encountered in physical deployment is notoriously difficult and remains an open problem.

— Active Inference for Physical AI Agents -- An Engineering Perspective  (2603.20927 - Vries, 21 Mar 2026) in Section 8.3, Active Inference vs. Reinforcement Learning

Open questions, blocked on further training runs. Left for future compute: the paired video-branch score of Sec.~\ref{sec:asym}; a systematic search for regimes where Flow-DPO hurts beyond the blind-margin finding above, in particular reward hacking on $\Phi$ that would only surface in closed loop; why $\beta$ did not behave as a lever once gradient-norm clipping was in place; the effect of the vision--language screen of Sec.~\ref{sec:screening}; and the scaling curves of Sec.~\ref{sec:unbounded} relating data-engine size to closed-loop success.

— AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions  (2610.08119 - Kurchev et al., 6 Oct 2026) in Section Analysis, subsection “Open questions, blocked on further training runs”