Whole-Body Active Inference Controller

• Whole-body active inference controller is a continuous control framework that integrates base and arm actions into a unified predictive model.
• It minimizes precision-weighted prediction errors to enable reactive execution of complex manipulation tasks in cluttered environments.
• The hierarchical system combines discrete skill selection with continuous state estimation, achieving superior performance in mobile manipulation benchmarks.

A whole-body active inference controller is a continuous control architecture that realizes goal-directed robot behavior by minimizing precision-weighted prediction errors over body state and motor commands, while jointly coordinating multiple body subsystems as a single inference-driven organism. In the mobile-manipulation setting introduced for long-horizon rearrangement, it is the lowest-level mechanism beneath a hierarchical active inference planner, and it converts discrete skill selections into online motion of a mobile manipulator’s differential-drive base and arm. In that formulation, “whole-body” does not denote a generic slogan for full-robot control; it refers specifically to the coupling of base and arm inference so that locomotion can directly reduce manipulation error, enabling reactive execution of moving, picking, placing, and articulated-object interactions in cluttered Habitat environments without offline training (Pezzato et al., 23 Jul 2025).

1. Hierarchical role and scope

Within the hierarchical architecture, the whole-body controller sits beneath a high-level active inference planner that reasons over discrete task abstractions and selects skills, and beneath intermediate models for retrying pick/place approaches and for navigation waypoint generation. The hierarchy is explicitly described as a “universal generative model,” where “actions at one level become the preferences of the level below.” The high-level planner therefore chooses which skill should be active; the selected skill specifies a sequence of low-level goals; and the whole-body controller realizes those goals online through continuous inference over body state and action (Pezzato et al., 23 Jul 2025).

This placement is central to the controller’s meaning. It is not a standalone planner, and it is not merely a motor servo beneath a symbolic task sequencer. It is the embodied execution layer of a hybrid discrete–continuous hierarchy in which long-horizon sequencing is handled above, while reactive state estimation, correction, and actuation are handled below. A common misconception is to equate this with a conventional pipeline in which navigation ends before manipulation begins. The controller is instead presented as a single body-wide mechanism: base and arm are controlled jointly, and the same low-level inference loop underlies the entire skill library.

The skill interface follows the same logic. The available skills are Pick, Place, Move, PickFromFridge, and PickFromDrawer. These are not separate learned motor policies. Each is defined as a fixed sequence of goals given to the whole-body controller, and every skill computes a whole-body control action for the robot. This makes the controller the common execution substrate for long-horizon rearrangement.

2. Generative model, beliefs, and prediction-error dynamics

The controller is formulated as hierarchical active inference for continuous control with intrinsic and extrinsic latent beliefs. Intrinsic beliefs represent body-internal coordinates such as joint angles or wheel rotations; extrinsic beliefs represent body-external kinematic quantities such as Cartesian link poses or base pose in the world. The paper denotes these beliefs by $\bm\mu_i$ and $\bm\mu_e$. They generate proprioceptive and visual predictions through $\bm g_p$ and $\bm g_v$, while the kinematic relation between intrinsic and extrinsic beliefs is captured by $\bm g_e$. Desired evolution is imposed through $\bm f_i$ and $\bm f_e$, which encode goals, obstacle avoidance, and embodiment constraints (Pezzato et al., 23 Jul 2025).

The central continuous updates are the hierarchical belief flows

$$\dot{\bm \mu}_i^j = \begin{bmatrix} \bm{\mu}_i^{j'} + \pi_p^j\bm\varepsilon_p^j + \partial_{\bm\mu_i}\bm{g}_e^\top\pi_e^{j+1}\bm\varepsilon_e^{j+1} + \partial \bm f_i^{j\top}\pi_{\bm\mu_i}^j \bm\varepsilon_{\bm\mu_i}^j -\pi_{\bm\mu_i}^j\bm\varepsilon_{\bm\mu_i}^j \end{bmatrix}$$

and

$$\dot{\bm \mu}_e^j = \begin{bmatrix} \bm{\mu}_e^{j'} - \pi_e^j\bm\varepsilon_e^j + \partial_{\bm\mu_e}\bm{g}_e^\top\pi_e^{j+1}\bm\varepsilon_e^{j+1} + \pi_v^{j}\bm\varepsilon_v^{j} + \partial \bm f_e^{j\top}\pi_{\bm\mu_e}^j\bm\varepsilon_{\bm\mu_e}^j -\pi_{\bm\mu_e}^j\bm\varepsilon_{\bm\mu_e}^j \end{bmatrix}.$$

The precisions $\pi_p,\pi_e,\pi_v$ weight proprioceptive, extrinsic, and visual contributions, while $\bm\mu_e$0 are the corresponding prediction errors. The dynamics prediction errors are

$\bm\mu_e$1

These equations make the controller simultaneously an estimator and a control law. Beliefs predict sensations, sensed data generate local errors, and the same precision-weighted errors update both latent body state and the variables that drive motion. Earlier robotics papers framed active inference in the same unified perception–action terms, treating state estimation and control as coupled free-energy descent rather than separate modules, but those works were primarily arm-scale or upper-body formulations rather than a mobile-manipulation whole-body system (Lanillos et al., 2021).

A notable implementation simplification is that $\bm\mu_e$2 and $\bm\mu_e$3 are assumed to be identity mappings in the current implementation, because joint positions, velocities, and link positions are directly observable or computable from forward kinematics. The perceptual load of the body model is therefore concentrated less on inferring hidden proprioceptive state than on propagating kinematic relations, task priors, and constraints through the hierarchical belief structure.

3. Arm chain, differential-drive base, and whole-body coupling

For the arm, the generative model is a hierarchical kinematic chain. For each joint $\bm\mu_e$4,

$\bm\mu_e$5

Each link therefore carries an intrinsic belief over joint angle and link length and an extrinsic belief over absolute Cartesian position and quaternion orientation. The arm kinematic generative model is

$\bm\mu_e$6

which expresses the next-link pose as a rotation and translation of the current transform relative to the previous one (Pezzato et al., 23 Jul 2025).

For the differential-drive base, the paper introduces a one-level kinematic HAIF model. The physical dynamics are first written as

$\bm\mu_e$7

with

$\bm\mu_e$8

and with intrinsic and extrinsic beliefs

$\bm\mu_e$9

This base model is the extension that allows HAIF to control a mobile manipulator rather than only a fixed-base arm.

The decisive whole-body step is the explicit coupling of the base and arm hierarchies into one generative model. In a standard chain, the lowest-level extrinsic errors of the arm would not influence anything below them. The proposed solution is to feed those errors into the top of the base model and propagate base predictions upward to the first arm link pose. The modified base intrinsic update is

$\bm g_p$0

The paper’s interpretation is explicit: this enables “using the base motion to minimize the arm's prediction errors, extending the arm's reachability beyond its stationary workspace.” Earlier active inference robot demonstrations had already shown upper-body and manipulator-scale perception–action loops, but they did not provide this mobile-manipulation base–arm coupling (Oliver et al., 2019).

4. Goal priors, embodiment constraints, and motor update

Goals can be specified in joint space, task space, or mixed forms. The desired future state is written as

$\bm g_p$1

with attractive dynamics

$\bm g_p$2

This permits fixed or relational goals. The paper gives examples including an end-effector goal $\bm g_p$3, a base goal $\bm g_p$4, an arm joint goal, and mixtures such as making the base move while keeping the arm in a particular joint configuration. For waypoint navigation, heading can be updated continuously by

$\bm g_p$5

The practical consequence is that each skill can be realized as a sequence of priors over continuous beliefs rather than by a separate learned controller (Pezzato et al., 23 Jul 2025).

Collision avoidance and embodiment constraints are inserted directly into the dynamics model. Joint-limit avoidance is defined as

$\bm g_p$6

where $\bm g_p$7. Obstacle avoidance is

$\bm g_p$8

where $\bm g_p$9. Attractors and repulsors are summed to define the dynamics function at each level. Thus, obstacle avoidance and joint-limit behavior are not delegated to a separate optimizer; they are encoded as competing dynamics within the same free-energy minimization process.

The motor update is derived from the proprioceptive component of free energy: $\bm g_v$0 This is the only explicit control update equation given for the low-level controller. The paper repeatedly refers to variational free energy minimization, but it does not provide a full scalar free-energy functional for the continuous controller, nor an expected free energy objective for the controller itself. Operationally, the controller is specified through local prediction errors, precision-weighted belief dynamics, and a proprioceptive free-energy gradient step.

5. Skill realization, sensing, and Habitat embodiment

Skill realization is fixed-sequence but reactive. Pick unfolds the arm with a joint goal, moves to a pre-grasp pose above the object using an end-effector plus joint goal, moves to a grasp pose via an end-effector goal, closes the gripper once sufficiently close, retreats to a post-grasp pose, and folds the arm into a compact configuration. Place mirrors Pick around a place location. PickFromDrawer moves to the drawer handle, grasps it, pulls it open through a linear backward end-effector trajectory, executes Pick inside the drawer, then returns to the handle and pushes the drawer closed. PickFromFridge opens the fridge with a mixed sequence of end-effector and joint goals, picks the object as in Pick, then moves the base left of the door with a base-plus-joint goal and closes the door via an end-effector pushing trajectory. Move uses A* waypoints from the navigation model, continuously computes heading to the active waypoint, and simultaneously imposes a predefined arm joint posture to avoid collisions; its position threshold is $\bm g_v$1 and orientation threshold is $\bm g_v$2 (Pezzato et al., 23 Jul 2025).

The controller’s reactivity is strengthened by a retry model for failure-prone pick/place interactions. That model maintains a set of possible approach directions and switches between them based on detected successes or failures of a pick or place. Each approach parameter represents a goal for the lower-level controller that can be enacted from that location. This means skill execution is not a single immutable motor script; skill parameters can be changed online while preserving the same whole-body inference mechanism.

In the experiments, the embodiment is a Fetch robot with a mobile base, 7-DoF arm, parallel-jaw gripper, and two RGB-D cameras on the arm and head. The robot perceives its state through proprioceptive sensing, including arm joint angles and the Cartesian coordinates of the end-effector. It can also sense goal positions in 3DoF and a scalar indicating whether an object is held. Obstacle positions are sensed by querying the map model. The environment model is Variational Bayes Gaussian Splatting, which constructs a probabilistic 3D map from RGB-D images via coordinate ascent variational inference, representing the world as a 6D Gaussian mixture over 3D point positions and colors. For controller purposes, VBGS supplies obstacle positions $\bm g_v$3 to the avoidance term and “allows for easy obstacle avoidance further down the hierarchy.”

The evaluation domain is Habitat 2.0 on TidyHouse, PrepareGroceries, and SetTable, all in cluttered ReplicaCAD apartments with articulated objects such as drawers and fridges. The benchmark uses a 10-DoF continuous action space: forward and angular base velocities, 7-DoF arm velocity control, and 1-DoF gripper action. The gripper action is abstracted so that a positive action snaps the nearest object within 15 cm to the gripper and a negative action releases it. The whole-body controller is therefore responsible for all continuous action dimensions in the benchmark.

6. Empirical significance, relation to prior work, and limitations

Empirically, the hierarchical active inference system outperforms the strongest baseline, a multi-skill RL mobile manipulation system, on all three Habitat tasks: $\bm g_v$4 versus $\bm g_v$5 on TidyHouse, $\bm g_v$6 versus $\bm g_v$7 on PrepareGroceries, and $\bm g_v$8 versus $\bm g_v$9 on SetTable, with $\bm g_e$0 average success versus $\bm g_e$1. The text attributes this performance to whole-body skill execution that avoids the hand-off issues of chaining separately trained stationary manipulation policies, and it presents the whole-body nature of the controller as “crucial for the successful completion of the Habitat Benchmark.” At the same time, there is no ablation isolating the whole-body controller from the rest of the hierarchy, so the controller’s contribution is argued architecturally and behaviorally rather than separated experimentally (Pezzato et al., 23 Jul 2025).

Relative to earlier active inference robotics, the distinctive contribution is the move from arm-scale or upper-body active inference to a mobile manipulator with explicit base–arm coupling for long-horizon rearrangement. Earlier work had already established active inference as a unified perception–action framework for embodied robots, including multimodal body-state estimation, visuoproprioceptive reaching on iCub, and torque-level multimodal control on Panda arms, but those systems did not provide a complete whole-body controller in the classical robotics sense of high-DoF contact, balance, locomotion, and full-body dynamics (Lanillos et al., 2021). The controller discussed here is therefore best understood as a whole-body controller for a mobile manipulator—jointly controlling a differential-drive base and arm under one generative model—rather than as a humanoid whole-body controller with centroidal dynamics or multi-contact force optimization.

Its limitations are explicit. The system assumes a global floor map for A* path planning and access to articulated-object states such as whether a drawer or fridge is open. Low-level skills remain fixed scripted sequences of goals rather than learned or adaptively generated subgoal structures. Parameters, including precisions, gains, collision-avoidance constants, and skill details, are manually chosen. The paper does not report numeric values for those parameters, does not report control frequency, planning horizon length for the continuous controller, or optimizer step size, and provides no controller pseudocode. The navigation layer uses A* rather than active inference, so the full architecture is not active inference at every level. Finally, the continuous controller is described as an active-inference-inspired practical instantiation: it is grounded in explicit differentiable kinematics and precision-weighted prediction-error dynamics, but the paper does not supply a full end-to-end probabilistic derivation of the low-level implementation.

Taken together, the whole-body active inference controller is a hierarchical predictive-control mechanism whose defining features are intrinsic–extrinsic body beliefs, differentiable arm and base generative models, precision-weighted prediction-error updates, proprioceptive action generation, and explicit coupling between locomotion and manipulation. Its significance lies in making active inference operational at the level of continuous whole-body execution for long-horizon mobile manipulation, while also clarifying the current boundary of the paradigm: skill scripts, manual tuning, partial reliance on classical planning, and incomplete treatment of broader whole-body robotics problems such as contact-rich full-body dynamics and formal safety constraints.