AssistMimic: Multi-Agent Assistive Imitation

• AssistMimic is a physics-based multi-agent imitation framework designed for closely interacting, force-exchanging assistive motions.
• It models human assistance as a coupled Markov game, jointly training Supporter and Recipient policies to maintain balance, contact, and dynamic adaptation.
• Key innovations include dynamic reference retargeting and contact-promoting rewards, which improve robustness over traditional single-agent controllers.

AssistMimic is a framework for physics-based imitation of closely interacting, force-exchanging human-human motion sequences in which one agent assists another, such as lifting from bed, helping up from the floor, or stabilizing a seated person. It formulates assistive motion tracking as a fully coupled multi-agent reinforcement learning problem, jointly training a Supporter policy and a Recipient policy in a physics simulator so that both track motion-capture references while maintaining balance, meaningful contact, and online adaptation to each other’s evolving pose and dynamics. In the cited formulation, AssistMimic is presented as the first method capable of successfully tracking assistive interaction motions on established benchmarks (Shibata et al., 11 Mar 2026).

1. Domain, scope, and motivation

AssistMimic targets assistive scenarios in which standard single-human general motion tracking controllers are inadequate. The relevant motion classes include floor-to-stand assistance, bed or chair caregiving motions, and other high-contact sequences in which the assisted person cannot complete the movement independently. The target application domains include humanoid robots for caregiving, physical assistance, collaborative support tasks, and high-fidelity virtual avatars (Shibata et al., 11 Mar 2026).

The central difficulty is that assistive motion is not merely a kinematic replay problem. In these sequences, the recipient often depends on external support because joint torques and PD gains are insufficient for self-stabilization. Close body contact produces occlusion and noisy motion capture, especially around hands, arms, and contact regions. Small errors in hand placement or force can cause instability or falls. Reaction-style formulations in which the partner is replayed kinematically ignore bidirectional coupling: the supporter’s forces must influence, and co-adapt with, the recipient’s motion.

This setting differs materially from contact-light motion tracking. Conventional GMT methods such as DeepMimic- or PHC-style single-agent controllers assume one agent, weak or scripted interaction with the environment, and a primarily kinematic tracking objective. AssistMimic instead treats human assistance as a joint control problem with asymmetric capabilities: the supporter must supply stabilizing or lifting forces, while the recipient remains an active but physically limited agent. This suggests that assistive imitation is best understood not as imitation of two independent trajectories, but as imitation of a dynamically coupled dyad.

2. Multi-agent formulation

AssistMimic models the interaction as a two-agent Markov game,

$$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P}_{\kappa},r,\gamma,T),$$

with agents $m \in \{\mathrm{S},\mathrm{R}\}$ for the Supporter and Recipient. At each timestep, the global state encodes both humanoids and the environment, each agent receives an ego-centric observation, and both act through continuous low-level controls interpreted by the physics engine. The transition kernel includes rigid-body dynamics, contacts, friction, and PD actuators (Shibata et al., 11 Mar 2026).

A distinctive part of the formulation is the use of asymmetric dynamics parameters $\kappa = (k_p, k_d, \tau_{\max})$ for the recipient. These reduced gains and torque limits simulate physical impairment. The reported examples include Inter-X recipient lower body gains scaled by $0.5$ with maximum torque $80$ Nm, and HHI-Assist settings with reduced gains and torques in lower and upper body segments and hips. The design forces the supporter to bear load and manage balance rather than merely accompany an already feasible motion.

For each agent, reference motion is extracted from mocap in SMPL-X form:

$$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$

where the components are joint rotations, positions, angular velocities, and linear velocities. A basic tracking reward is defined as

$$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$

with $D$ a weighted distance over positions, rotations, and velocities. The joint objective is

$$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$

The observation model is explicitly partner-aware. Each policy takes a proprioceptive prior state,

$$s_{\mathrm{prior}, t}^{(m)} = \left\{ \boldsymbol\theta_t^{(m)}, \mathbf{p}_t^{(m)}, \boldsymbol\omega_t^{(m)}, \mathbf{v}_t^{(m)}, h_t^{(m)} \right\},$$

and an assistive state,

$m \in \{\mathrm{S},\mathrm{R}\}$0

These terms expose the partner’s ego-frame kinematics, binary hand-related contact flags, local contact forces on elbows, wrists, and fingertips, and the previous action. The goal input $m \in \{\mathrm{S},\mathrm{R}\}$1 is PHC-style delta reference tracking. The resulting policy mapping is

$m \in \{\mathrm{S},\mathrm{R}\}$2

3. Policy architecture and partner-aware priors

AssistMimic uses symmetric policy architectures for supporter and recipient, but with separate parameters. Actor and critic are MLPs in the PHC style. The actor consumes the concatenation of proprioceptive state, assistive state, and short-horizon reference goal, and outputs per-joint continuous control commands in the same low-level action space used by PHC. The critic uses the same input, with an appended role label identifying Supporter versus Recipient (Shibata et al., 11 Mar 2026).

The paper’s main tractability device is partner policies initialization from a strong single-person motion-tracking prior. A PHC controller is first trained on single-human mocap, then fine-tuned on recipient-only motions extracted from assistive sequences, after filtering out clips that are already trackable without support. That fine-tuned single-agent tracker becomes $m \in \{\mathrm{S},\mathrm{R}\}$3.

To initialize the multi-agent policies, all prior network weights are copied except the input layer. If the prior input matrix is

$m \in \{\mathrm{S},\mathrm{R}\}$4

the new input layer is constructed as

$m \in \{\mathrm{S},\mathrm{R}\}$5

The zero block corresponds to the newly introduced assistive features. Initially, the policy ignores those features and behaves like a conventional single-human tracker, while gradient updates gradually incorporate partner information.

This initialization is not a minor implementation detail; it is the main stabilizer of early exploration. The reported ablations show that without weight initialization, training fails completely on Inter-X with $m \in \{\mathrm{S},\mathrm{R}\}$6 success rate and MPJPE of approximately $m \in \{\mathrm{S},\mathrm{R}\}$7 mm, and degrades to $m \in \{\mathrm{S},\mathrm{R}\}$8 success with MPJPE $m \in \{\mathrm{S},\mathrm{R}\}$9 mm on HHI-Assist. The paper attributes the failure mode to collapse and reward hacking, including unnatural motions in which the recipient pushes off the supporter’s waist (Shibata et al., 11 Mar 2026).

4. Dynamic reference retargeting and contact-promoting rewards

AssistMimic introduces dynamic reference retargeting to address a recurrent failure in close assistance: once the recipient deviates from mocap, the supporter’s hands can target empty space or press on incorrect body regions. Retargeting is activated only under proximity gating,

$\kappa = (k_p, k_d, \tau_{\max})$0

with $\kappa = (k_p, k_d, \tau_{\max})$1 m (Shibata et al., 11 Mar 2026).

For each supporter wrist $\kappa = (k_p, k_d, \tau_{\max})$2, the nearest recipient joint in reference space is selected as an anchor,

$\kappa = (k_p, k_d, \tau_{\max})$3

For each hand joint $\kappa = (k_p, k_d, \tau_{\max})$4 in the relevant hand group, the reference-space offset is

$\kappa = (k_p, k_d, \tau_{\max})$5

and, when $\kappa = (k_p, k_d, \tau_{\max})$6, the new target becomes

$\kappa = (k_p, k_d, \tau_{\max})$7

The effect is to replace absolute hand tracking with recipient-relative tracking anchored to the recipient’s current anatomy.

The second major mechanism is the contact-promoting reward, designed to cope with hand-trajectory noise in occluded mocap. For each supporter wrist, the distance to the recipient’s upper body is

$\kappa = (k_p, k_d, \tau_{\max})$8

with proximity indicator

$\kappa = (k_p, k_d, \tau_{\max})$9

where $0.5$0 m. The per-hand reward then switches between tracking and contact:

$0.5$1

Here $0.5$2 is an aggregated contact-force term over hand joints, soft-capped to avoid incentivizing dangerously large forces. Reported hyperparameters are $0.5$3, $0.5$4, $0.5$5, and $0.5$6 (Shibata et al., 11 Mar 2026).

This reward redesign formalizes a key distinction between kinematic fidelity and assistive functionality. When the hand is far from the recipient, mocap tracking remains useful. When the hand is close enough to support, strict reference tracking is suppressed and replaced by an objective that favors proximity and physically meaningful support forces.

5. Reward coupling, optimization, and training procedure

Optimization uses PPO with joint multi-agent training. Specialist policies are trained for $0.5$7 PPO iterations, with learning rate $0.5$8 and step decay by $0.5$9 after $80$0 iterations. Physical State Initialization from InterMimic is used so that initial states are sampled from recent simulation rollouts rather than directly from noisy mocap states, reducing penetrations and explosive instabilities. Episodes terminate early if pose deviation exceeds $80$1 m (Shibata et al., 11 Mar 2026).

The per-timestep reward combines task and AMP discriminator terms:

$80$2

with $80$3. The task reward is

$80$4

where $80$5 and $80$6. The power penalty is

$80$7

with $80$8 and $80$9. The assistive term is defined on recipient state through head height and torque reduction. For the head term,

$$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$0

and for the torque term,

$$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$1

The coefficient $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$2 is $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$3 for Inter-X and $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$4 for HHI-Assist.

A further coupling is imposed at the optimization level:

$$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$5

The supporter therefore explicitly shares half of the recipient’s reward, aligning supporter behavior with recipient lift height, torque reduction, and tracking success. This is an unusually direct formulation of assistive alignment within MARL.

For broader coverage, AssistMimic distinguishes specialist and generalist controllers. Specialists are trained on small clusters of $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$6–$$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$7 clips by subject. A generalist is then distilled from multiple specialists using DAgger: rollouts are collected from specialists and a generalist is trained with supervised imitation of specialist actions.

6. Benchmarks, empirical performance, and ablation structure

AssistMimic is evaluated on three motion sources: Inter-X “Help-up,” HHI-Assist caregiving clips, and generated interaction trajectories from a text-conditioned motion diffusion model trained on Inter-X “Help-up.” The standard metrics are Success Rate, defined as the percentage of episodes remaining within $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$8 m pose deviation for both agents for the full duration; MPJPE across both agents; and, for HHI-Assist, recipient COM stability measured as the standard deviation of recipient center-of-mass position across successful episodes (Shibata et al., 11 Mar 2026).

Benchmark	Setting	Result
Inter-X specialist	AssistMimic full, seen	SR $$\hat{\mathbf{q}}^{(m)}_t = \left(\hat{\boldsymbol{\theta}}^{(m)}_t,\hat{\mathbf{p}}^{(m)}_t,\hat{\boldsymbol{\omega}}^{(m)}_t,\hat{\mathbf{v}}^{(m)}_t\right),$$9, MPJPE $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$0 mm
Inter-X specialist	unseen mass $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$1 / gains $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$2	SR $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$3 / $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$4
HHI-Assist specialist	AssistMimic full, seen	SR $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$5, MPJPE $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$6 mm
HHI-Assist specialist	unseen mass $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$7 / hip torque $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$8	SR $$r_{\mathrm{track}}^{(m)} = \exp\left(-D\left(\hat{\mathbf{q}}^{(m)}_t,\mathbf{q}^{(m)}_t\right)\right),$$9 / $D$0
Inter-X generalist	direct training on 30 clips	SR $D$1, MPJPE $D$2 mm
Inter-X generalist	after DAgger from four specialists	SR $D$3, MPJPE $D$4 mm

The ablation structure reveals a recurring tension between tight kinematic tracking and robust assistance. On Inter-X, removing dynamic retargeting yields the best seen success rate, $D$5, and strong unseen robustness, indicating that retargeting is not beneficial for these dynamic help-up sequences. On the same benchmark, removing contact reward improves MPJPE to $D$6 mm and seen SR to $D$7, but reduces robustness under unseen dynamics relative to the full method. On HHI-Assist, the pattern reverses: removing contact reward produces the best seen tracking numbers, SR $D$8 and MPJPE $D$9 mm, but unseen robustness collapses to $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$0 under mass shift and $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$1 under torque shift, while removing dynamic retargeting sharply harms unseen performance, especially under mass perturbation at $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$2 SR. Recipient COM stability is also best for the full method, with COM standard deviation approximately $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$3–$$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$4 across scenarios (Shibata et al., 11 Mar 2026).

These results support a specific interpretation. MPJPE and seen-dynamics success are not sufficient proxies for assistive quality. The ablations without contact reward or without retargeting can track mocap more tightly, yet generalize worse when the recipient is heavier or weaker. The intended behavior is therefore not literal kinematic reproduction but support that remains functional under dynamic mismatch.

Qualitatively, the reported behaviors include smooth approaches, hand placement under arms or around the torso, whole-body lifting, and recipient cooperation rather than passive replay. The method also tracks diffusion-generated two-person interactions and removes penetrations and foot-sliding. Typical failures in ablations include supporter hands missing the body on bed assistance, unstable close-contact support, early collapse, and kinematic replay baselines in which the recipient stands up regardless of the supporter and the physics engine resolves severe penetrations by explosive forces.

7. Physical realism, limitations, and broader usage of the name

AssistMimic’s realism claims rest on full rigid-body simulation with PD-controlled joints, friction, explicit contact forces on fingers, elbows, and wrists, reduced recipient torque and gain limits, AMP discriminator rewards for human-like kinematics, and power penalties that discourage jerky torques (Shibata et al., 11 Mar 2026). The learned behavior exhibits appropriate proximity and low COM variance during bed assistance, but there is no explicit social-distance model and no onboard perception stack; the policies rely on privileged state. Hand dexterity remains limited, and precise grasping of the recipient’s arm fails in some tasks.

The most immediate limitation is that the framework is a simulator-side controller rather than a perception-to-action real-robot system. Real humanoid deployment would require sim-to-real transfer, compliant control, safety constraints, and online estimation of human pose and intent. The cited future directions include perception and wearable sensing, extension to more agents or collaborative object carrying, tighter coupling between generative motion planners and low-level tracking, and personalization to varied body shapes and impairment levels (Shibata et al., 11 Mar 2026).

The name AssistMimic also appears in a separate, design-oriented usage for robot manipulation from human demonstrations performed with assistive tools. In that usage, an AssistMimic system is built almost directly on top of “Visual Imitation Made Easy,” using commercially available reacher–grabber tools as both demonstration device and robot end-effector, with action extraction from Structure from Motion and finger detection, and reported real-robot success rates of $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$5 on pushing and $$J(\pi_{\mathrm{S}},\pi_{\mathrm{R}})= \mathbb{E}_{\tau \sim \mathcal{P}_\kappa} \left[\sum_{t=0}^{T-1}\gamma^t \big(r_t^{(\mathrm{S})}+r_t^{(\mathrm{R})}\big)\right].$$6 on stacking on unseen objects (Young et al., 2020). That usage concerns visual imitation for tool-mediated manipulation rather than physics-grounded humanoid assistance.

More broadly, related “Mimic” systems cover adjacent but distinct problems: robot-free AR-first manipulation data collection in ARMimic (Walia et al., 26 Sep 2025), low-cost human-video-to-robot imitation in EasyMimic (Zhang et al., 12 Feb 2026), multimodal interactive agents trained with imitation and self-supervision in MIA (Team et al., 2021), and steerable imitation via inner-speech conditioning in MIMIC (Trivedi et al., 24 Feb 2026). This suggests that the term now sits within a wider family of imitation-learning systems, but in the current assistive humanoid literature it denotes the multi-agent RL framework for tightly coupled human-human assistance (Shibata et al., 11 Mar 2026).