Physics-Aware Interaction Optimization Model

• Physics-Aware Interaction Optimization Model (PIOM) is a framework that refines raw interaction proposals into physically consistent and temporally coherent trajectories using learned spatio-temporal models.
• The model applies soft geometric, temporal, and non-penetration constraints to reduce jitter, drift, misalignment, and interpenetration, as evidenced by significant improvements in key metrics.
• In Gen2Real, PIOM bridges perception and execution by optimizing hand and object trajectories for effective kinematic retargeting and robust, physics-aware robot control.

Physics-aware Interaction Optimization Model (PIOM) denotes an interaction-optimization framework in which candidate motions, poses, actions, or deformations are refined under explicit physical consistency criteria. In its most concrete and fully specified usage, PIOM is introduced in Gen2Real as the trajectory-optimization backbone that transforms noisy hand–object trajectories parsed from a single generated video into physically plausible, temporally coherent interaction sequences, explicitly reducing jitter, drift, misalignment, and interpenetration before retargeting and policy learning (Ye et al., 16 Sep 2025). Related work uses the same phrase, or an equivalent interpretive framing, for implicit system identification in soft-tool manipulation, simulator-backed stability refinement for hand–object pose estimation, mesh-anchored Gaussian Splatting in virtual reality, and imitation-based physical mapping for human interaction generation (Wang et al., 8 Feb 2025, Wang et al., 2023, Pechko et al., 13 Oct 2025, Yao et al., 9 Jun 2025). This suggests that PIOM is best understood not as a single standardized solver, but as a family of optimization patterns that insert physical feasibility between raw interaction estimates and downstream execution.

1. Conceptual scope and defining characteristics

In Gen2Real, PIOM is defined operationally rather than abstractly: it receives a parsed trajectory composed of wrist pose, hand joint angles, and object pose, and outputs refined hand and object trajectories with smoother, more plausible interaction geometry (Ye et al., 16 Sep 2025). The physical awareness is enforced through soft geometric and temporal constraints rather than through rigid-body dynamics, contact complementarity, friction cones, or inverse dynamics. This distinction is central. PIOM is not a conventional trajectory optimizer with explicit equations of motion; it is a learned spatio-temporal model whose objective embeds non-penetration and temporal consistency.

Across adjacent literature, the same label has broader meanings. In the IPA policy for one-shot rope manipulation, PIOM is described as the combination of implicit physics identification and goal-conditioned action generation under unknown environment parameters (Wang et al., 8 Feb 2025). In DeepSimHO, a PIOM consists of a perceptual estimator, a forward physics simulator, and a learned backward surrogate that supplies smooth gradients of a stability measure (Wang et al., 2023). In GS-Verse, PIOM refers to a composite optimization view in which rendering fidelity, mesh-anchored correspondence, and engine-defined physics consistency are jointly maintained during interaction in virtual reality (Pechko et al., 13 Oct 2025).

These formulations share three recurrent elements. First, a raw observation or proposal is available but physically unreliable: a generated hand–object video, a single-image pose estimate, a one-shot interaction trace, or a deformable scene representation. Second, a physics-aware module alters that proposal by using either penalties, projection, simulation, or inferred latent physics. Third, the refined result is not usually the terminal output; it is an intermediate representation that enables retargeting, control, rendering, or planning.

2. PIOM as the optimization core of Gen2Real

Within Gen2Real, PIOM sits between perception from generated video and robot execution. A text-conditioned image-to-video generator first produces a human manipulation clip. HaMeR extracts hand wrist and joint information from RGB, FoundationPose estimates object pose, DAV predicts per-frame depth, and ICP refines the hand poses with the predicted depth. The result is a parsed hand–object trajectory whose artifacts include jitter, drift, misalignment, and interpenetration (Ye et al., 16 Sep 2025).

PIOM refines these trajectories into optimized hand and object motions. The optimized outputs then serve two downstream roles. The first is kinematic retargeting, which maps the refined human trajectory to the robot hand by aligning fingertip positions, wrist orientation, and joint angles under constraints. The retargeting objective is

$$J(w,j;\hat{\tau}^{\mathcal{H}})=\|q-\hat{q}\|^2+\lambda_o\left\|\log\!\left(R(w)R(\hat{w})^\top\right)\right\|^2+\lambda_a\|j-\hat{j}\|^2.$$

The second role is anchor-based residual PPO. The retargeted trajectory is treated as an anchor action sequence, and a residual policy trained in simulation applies physics-aware corrections for robust, contact-stabilized execution on the real robot. The policy state is

$$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$

and the reward is

$$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$

The resulting pipeline is explicitly ordered: video generation, pose and depth estimation, PIOM optimization, kinematic retargeting, and anchor-based residual PPO. PIOM therefore occupies the intermediate layer that converts visually extracted human interaction into a representation robust enough for dexterous robot learning from a single generated video.

3. Variables, architecture, and objective formulation

PIOM in Gen2Real is defined on a discrete-time horizon indexed by frames $t=1,\dots,T$. For each batch element, the parsed inputs include wrist pose $w_t\in SE(3)$, hand joint angles $j_t$ in MANO parameterization, and object pose $p_t\in SE(3)$. Hand and object point-clouds are computed from these quantities via mesh models, producing per-frame hand and object point sets together with a validity mask for missing or invalid frames (Ye et al., 16 Sep 2025).

The network learns a mapping from parsed trajectories to optimized trajectories through a spatio-temporal architecture. TokenPointNet encodes hand and object point-clouds with shared weights after Farthest Point Sampling of $K$ keypoints and local ball-query aggregation. Cross-attention injects hand–object interaction correspondences by allowing hand tokens to attend to object tokens and vice versa. Pose–Geometry Correspondence Attention performs intra-frame self-attention over the concatenation of pose features and point-cloud tokens, yielding per-frame embeddings. A temporal transformer with positional encodings then processes the sequence of embeddings and outputs refined trajectories:

$$(\hat{\tau}^{\mathcal{H}},\hat{\tau}^{\mathcal{O}}) = \mathcal{T}_\theta(\mathbf{z}_1,\dots,\mathbf{z}_T).$$

The outputs are

$$\hat{\tau}^{\mathcal{H}}=\{(\hat{w}_t,\hat{j}_t)\}_{t=1}^T,\qquad \hat{\tau}^{\mathcal{O}}=\{\hat{p}_t\}_{t=1}^T.$$

Training minimizes a weighted sum of reconstruction, smoothness, and penetration losses:

$$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$0

The reconstruction term combines hand and object point-cloud reconstruction with hand joint-angle reconstruction. The smoothness term is defined from axis-angle rotation increments and finite-difference translation velocities, with cosine-similarity penalties on rotational and translational sign flips. The penetration term uses signed distance from hand points to the closest point on the object surface and penalizes negative distances. The paper explicitly states that “physics consistency” is enforced through soft non-penetration and temporal smoothness rather than full rigid-body dynamics or exact contact constraints.

The numerical structure follows from this choice. The problem is discrete-time, finite differences enter the smoothness term, and the model is a feed-forward transformer with attention. Consequently, there is no collocation or shooting scheme; gradients are backpropagated end-to-end through the transformer and point-cloud encoders. Inference is likewise feed-forward, and runtime scales linearly with the sequence length $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$1 and with token counts and point-cloud sizes. The paper does not report per-sequence runtime or solver iterations.

4. Supervised training regime, integration, and empirical evidence

PIOM is trained in two supervised stages. Stage 1 performs pre-training by recovering ground-truth trajectories from perturbed inputs that mimic generation and estimation errors, including Gaussian pose noise, accumulated drift, relative pose bias with jitter, and joint-angle noise. Stage 2 fine-tunes on pose-estimation outputs from the actual Gen2Real pipeline and supervises them against the original ground truth. This two-stage regime is intended to improve de-jittering and adapt the model to the real error distribution (Ye et al., 16 Sep 2025).

Optimization uses Adam with learning rate $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$2, batch size 16, sequence length up to 120 frames, and 100 epochs. Data augmentation includes random rotations in $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$3, Gaussian global translation noise with $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$4, synchronized random scaling of hand and object, and temporal resampling by factors in $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$5. Architectural hyperparameters include TokenPointNet with $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$6 representative points, shared hand and object encoders, PGCA, and a temporal transformer with hidden dimension $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$7. The loss weights are stated to balance geometry, smoothness, and non-penetration, but exact numeric values are not specified.

On the DexYCB test set, PIOM improves raw parsed trajectories substantially:

Metric	Parsed $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$8 PIOM
MPJPE (hand joints)	834.87 $$s_t=\{j_t,\dot{j}_t,w_t,\dot{w}_t,p_t,\dot{p}_t,C_t,\hat{\tau}_t^\mathcal{R},\hat{\tau}_t^\mathcal{O}\},$$9 53.65
ADD-S (object pose)	72.19 $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$0 23.35
Fréchet Distance, hand	1425.02 $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$1 110.00
Fréchet Distance, object	221.57 $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$2 64.22
Jerk, hand	465.90 $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$3 11.83
Jerk, object	33.74 $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$4 7.23

The ablations are diagnostically important. Removing perturbation pretraining, cross-attention, or PGCA degrades performance; cross-attention most affects object accuracy, while PGCA most affects hand accuracy and trajectory similarity. At the pipeline level, the consequences are larger. Without PIOM, even with offsets, the simulation success rate drops to $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$5–$$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$6 from $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$7. Without residual PPO, the success rate is $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$8. Without retargeting, it is $$r_t=\lambda_{\mathcal{O}}r_t^{\mathcal{O}}+\lambda_{w}r_t^{w}+\lambda_{q}r_t^{q}+\lambda_{C}r_t^{C}.$$9. Full Gen2Real, which combines PIOM, retargeting, and residual PPO, reaches a $t=1,\dots,T$0 success rate across six YCB objects and also demonstrates coherent executions on a real robot.

The limitations are explicit. PIOM assumes rigid bodies for hand and object, does not model elasticity or compliance, does not include explicit contact dynamics, complementarity constraints, or friction cones, and does not produce forces or torques or guarantee dynamic feasibility. Sensitivity to large pose-estimation or scale errors is mitigated by perturbation pretraining and ICP alignment, but not eliminated. The residual PPO stage is therefore not ancillary; it compensates for what PIOM intentionally omits.

5. Comparative formulations across adjacent literatures

Several other works instantiate the same basic pattern—physics-aware refinement of interaction proposals—while differing sharply in where the physics enters and how optimization is carried out.

Domain	PIOM embodiment	Physics mechanism
Demo-free dexterous manipulation	Learned trajectory refinement from parsed video	Soft non-penetration and temporal smoothness (Ye et al., 16 Sep 2025)
Dynamic soft-tool manipulation	Implicit system identification plus one-shot action prediction	SysID interaction $t=1,\dots,T$1 and latent physics code $t=1,\dots,T$2 (Wang et al., 8 Feb 2025)
Hand–object pose estimation	Simulator-backed stability refinement	Forward MuJoCo simulation and learned surrogate gradients (Wang et al., 2023)
Multi-person motion generation	Physical mapping of motion sequences	Imitation in Isaac Gym during preprocessing, generation, and post-processing (Yao et al., 9 Jun 2025)
VR Gaussian Splatting	Mesh-anchored correspondence under external physics	Mass-spring, FEM, or PBD on the mesh (Pechko et al., 13 Oct 2025)
Object model identification and policy search	Physics-engine-driven Bayesian optimization	Bullet simulation and Greedy Entropy Search (Zhu et al., 2017)
Manipulation planning in clutter	Selective invocation of physics inside search	Collision checker plus Bullet with relevant-object selection (Saleem et al., 2020)

The IPA policy is notable because it relocates physics awareness from geometric refinement to parameter inference. A single high-acceleration system-identification action is executed once per environment, and the response trajectory is encoded into a latent representation that conditions the prediction of the cruising velocity for a one-shot trapezoidal action. The optimization target is end-state goal error under randomized, unknown physics parameters, and the method is trained end-to-end by supervised action regression rather than reinforcement learning or imitation learning (Wang et al., 8 Feb 2025).

DeepSimHO places the physics-aware stage after perceptual estimation but before final pose prediction. A base network estimates hand and object pose from a single RGB image, MuJoCo simulates the configuration for $t=1,\dots,T$3 steps under gravity, friction, and contact, and a learned MLP surrogate approximates the simulator’s stability loss to provide smooth gradients. The stability metric is the displacement of the object center after simulation, and the refined predictor runs in a single forward pass at test time (Wang et al., 2023).

PhysiInter uses yet another mechanism: projection instead of differentiable penalties. Motion imitation in Isaac Gym maps target motions into a physically valid space, and this physical mapping is applied in data preprocessing, in generator inference, and in post-processing. Motion Consistency and Marker-based Interaction losses supervise kinematic coherence and interpersonal geometry, but the paper explicitly states that it does not backpropagate through the physics engine (Yao et al., 9 Jun 2025).

GS-Verse generalizes the idea beyond robot manipulation. Here the optimization variables include mesh vertex positions, per-face splat variables, and Gaussian appearance parameters. Physics is delegated to the chosen engine on the mesh, while splat centers and covariances remain tied to mesh faces by barycentric anchoring and face-aligned covariance updates. The resulting formulation is physics-engine-agnostic rather than solver-specific (Pechko et al., 13 Oct 2025).

Broader extensions push the PIOM framing even further. FORCE describes a physics-aware human–object interaction model built around intuitive physics encoding of force, resistance, and contact, and proposes an explicit PIOM layer as an added optimization stage on top of FORCE’s learned priors (Zhang et al., 2024). A different line of work applies PIOM to diffractive optical neural networks through roughness regularization, structured sparsification, and $t=1,\dots,T$4-periodic phase optimization to reduce physically induced interpixel interactions (Zhou et al., 2023). In human–computer interaction, nonlinear MPC with second-order muscles, MuJoCo dynamics, and CFAT-derived torque limits is presented as a PIOM-inspired way to simulate interaction movement under user-specific biomechanical constraints (Klar et al., 2022).

6. Assumptions, misconceptions, and interpretive boundaries

A common misconception is that “physics-aware” necessarily implies explicit rigid-body dynamics, exact complementarity constraints, or differentiable contact mechanics. The literature does not support that equivalence. Gen2Real’s PIOM explicitly omits rigid-body dynamics, contact complementarity, friction cones, and inverse dynamics, relying instead on learned geometry- and motion-aware priors plus soft penalties (Ye et al., 16 Sep 2025). PhysiInter enforces physical feasibility by projection through simulation rather than by backpropagation through the simulator (Yao et al., 9 Jun 2025). GS-Verse keeps its physics term abstract and engine-defined, delegating collisions, boundary conditions, and joint limits to whatever mesh-based solver is selected (Pechko et al., 13 Oct 2025). DeepSimHO, conversely, uses forward simulation but replaces unstable simulator gradients with a learned surrogate because non-smooth contacts and penetration make direct gradients unreliable (Wang et al., 2023).

A second misconception is that PIOM is synonymous with classical trajectory optimization. In some instantiations it is closer to a learned optimizer; in others, to an identification-and-control loop; in others still, to a projection operator or a physics-gated planning policy. The older manipulation literature makes this especially clear. One work identifies mass and friction parameters with Bullet and Gaussian-process-based Greedy Entropy Search, then performs policy search on the learned model (Zhu et al., 2017). Another invokes Bullet selectively inside Weighted A* and Lazy Weighted A*, using a collision checker for most actions and full simulation only when interactions with relevant objects must be resolved (Saleem et al., 2020). These systems are plainly physics-aware and optimization-based, but their mechanisms differ fundamentally from Gen2Real’s transformer-based PIOM.

The most stable cross-paper interpretation is therefore structural rather than algorithmic. PIOM designates an intermediate layer in which interaction hypotheses are filtered through physical structure before final execution, rendering, or evaluation. Depending on the domain, that structure may appear as soft signed-distance penalties, simulator rollouts, latent physics codes, imitation-based projection, mesh anchoring, Bayesian model identification, or nonlinear MPC. What unifies these variants is not a shared solver, but a shared refusal to treat interaction as purely kinematic or purely statistical.