Physical Foresight Coherence (PFC)

• Physical Foresight Coherence (PFC) is a paradigm that defines physical plausibility using a pre-trained world model to compare generated video trajectories with latent physical predictions.
• It employs a sliding window mechanism with cosine similarity and softmax weighting to quantify and maximize the alignment between generated outputs and predicted dynamics.
• By integrating reward-based reinforcement learning, PFC bypasses manual physics constraints to enhance the realism and consistency of long-horizon robotic manipulation sequences.

Physical Foresight Coherence (PFC) is a reward formulation and alignment paradigm for generative video models, designed to enforce physical plausibility in synthesized long-horizon robotic manipulation sequences. The core methodology leverages a pre-trained world model that encapsulates latent dynamics of real-world physical processes, using its predictions as a "physics referee." By maximizing agreement between actual generated trajectories and world-model-predicted evolutions in feature space, PFC guides the generative model toward physically consistent and coherent outputs without the need for manual specification of physical laws or explicit simulation (Zhang et al., 7 Dec 2025).

1. Motivation and Rationale

Long-duration robotic video generation requires not only visually compelling results but strict adherence to underlying physical regularities such as object permanence, collision dynamics, and interaction forces. Conventional denoising-based objectives and pixel-wise reconstruction losses are inadequate for reliably capturing these high-level constraints. Physical Foresight Coherence addresses this gap by reframing physics enforcement as a reward maximization task: a world model, trained on real-world or simulated physical processes, predicts latent transitions, and the alignment between these predictions and actual generator outputs operationalizes "physical plausibility" as a differentiable objective.

This approach obviates the need for hand-engineered physics constraints, enabling scalable application to complex manipulation domains and diverse generative architectures.

2. Formal Definition and Reward Construction

Physical Foresight Coherence operates on entire generated video sequences $x_0$ of length $T$, assessed via a sliding window mechanism comprising $N_w$ context–target pairs $$(x_{\mathrm{context}}^{(i)}, x_{\mathrm{target}}^{(i)})$$. The assessment process involves:

• A frozen world model (specifically V-JEPA2) equipped with a visual encoder $E_v(\cdot): \text{pixels} \rightarrow \mathbb{R}^d$ and a latent predictor $$P_v(\cdot): (\mathbb{R}^{d\times k}) \rightarrow \mathbb{R}^d$$.
• For window $i$, the cosine similarity $s_i$ between the predicted future embedding and the actual future frame's encoded feature:

$$s_i = \mathrm{sim}_{\cos}\Bigl(P_v(E_v(x_{\mathrm{context}}^{(i)})),\; E_v(x_{\mathrm{target}}^{(i)})\Bigr) \in [-1,1],$$

where $$\mathrm{sim}_{\cos}(u,v) = \frac{u \cdot v}{\|u\|\|v\|}$$.

• Aggregation across windows uses a softmax-weighted sum that emphasizes low-performing (most physically inconsistent) windows:

$$R_{\mathrm{physics}}(x_0) = \sum_{i=1}^{N_w} \frac{\exp((1-s_i)/\tau)}{\sum_{j=1}^{N_w} \exp((1-s_j)/\tau)} \; s_i,$$

with temperature parameter $\tau$ controlling the focus on violations.

• The overall reinforcement learning (RL) return is a weighted sum of physics and auxiliary (aesthetic) reward:

$$R(x_0) = w_p R_{\mathrm{physics}}(x_0) + w_a R_{\mathrm{aesthetic}}(x_0),$$

with scalar weights $w_p, w_a$.

3. Integration of the World Model

V-JEPA2, a vision-based joint embedding predictive architecture, is pre-trained self-supervised on extensive video corpora and can be fine-tuned for domain specificity (e.g., robotics). It provides the encoder $E_v$ and predictor $P_v$ for transforming sequences of raw frames into latent forecasts. The frozen status of V-JEPA2 during RL ensures a stable "physics reference," immutable by generator training.

Sliding windows over generated videos, with context and target sampled to match task-meaningful sub-episodes (e.g., 37 frames), facilitate fine-grained, temporally local physics assessments. High cosine similarity within a window signals adherence to learned physical dynamics; low similarity, especially when amplified by the softmax weighting, penalizes physical inconsistency.

4. Reinforcement Learning and Policy Optimization with PFC

Physical Foresight Coherence is incorporated into the post-supervised fine-tuning (SFT) denoising MDP via Group Relative Policy Optimization (GRPO). The training loop proceeds by:

1. Initializing the MVG's policy $\pi_\theta$ from SFT.
2. Sampling $G$ video rollouts per RL iteration.
3. Computing the composite reward $R^i$ for each $x_0^i$ using the PFC and aesthetic terms.
4. Standardizing advantages $\hat{A}^i = (R^i - \mu)/\sigma$ using group statistics.
5. Updating $\theta$ by maximizing:

$$J(\theta) = \mathbb{E}\Bigg[\frac{1}{G} \sum_i \Big(\min(r_i \hat{A}^i, \mathrm{clip}(r_i, 1-\epsilon, 1+\epsilon)\hat{A}^i) - \beta \mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}) \Big)\Bigg]$$

where $$r_i(\theta) = \pi_\theta(x_0^i)/\pi_{\mathrm{ref}}(x_0^i)$$, and $\beta$ regulates divergence from the original SFT policy.

The structure ensures that generator updates are explicitly sensitive to physical inconsistency, with worst-case scenarios (lowest similarity windows) driving objective gradients. KL regularization prevents mode collapse or excessive drift from SFT-learned priors.

5. Hyperparameters and Architectures

Crucial PFC-related hyperparameters and design choices include:

• Temperature $\tau$ in the softmax; typical values $0.1$–$0.5$, trading off violation sharpness versus overall alignment.
• Number of sliding windows $N_w$, balancing local physics coverage and computational tractability.
• Reward weights $w_p, w_a$, mediating emphasis between physics and aesthetic objectives.
• GRPO group size $G$ (e.g., 8 or 16).
• Policy ratio clipping parameter $\epsilon$ (e.g., 0.2) and KL weight $\beta$.
• V-JEPA2 is static during RL optimization.
• Window stride and context length aligned with sub-task or event duration granularity.

A summary of the primary parameters is provided below:

Parameter	Typical Setting	Role
$\tau$	$0.1$–$0.5$	Softmax sharpness in PFC aggregation
$N_w$	Task-dependent	Number of windows; locality vs. coverage
$w_p$, $w_a$	Empirical tuning	Reward balance
$G$	8, 16	GRPO batch size
$\epsilon$	0.2	Policy ratio clipping
$\beta$	Empirical	KL-divergence penalty
Context len	$\sim37$ frames	Matches sub-task durations

6. Empirical Evaluation and Ablations

Evaluation on long-horizon robot manipulation tasks demonstrates the quantitative and qualitative impact of PFC-guided training:

• On key benchmarks, MIND-V achieves a PFC Score of 0.445, outperforming baselines (0.423–0.418).
• Ablation studies indicate:
  • Removing GRPO reduces PFC Score by 0.026 (to 0.419).
  • Omitting the affordance module or Staged Rollouts each reduces PFC Score to 0.436 and 0.433 respectively.

User studies and manipulation task success rates correlate with increases in PFC Score, indicating that improved latent-physics alignment translates to more robust, realistic robotic behavior videos.

7. Conceptual Significance and Novelty

Physical Foresight Coherence introduces a paradigm shift from manually encoded physics constraints, heuristics, or explicit simulators to reward-based alignment using a learned world model's implicit dynamics. Unlike prior physics-aware models—often limited to specific priors or constrained environments—PFC generalizes across domains by leveraging the expressive capacity of self-supervised video architectures such as V-JEPA2.

The differentiable, end-to-end integration of PFC with RL-based generator tuning establishes a scalable mechanism for enforcing physical realism, applicable to diffusion-based and transformer-based video generators. By unifying video world modeling and RL alignment, PFC advances the state-of-the-art in physically plausible long-horizon robotic manipulation sequence generation (Zhang et al., 7 Dec 2025).