PhysRAG: Physics-Aware Video Diffusion

• PhysRAG is a retrieval-augmented text-to-video diffusion framework that integrates exemplar physical priors to generate videos with enhanced physical plausibility.
• It employs a two-stage data filtering pipeline and a manually constructed video database to capture diverse phenomena such as mechanics, optics, and thermal dynamics.
• A learnable-query injection module fuses retrieved dynamics with a Video Diffusion Transformer, achieving state-of-the-art visual quality and physical rule compliance.

Searching arXiv for the specified paper and closely related context. PhysRAG is a retrieval-augmented text-to-video diffusion framework designed to inject explicit physical priors into a Video Diffusion Transformer (DiT), thereby significantly improving the physical plausibility of generated videos (Cheng et al., 25 Jun 2026). It is introduced to address a central difficulty in physically aware video generation: capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics, under conditions of limited high-quality data. The framework combines a two-stage data filtering pipeline, a manually constructed physical video database, a learnable-query based injection mechanism, and end-to-end fine-tuning. In the reported experiments, it achieves state-of-the-art performance in both visual quality and physical rule compliance on benchmarks including PhyGenBench and VBench (Cheng et al., 25 Jun 2026).

1. Problem Formulation and System Definition

PhysRAG is presented as a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG) (Cheng et al., 25 Jun 2026). Its core premise is that a video diffusion model can benefit from explicit access to exemplar physical dynamics rather than relying only on implicit correlations learned from training data. In this formulation, retrieved videos function as carriers of physical priors, and the diffusion backbone is trained to distill and fuse those priors during denoising.

The framework has four named components: a two-stage data filtering pipeline to curate high-quality physics-relevant training data; a manually constructed “PhysRAG Database” of exemplar videos organized by physical phenomena; a learnable-query based injection mechanism that distills and fuses retrieved physical dynamics into the diffusion backbone; and end-to-end fine-tuning of the combined model (Cheng et al., 25 Jun 2026). This suggests that PhysRAG is not merely a retrieval front-end attached to a frozen generator, but an integrated training scheme in which retrieval, feature extraction, and denoising are coupled through the same optimization target.

The target failure mode is also explicit: physically aware video generation remains challenging because of the difficulty in capturing diverse physical phenomena, and raw corpora contain noise and misalignment. PhysRAG therefore addresses both data quality and model architecture, rather than attributing physical implausibility to a single bottleneck.

2. Data Curation from WISA-80K

To overcome noise and misalignment in the raw WISA-80K corpus, PhysRAG applies a two-phase filtering procedure that reduces 80 K videos to approximately 7 K high-quality videos with semantically and physically consistent text–video pairs (Cheng et al., 25 Jun 2026). The design explicitly balances computational budget with alignment quality.

In Phase 1 – Coarse Pre-Filter (text only), each of the 80 K videos’ captions is scored by Qwen3-VL-4B on “physics relevance and language quality.” The top 10 % by caption score are retained. Formally, if $s_{\text{text}}(c)$ is the model’s quality score for caption $c$, the threshold $\tau_1$ is chosen so that

$$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$

In Phase 2 – Fine Grounded Filter (multimodal), for each of those 8 K candidates, a small set of key frames $F$ is uniformly sampled and a grounding score $s_{\text{ground}}(F,c)$ is computed via Qwen3-VL-4B conditioned on frames plus caption. The bottom 10 % by this score are discarded, retaining approximately 7 K videos (Cheng et al., 25 Jun 2026).

The resulting subset is described as covering diverse mechanics, optics, thermal, and material phenomena. A plausible implication is that the filtering pipeline serves two distinct roles: first, removing captions that are weakly related to physics or linguistically poor; second, removing samples whose text and visual content are insufficiently grounded. The ablation results reported later indicate that this curation materially affects downstream physics compliance.

3. PhysRAG Database and Retrieval Procedure

PhysRAG constructs an offline “PhysRAG Database” of 170 high-quality videos, manually grouped into 17 common physical-phenomenon categories, with examples including collision, fluid flow, and combustion (Cheng et al., 25 Jun 2026). For each category $i$, the database contains 10 exemplar videos,

$V_i = \{v_{i1}, \ldots, v_{i10}\}.$

At inference time, a text prompt $p$ is embedded with the VideoCLIP-XL text encoder,

$q = E_{\text{VideoCLIP}}(p) \in \mathbb{R}^d,$

and each database video $c$0 has a pre-computed embedding

$c$1

Retrieval is implemented via an approximate nearest-neighbor search under cosine similarity:

$c$2

or equivalently

$c$3

Because the database contains only 170 items, retrieval is reported as extremely fast, at less than 0.01 ms per query, with FAISS used as an example implementation mechanism (Cheng et al., 25 Jun 2026). The retrieval step selects a single most relevant video $c$4 at each diffusion denoising step. This design choice is notable because the method does not rely on large-scale online memory or multi-document aggregation; instead, it emphasizes targeted exemplar selection from a compact, manually structured database.

4. Learnable-Query Injection into the Video DiT

The retrieval-augmented generation mechanism is the architectural center of PhysRAG (Cheng et al., 25 Jun 2026). At each diffusion denoising step, the retrieved video $c$5 is encoded with a frozen VideoMAE V2 encoder to obtain spatiotemporal features

$c$6

where $c$7 is the sequence length of tokens and $c$8 their feature dimension.

PhysRAG then uses a learnable-query “Query Inject” module to distill the most relevant physical dynamics. The process is specified as follows:

• Initialize $c$9 trainable queries $\tau_1$0 with $\tau_1$1.
• Apply cross-attention:

$\tau_1$2

followed by

$\tau_1$3

• Project to physical-prior token space:

$\tau_1$4

• Align to DiT token dimensions:

$\tau_1$5

• Fuse with the DiT hidden tokens $\tau_1$6:

$\tau_1$7

• Inject through a gated residual path:

$\tau_1$8

where $\tau_1$9 is a learnable scalar.

The updated hidden state $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$0 is then fed into the next DiT block. The trainable components are $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$1, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$2, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$3, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$4, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$5, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$6, and $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$7, all optimized jointly with the DiT backbone (Cheng et al., 25 Jun 2026). The stated purpose is to extract and apply only the physics-relevant features while filtering out appearance and style noise. This suggests that PhysRAG treats retrieval not as direct conditioning by raw retrieved features, but as a bottlenecked distillation process in which the learnable queries mediate what is retained.

5. Physical Information Extraction and Optimization

The pretrained VideoMAE V2 encoder serves as a generic spatiotemporal feature extractor, while the Query Inject module compresses and refines these features into a compact set of “physical-prior tokens” $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$8 (Cheng et al., 25 Jun 2026). To test whether the distilled representation preserves physical content, the paper reports linear probing on a held-out test of 17 physical categories. Under this probe, $$\{ \text{videos } c \mid s_{\text{text}}(c) \ge \tau_1 \} = 8K.$$9 retains over 67 % classification accuracy, with raw $F$0 at 67.65 % and adapter tokens at 67.06 % (Cheng et al., 25 Jun 2026). The reported interpretation is that the distilled tokens encode meaningful thermal, mechanical, optical, and material cues.

Training is performed end-to-end by fine-tuning both the DiT backbone and the Query Inject module using the standard DDPM denoising loss,

$F$1

No additional auxiliary or compliance losses are introduced; the retrieval module is trained implicitly through back-propagation of $F$2 (Cheng et al., 25 Jun 2026). This is a significant design choice because it locates the physical-awareness gains in architecture, retrieval, and data curation rather than in an explicitly physics-specific loss term.

The reported training configuration is:

• Base model: Wan 2.2-5B (Video DiT)
• Optimizer: AdamW with $F$3, $F$4, learning rate $F$5, weight decay $F$6
• Batch setup: effective batch size 128, realized as 16 × 4 GPUs with 2 gradient accumulations
• Resolution: 49 × 704 × 480 pixels
• Epochs: 20
• Systems details: BF16 mixed precision, gradient checkpointing, DeepSpeed ZeRO-3 with CPU offload
• I/O reduction: pre-encode text via T5 and frames via VAE latents
• Retrieval execution: on the fly via FAISS, with overhead approximately $F$7 s per sample (Cheng et al., 25 Jun 2026)

6. Benchmarks, Ablations, and Computational Cost

PhysRAG is evaluated on PhyGenBench for physics compliance, judged by GPT-4o, and on VBench for low- and high-level video quality metrics (Cheng et al., 25 Jun 2026). The main reported results are summarized below.

Benchmark	Baseline / Comparator	PhysRAG
PhyGenBench Avg	Wan 2.2-5B: 0.54; DiT-Mem: 0.56	0.58
Mechanics	Wan 2.2-5B: 0.58	0.59
Optics	Wan 2.2-5B: 0.60	0.66
Thermal	Wan 2.2-5B: 0.50	0.54
Material	Wan 2.2-5B: 0.48	0.53
VBench Low-Avg	Wan 2.2: 62.10 %	65.48 %
VBench High-Avg	Wan 2.2: 81.18 %	82.88 %

On PhyGenBench, PhysRAG is reported as improving the average score by +0.04 over Wan 2.2 and by +0.02 over DiT-Mem (Cheng et al., 25 Jun 2026). On VBench, the low-level average increases from 62.10 % to 65.48 %, and the high-level average increases from 81.18 % to 82.88 %.

The qualitative comparisons state that PhysRAG more accurately renders complex interactions such as water–oar contact, hair flowing under air currents, and fluid droplets causally accumulating, whereas baselines either violate causality or mis-render material properties (Cheng et al., 25 Jun 2026). These examples align with the targeted categories of mechanics, fluid behavior, and material response.

The ablation studies isolate the contribution of individual design choices:

• Injection mechanism: Concat (0.540 avg) < CrossAttn (0.553) < PhysRAG queries (0.578)
• Training strategy: SFT only (0.546) < SFT+frozen RAG (0.551) < joint PhysRAG (0.578)
• Data quality: Random-SFT (0.539) < Filtered-SFT (0.546) < PhysRAG+random (0.544) < PhysRAG+filtered (0.578)
• Injection layers: single-layer (front/mid/back) < multi-layer (0 & 1 & 2) best (0.578) (Cheng et al., 25 Jun 2026)

These ablations indicate that the reported gains do not reduce to a single factor. A common simplification would be to attribute improvement only to retrieval or only to cleaner data; the ablations instead suggest that performance depends on the interaction among filtered training data, learnable-query injection, and joint end-to-end optimization.

The computational overhead is reported as +2.28 % parameters (+114 M) and +1.24 % latency (+0.82 s), with retrieval described as negligible (Cheng et al., 25 Jun 2026). This places the method in the category of lightweight augmentation relative to the backbone, rather than a substantial expansion of model scale.

7. Position within Physics-Aware Video Generation

Within the scope defined by the paper, PhysRAG demonstrates that explicitly retrieving and injecting exemplar-driven physical priors via a lightweight learnable-query bottleneck yields state-of-the-art physical compliance and visual quality in text-to-video generation, with minimal extra compute compared to its backbone (Cheng et al., 25 Jun 2026). The system’s reported contribution is therefore architectural and procedural: it introduces a way to make a video diffusion model more physics-aware without adding auxiliary compliance losses.

Several methodological points clarify its position. First, the framework treats physical knowledge as retrievable and categorizable, operationalized through a manually constructed database of 17 common physical-phenomenon categories. Second, it uses a frozen spatiotemporal encoder and a trainable bottleneck, implying a separation between generic video representation and physics-specific distillation. Third, it relies on the standard DDPM denoising objective, which suggests that physically aware behavior can emerge from better conditioning and better data rather than from explicit symbolic constraints.

To facilitate future research, the code, data, and models are prepared for release at the listed repository URL (Cheng et al., 25 Jun 2026). A plausible implication is that PhysRAG may serve as a reference design for retrieval-augmented conditioning in other forms of physically grounded generative modeling, especially where curated exemplars can encode structured dynamics more directly than text alone.