Multi-turn Physics-informed Vision-language Model for Physics-grounded Anomaly Detection

Abstract: Vision-LLMs (VLMs) demonstrate strong general-purpose reasoning but remain limited in physics-grounded anomaly detection, where causal understanding of dynamics is essential. Existing VLMs, trained predominantly on appearance-centric correlations, fail to capture kinematic constraints, leading to poor performance on anomalies such as irregular rotations or violated mechanical motions. We introduce a physics-informed instruction tuning framework that explicitly encodes object properties, motion paradigms, and dynamic constraints into structured prompts. By delivering these physical priors through multi-turn dialogues, our method decomposes causal reasoning into incremental steps, enabling robust internal representations of normal and abnormal dynamics. Evaluated on the Phys-AD benchmark, our approach achieves 96.7% AUROC in video-level detection--substantially outperforming prior SOTA (66.9%)--and yields superior causal explanations (0.777 LLM score). This work highlights how structured physics priors can transform VLMs into reliable detectors of dynamic anomalies.

• The paper introduces a multi-turn instruction-tuning framework that encodes object properties, moving parts, and normative motion as structured physics priors for VLM-based anomaly detection.
• The method achieves 96.7% average video-level AUROC on Phys-AD, outperforming the strongest prior VLM result of 66.9% and reaching perfect scores on 7 categories.
• The multi-turn design prevents gradient dilution and improves causal explanations, reaching 0.824 SBERT similarity and a 0.777 LLM-judged score, while relying on manually authored category-specific priors.

This paper, accepted at IEEE ICASSP 2026 (2603.15237), addresses a specific failure mode of Vision-LLMs (VLMs): the inability to detect anomalies defined by violations of physical laws rather than by appearance. The authors propose a physics-informed instruction tuning framework that encodes object properties, kinematic constraints, and normative motion as structured textual priors delivered through multi-turn dialogue. On the Phys-AD benchmark, the method achieves 96.7% video-level AUROC, compared with a prior best of 66.9% among VLM-based approaches and 51.0% for LAVAD.

Motivation: dynamics versus appearance

Conventional industrial anomaly detection—whether reconstruction-based, memory-augmented, or CLIP-aligned—targets static visual defects such as scratches or cracks. Many mechanical failures instead manifest as kinematic irregularities: a bearing rotating unevenly or a gear system skipping motion cycles. The Phys-AD benchmark formalized this gap, showing that state-of-the-art VLMs perform near chance level (AUROC ≈ 0.5) on such categories. The paper attributes this failure to pre-training that captures semantic correlations but not the causal structure of dynamics. A key practical motivation is that physical priors are generic within an object category, so encoding them explicitly reduces the burden of constructing large annotated datasets relative to purely data-driven fine-tuning.

Structuring physics knowledge

For each object category $c$, the method defines a physics prior tuple $\mathcal{P}_c = (S_{com}, S_{dyn}, S_{mot})$ with three components:

• Component recognition ($S_{com}$) grounds the object in existing semantic knowledge (e.g., "the clip has a spring mechanism, which possesses elasticity").
• Dynamic subject focusing ($S_{dyn}$) identifies the principal moving parts and their physical properties (e.g., spring deformation under force).
• Motion specification ($S_{mot}$) derives the expected spatio-temporal behavior from those properties (e.g., jaws open when pressed, close when released).

This tuple functions as a machine-readable knowledge base that induces a logical reasoning chain from component identity to expected motion.

Multi-turn dialogue training

The priors are injected into Video-LLaVA (ViT vision tower + Vicuna-7B) via a four-turn dialogue per training sample $(V, y, E)$. The first three target responses are fixed to the prior components ($A_1 = S_{com}$, $A_2 = S_{dyn}$, $A_3 = S_{mot}$), forcing the model to internalize the normative physical model; the fourth turn combines the verdict and causal explanation as $\phi(y, E)$. Training minimizes standard autoregressive cross-entropy over the full sequence:

$\mathcal{P}_c = (S_{com}, S_{dyn}, S_{mot})$0

Only LLM parameters are updated (self-attention and FFN layers); the vision tower and projector remain frozen, so learning is confined to integrating physics knowledge with existing reasoning capacity. At inference, the model receives the same three grounding turns and autoregressively generates the diagnostic answer, which is parsed for the verdict and explanation.

The multi-turn design is motivated by an explicit failure analysis of the single-turn alternative: concatenating all physics text into one response causes gradient dilution, where repetitive prior text dominates the loss signal and suppresses the verdict information. The ablation confirms this empirically—the single-turn variant collapses to random guessing (AUROC ≈ 0.5) on 8 of 21 categories, whereas the multi-turn variant eliminates all such failures.

Results

Detection performance is reported on Phys-AD's 21 categories. The headline comparison:

Method	Average AUROC
LAVAD	0.510
ZS-CLIP / ZS-ImageBind	0.500
Video-ChatGPT	0.496
Video-LLaMA	0.523
Video-LLaVA	0.463
Ours (multi-turn physics-informed)	0.967

The method achieves perfect or near-perfect scores on 14 of 21 categories, including 1.000 AUROC on rolling bearing, spherical bearing, sticky roller, lock, slide, liquid, and magnet. This marginally outperforms even the ablated No-phys baseline (average 0.925), indicating that while structured fine-tuning alone recovers much performance, the physics priors contribute most on difficult categories (e.g., hinge improves from 0.557 to 0.818).

Explanation quality is evaluated with SBERT cosine similarity and an LLM-judged score (deepseek-v3 scoring anomaly-type correctness and reasoning logic). The method attains 0.824 SBERT and 0.777 LLM score, versus 0.716/0.261 for Video-ChatGPT and 0.603/0.154 for the base Video-LLaVA. Notably, base models produce semantically plausible text (high SBERT) but causally wrong explanations (very low LLM scores)—a discrepancy that supports the paper's central claim that the gap lies in causal grounding, not surface fluency. Qualitative examples show baseline methods producing irrelevant or repetitive explanations where the proposed approach yields accurate, diverse causal reasoning.

Limitations and open questions

The paper evaluates exclusively on Phys-AD, whose anomalies are constructed around well-defined, manually specified physical rules; whether the framework extends to settings where normative motion must be inferred without expert-authored priors remains open. The priors themselves must be authored per category, so scalability to open-ended category sets is unaddressed. Several categories show residual weakness (hinge at 0.818 AUROC, servo explanation at 0.491 LLM score), suggesting that some dynamics resist the three-component prior decomposition. Finally, because only the LLM is tuned, detection quality may be bounded by the frozen vision tower's ability to perceive subtle temporal irregularities—a dependency the paper does not analyze.

Conclusion

The paper demonstrates that repurposing instruction tuning for deep domain-knowledge injection—rather than mere task alignment—can close a large causal-reasoning gap in VLMs, yielding a 29.8-point AUROC gain over the strongest prior VLM baseline on Phys-AD. The central empirical finding is architectural: step-wise multi-turn delivery of structured priors is necessary, since single-turn injection fails to converge on more than a third of categories.