PhysAgent: Modular AI for Physical Reasoning & Therapy

• PhysAgent is a modular AI system that integrates specialized vision-language modules, physics knowledge, and agentic orchestration for enhanced physical reasoning and personalized therapy.
• In physical reasoning, it wraps pre-trained vision models with physics rule retrieval and chain-of-thought prompt integration to boost performance on structured tasks.
• In physiotherapy, it employs a multi-agent system for real-time video synthesis, pose estimation, and deterministic feedback, ensuring patient-specific safety and efficacy.

PhysAgent denotes two distinct but convergent strands in AI-driven physical systems: (1) an agentic multi-module wrapper to enhance the physical world understanding capabilities of state-of-the-art vision-LLMs in visual reasoning (primarily in the context of PhysBench) (Chow et al., 27 Jan 2025), and (2) a multi-agent framework for personalized physiotherapy that leverages generative AI, computer vision, and agentic autonomy for real-time patient-specific feedback (Dharmaratnakar et al., 22 Apr 2026). Both integrate heterogenous AI modules with specialized roles, operate in closed or partially closed feedback loops, and seek to bridge perceptual, reasoning, and domain knowledge gaps inherent in single-model systems. Below, the various incarnations and applications of the PhysAgent paradigm are set out systematically.

1. Conceptual Overview and Definitions

PhysAgent (in the context of physical world understanding) is a modular, zero-shot augmentation framework designed to enhance the performance of vision-LLMs (VLMs) such as GPT-4o on physical reasoning and physics-grounded perception tasks. It acts as a “plug-in” architecture, inserting specialized perceptual and knowledge components between input stimuli (image/video/text) and VLM processing, with no end-to-end retraining. In the context of personalized physiotherapy, PhysAgent denotes a multi-agent system (MAS) architecture comprising cooperating agents responsible for converting clinical intent into actionable personalized multimedia content and closed-loop feedback (Dharmaratnakar et al., 22 Apr 2026). The term may also be encountered as a generic descriptor for agentic architectures catalyzing the physicalization of language-based reasoning, including, for example, the MadAgents paradigm in particle-physics workflows (Plehn et al., 28 Jan 2026).

2. Architecture and Core Components

PhysAgent for Physical World Reasoning (Chow et al., 27 Jan 2025)

The PhysAgent architecture wraps an off-the-shelf VLM with:

• Vision Foundation Models: Offloaded perception, including Depth-Anything v2 (monocular depth estimation), SAM (segmentation), and GroundingDINO (open-set object detection).
• Physics Knowledge Memory: A small retrievable database of salient physical rules (task-triggered).
• Task-Specific Prompt Activation: Automatic classification of each incoming QA/image task into one of 19 physics subtypes; retrieval of a minimal physics fact.
• Chain-of-Thought Integration: Sequential composition of the user's text query, foundation model outputs, and retrieved rule into a composite prompt, supplied to the VLM for stepwise reasoning.

The process flow is:

Question/Image → Prompt Classifier → (Retrieve Physics Rule) ↓ Foundation Models → f_fm(I) ↓ Aggregate: [Question; Physics Rule; f_fm(I)] → VLM → Answer

There is no compositional embedding; the pipeline is realized entirely through modular prompt engineering and separate inference through the respective models.

PhysAgent for Personalized Physiotherapy (Dharmaratnakar et al., 22 Apr 2026)

The physiotherapy PhysAgent instantiates as a closed-loop MAS with the following micro-agents:

• Clinical Extraction Agent: Converts unstructured clinical prescriptions into JSON kinematic/safety constraints via an LLM or NER-based parser.
• Video Synthesis Agent: Invokes a patient-specific video generation model (e.g., video diffusion/transformer with constraint injection) to produce precise, ROM-constrained instructional media.
• Vision Processing Agent: Performs continuous pose estimation from real-time RGB streams (MediaPipe Pose or YOLO-Pose), extracting joint angles at ≥30 FPS within a latency of ~28 ms/frame and mean angular RMSE ≈ 3.2°.
• Diagnostic Feedback Agent: Applies constraint-logic matching to live pose data, issuing real-time, safety-validated, context-sensitive feedback (stop, encouragement, or correction) via deterministic templates and, optionally, paraphrased for empathy by a local LLM.

Pseudocode (condensed) for the online loop:

1. S.pose ← VisionAgent(RealtimeFrame) 2. θ_current ← ExtractJointAngle(S.pose) 3. Compare θ_current vs θ_target ± δ → C_t 4. Output feedback C_t; S.feedback ← C_t

Shared state object S persists and synchronizes across all modules; all feedback is explainable and audit-logged.

3. Technical Workflow and Algorithms

PhysAgent (Physical World Reasoning)

Prompt Engineering, No Finetuning: PhysAgent does not require retraining of backbone models. Task-relevant physics facts are hand-defined and retrieved at inference, and the outputs of perception models are formatted and directly concatenated to the VLM’s prompt.

Vision Models Integration: Foundation models are called as zero-shot modules; their outputs (e.g., bounding boxes, masks, depth maps) are summarized in text form for the VLM.

Chain-of-Thought (CoT) Prompting: For each query, the assembled prompt enjoins the VLM to “think step by step,” incorporating retrieved rules and structured vision outputs. Experimental ablation demonstrates that this composition outperforms both descriptive CoT and oracle neuro-symbolic baselines (Chow et al., 27 Jan 2025).

PhysAgent (Personalized Physiotherapy)

• Constraint Extraction: Uses NER or LLM-based parsing to extract schema elements (joint, axis, limits) and formalizes kinematic constraints ($θ_{min} ≤ θ ≤ θ_{max}$). Outputs are stored JSONs for downstream agents.
• Personalized Video Synthesis: Conditions a video diffusion/transformer generative model on extracted constraints by prompt and cross-attention injection; enforces strict ROM adherence via explicit penalty in the loss (e.g., $$L_{ROM} = λ · \max(0, θ_{gen_{max}} – θ_{target})^2$$), in addition to pixel reconstruction and temporal consistency terms.
• Real-Time Pose Estimation: Computes 2D keypoints, forms joint vectors, and calculates angles using $θ = \arccos((v_1 · v_2)/(\|v_1\|\|v_2\|))$. Smoothing is optionally applied (e.g., Kalman filter). Targets <50 ms processing per frame.
• Feedback Logic: Deterministic rules based on deviation thresholds ($δ_{crit}$, $δ_{opt}$, $δ_{under}$) decide between stop, praise, or correction; all feedback messages are fully explainable.

4. Empirical Performance and Benchmarking

PhysBench Results (Chow et al., 27 Jan 2025)

PhysAgent (wrapped around GPT-4o) evaluated on the 10,002-question PhysBench suite achieves the following top-1 multiple-choice accuracy boosts over baseline GPT-4o:

• Object Property: 56.9% → 58.4% (+1.5 pp)
• Object Relationships: 64.8% → 84.2% (+19.4 pp)
• Scene Understanding: 30.1% → 45.0% (+14.9 pp; +49.5%)
• Physics-based Dynamics: 46.9% → 51.3% (+4.4 pp)
• Overall: 49.49% → 59.73% (+10.24 pp; +20.7%)

The “18.4% improvement” refers to the overall relative gain across categories. Prompting baselines and oracle neuro-symbolic pipelines either fail to match or degrade performance.

Physiotherapy Agent Clinical Pipeline (Dharmaratnakar et al., 22 Apr 2026)

Bench and planned clinical evaluations:

• Bench (Healthy Volunteers): Vision agent angle estimates vs IMU ground-truth, RMSE ≈ 3.2°, latency ≈ 28 ms/frame.
• Pilot RCT: N ≈ 30, comparing adherence (session count), ROM improvement, patient-reported pain/satisfaction, and logged safety events between PhysAgent and standard video arms.
• Scaled Trial: Multi-site, N > 200, targeting statistical significance in functional recovery improvements.
• Safety Protocols: Critical violations ($θ_{current} > θ_{max} + δ_{crit}$) trigger hard stops; all decisions are explainable and logged.

5. Error Analysis, Limitations, and Future Directions

• Dominant Error Modes in PhysAgent (Visual Reasoning): 37–45% from perceptual failures of foundation models (e.g., depth misestimation, segmentation errors); 23–35% from physics knowledge gaps; the remainder due to VLM reasoning breakdowns.
• Knowledge Memory: Currently handcrafted and limited; scaling via automated extraction from domain sources is an open direction.
• No End-to-End Training: Both PhysAgent variants rely on modular inference and prompt engineering rather than differentiable, holistic learning.
• Extension Potential: The underpinning paradigm (agentic composition, module orchestration, memory retrieval, explainable outputs) generalizes to other physical and simulation domains—see MadAgents (Plehn et al., 28 Jan 2026) for high-energy physics, or analogous principles in PhysicsAgentABM (Venkatesh et al., 5 Feb 2026) and the physiotherapy system.

6. Related Paradigms and Broader Impact

The agentic, modular architecture underlying PhysAgent is mirrored in:

• Computational Physics and Simulation: MadAgents leverages specialized agents (MadGraph interface, scripting, plotting, PDF parsing) under an orchestrator for particle-physics event generation and analysis (Plehn et al., 28 Jan 2026).
• Health and Personal Assistance: Multi-agent personal health agents (PHA) adopt similar decompositions—data-science, domain-expert, and coaching sub-agents (Heydari et al., 27 Aug 2025).
• Agent-Based and Neuro-symbolic Modeling: PhysicsAgentABM demonstrates robust neuro-symbolic agent clustering, mechanistic reasoning, and data-driven fusion for population-level state transition simulation (Venkatesh et al., 5 Feb 2026).

The unifying trend is toward distributed expert modules with explicit interfaces, shared memory, and orchestrated decision flow, with explicit performance, safety, and explainability constraints.

7. Significance and Open Challenges

PhysAgent, as instantiated in both the vision-language and physiotherapy contexts, evidences that closed-loop orchestration of specialist AI modules—rather than monolithic model scaling—can yield substantial, domain-grounded gains in physical-world reasoning and embodiment. Open challenges include end-to-end differentiable integration, scaling and automating knowledge memory, adaptation to non-multiple-choice and open-ended tasks, and real-world robustness beyond simulation or camera-based setups. The efficacy of prompt engineering, retrieval-augmented pipelines, and deterministic safety controllers in agentic systems is domain-dependent and remains an active area of research.

---

References: (Chow et al., 27 Jan 2025, Dharmaratnakar et al., 22 Apr 2026, Plehn et al., 28 Jan 2026, Venkatesh et al., 5 Feb 2026, Heydari et al., 27 Aug 2025).