APT: Atomic Physical Transitions for Causal Video-Language Understanding

Abstract: Physical events are not understood by their names alone, but by the causal state changes that compose them. A clip-level label such as "bounce" can be correct while hiding the process that makes the event physically valid, from support loss and contact onset to rebound and settling. To make this hidden process explicit, we introduce Atomic Physical Transitions (APTs): minimal, temporally localized state changes that bind a visible cue to an active physical mechanism and before/after dynamical regimes. An APT chain represents a video as an ordered causal transition sequence rather than a single aggregate event label: event labels tell what happened; APT chains explain why it happened. To make APTs learnable by VLMs, we construct mixed-source APT data from human annotations and simulator ground truth, covering 14 transition types across contact, gravity, friction, and rotation/stability, with 27,303 timed instances over 1,246 trials. Using this data, we find that current VLMs miss transition-level physics, with zero-shot recall at most 14% and errors dominated by missed transitions. Direct fine-tuning on APT chains improves transition detection but causes event-level forgetting, indicating that the model learns a specialized answer format rather than a reusable physical representation. We therefore propose APT-Tune, a parameter-efficient recipe that teaches VLMs to use causal transitions without forgetting how to answer video questions. It combines image-pad-aware supervision, format-conditional co-training, and mechanism-conditioned domain-to-type decoding to make APT learning format-robust and physically grounded. With only 11 M LoRA parameters on Qwen3-VL-2B, APT-Tune substantially improves APT recall while also improving event-level video transfer. These results show that APTs are not a new answer format, but a human-aligned causal supervision signal for physical video understanding.

• The paper introduces APT, a methodology that decomposes video events into temporally localized, mechanism-typed causal transitions.
• It presents a hybrid pipeline combining human validation and LLM-generated templates to generate 27,303 transition labels from 1,246 trials.
• APT-Tune, a parameter-efficient LoRA adaptation, improves chain-level recall from 10–14% to up to 53% while preserving event-level performance.

Atomic Physical Transitions: A Framework for Causal Video-Language Understanding

Motivation and Formalization

The paper "APT: Atomic Physical Transitions for Causal Video-Language Understanding" (2606.18586) establishes a paradigm for video-LLMs (VLMs) to move beyond outcome-based event labels toward temporally structured, mechanism-typed causal representations. Current physical video benchmarks typically supervise and evaluate on clip-level targets—event names, answers, plausible outcomes—without explicating the causal intermediates that render events physically valid. The authors argue that such approaches elide the compositional structure intrinsic to physics: empirical studies show experts segment activity at meaningful state boundaries and reason in terms of physical mechanisms rather than surface features.

To address this, Atomic Physical Transitions (APTs) are defined as temporally localized, visually grounded state changes parameterized by an active mechanism and before/after dynamical regimes, decoupled from object or material class. Each APT is minimally atomic with respect to a taxonomy covering contact, gravity, friction, and rotation/stability (14 types in total). Videos are thus represented as ordered chains of APTs, each with a timestamp, mechanism domain, transition type, state change, and visual evidence—exposing not just what happened but why.

Figure 1: Distinction between outcome-level event labels and chain-level causal APT annotations; APT chains decompose clips into ordered mechanism-typed transitions.

This formalism bridges the gap between event recognition and full physical simulation, supporting annotation, evaluation, and simulation specification. By focusing on the causal process underlying an event, APTs provide a human-aligned supervision signal for transition-level physical understanding.

Benchmark Construction and Taxonomy

Acquisition of transition-level supervision is nontrivial: ambiguous, brief transitions demand physically informed annotation. The authors construct APT data using a hybrid pipeline: human-labeled anchors on CLEVRER and Physion++ validate semantics, while physically calibrated simulation with domain randomization, LLM-driven YAML templates, and VLM grounding generates scalable trials. Simulator traces—poses, velocities, contact pairs, and support relations—are parsed into realized APT chains, with human validation ensuring perceptual alignment.

The taxonomy encompasses 14 transition types across four domains:

• Contact/Collision: contact initiation/break, elastic rebound, inelastic capture
• Gravity/Projectile: free-fall onset, projectile apex, landing
• Friction/Surface: static breakaway, sliding arrest, surface transition
• Rotation/Stability: rotation onset/arrest, toppling, settling

Each transition is defined by domain, typed state change, and visual evidence, enabling domain-agnostic annotation and robust chain-level evaluation (see taxonomy expansion in Appendix).

Figure 2: Data construction pipeline—human anchors, LLM-driven scene proposals, domain-randomized simulation, VLM semantic grounding, and trace-based APT label extraction.

This pipeline yields 27,303 timed transition labels across 1,246 trials with full domain/type coverage. Metrics include recall (fraction of GT APTs detected within ±200 ms), type accuracy, and median timing error.

Diagnostic Evaluation and Failure Analysis

Initial evaluation of eight frontier VLMs (Qwen3-VL, InternVL3.5; scales 2B–8B; GPT-4.1; Gemini Flash) demonstrates that event-level competence does not transfer to transition-level causality. Zero-shot recall for APT chain recovery saturates at 10–14%; errors arise predominantly from missed transitions rather than timestamp localization. Increasing backbone scale yields negligible improvement, confirming a representational bottleneck rather than mere capacity shortfall.

Figure 3: Comparison of event-level scoring versus APT chain evaluation; APT exposes missed causal transitions not captured by outcome-level metrics.

APT-Tune: Format-Robust, Causal Fine-Tuning

Direct SFT on APT chains improves transition detection but induces specialist collapse—models overfit to the schema and forget event-level answering, invalidating broad transfer. To resolve this, the paper introduces APT-Tune, a parameter-efficient LoRA adaptation with two design principles:

1. Image-pad-aware masking: Multimodal SFT processors expand <image> into hundreds of <image_pad> tokens; loss computation is restricted to answer tokens, preventing pad-token reconstruction.
2. Format-conditional co-training: APT JSON, MCQ, and one-sentence description formats are mixed in training, preserving format generality and preventing APT-specialist collapse.

Mechanism-conditioned prompts (domain→type decoding) further encourage physical causality over label memorization.

Figure 4: APT-Tune: loss masking over answer tokens and format-conditional co-training to prevent APT-format specialization and preserve event-level competence.

APT-Tune, with only 11M LoRA parameters, raises recall to 38–53% (4x zero-shot baseline), scales with backbone size (scale-sensitive post-tune), and transfers both to event-level benchmarks (MVBench) and OOD physics datasets (PhysBench)—gains up to 18.2pp over baseline. Ablation shows both masking and format co-training are essential for format-robust improvement.

Practical and Theoretical Implications

The work has several salient implications:

• Diagnosis of physical reasoning: APT chain recovery exposes failures in causal reasoning missed by clip-level evaluation, providing a granular lens for VLM assessment.
• Training mechanism: Leveraging transition-level supervision enhances physically grounded representations and event-level generalization, illuminating the path toward format-robust, mechanism-aware multimodal models.
• Bridge to simulation and annotation: The taxonomy and mixed-source construction pipeline offer both annotation and simulation generation targets, enabling compositional, controllable dataset expansion.
• Broader impact: Potential applications include robotics, embodied AI, scientific video analysis, and safety-critical deployment where exposure of brittle causal reasoning is vital.

Future developments may:

• Expand taxonomy to cover fluid, granular, deformable, and non-rigid physics
• Integrate APT chains with object-centric simulation models for compositional physical abstraction
• Refine annotation/simulation pipelines for broader transfer to real-world and unstructured video domains

Conclusion

Atomic Physical Transitions introduce a systematic, mechanism-typed chain-level supervision unit for physical video-language modeling. Current VLMs fail to recover causal transitions in the absence of targeted supervision. APT-Tune enables efficient adaptation, yielding strong gains in both chain-level recall and transfer tasks without forfeiting event-level competence. The results substantiate APTs as a physically grounded, human-aligned causal representation for the next generation of video-LLMs.