Interaction Track Analysis

• Interaction tracks are spatial–temporal trajectories that capture the evolution and identity-preserving motion of subjects and objects during scene-specific interactions.
• They are extracted and verified using advanced methods such as GroundingDINO, SAM2, and LLM-based semantic parsing, ensuring high-fidelity mask tracks.
• Evaluation protocols like AAS, KISA, and SGI, combined with targeted regularization, enhance semantic grounding, temporal stability, and tracking accuracy.

An interaction track refers to a spatial–temporal trajectory capturing the evolution and identity-preserving motion of one or more instances (subjects, objects, or agents) as they engage in a scene-specific physical or semantic relationship over time. In contemporary computational contexts, interaction tracks form the basis for rigorous analysis, modeling, and evaluation of inter-instance dynamics—be it in video generation, multi-object tracking, simulation, testing, or physical measurement of interactive behaviors.

1. Formalization and Extraction of Interaction Tracks

Interaction tracks encode multi-instance relationships by delineating both the identity and physical boundaries of each subject and object, along with the event-specific interaction region. In modern datasets such as MATRIX-11K, every video sequence is annotated with:

• A set of instance IDs $K=\{k_1,\ldots,k_n\}$, each with a base noun class and spatial descriptor;
• A set of interaction triplets $$R=\{\langle k_\mathrm{sub}, \mathrm{verb}, k_\mathrm{obj}\rangle\}$$, extracted and verified with LLMs such as Llama 3 and GPT-4;
• Per-frame mask tracks $\{M_{k}\}$ for each $k$, extracted via GroundingDINO proposals, vision-language verification, and propagated with Segment-Anything 2 (SAM2);
• QC steps ensuring that mask tracks with drift or hallucination are removed, preserving high-fidelity correspondence (Jin et al., 8 Oct 2025).

This enables creation of mask tracks $M_{e}$ for each role $$e \in \{\mathrm{sub}, \mathrm{obj}, \mathrm{verb}\}$$, which serve as authoritative ground truth for subsequent attention or tracking analysis.

2. Interaction Tracks in Video Transformers: Semantic Grounding and Propagation

Video diffusion transformers (DiTs) encode cross-modal and temporal dependencies via multi-dimensional attention mechanisms. MATRIX systematizes the analysis along two axes:

• Semantic grounding (video-to-text attention): Attention maps $A^{v2t}$ from video patches to text tokens (nouns, verbs) are aggregated per role, e.g., $A^{v2t}_\mathrm{sub}$ for "subject" tokens. Effective grounding mandates $A^{v2t}_e$ concentrates on the corresponding mask track $M_e$.
• Semantic propagation (video-to-video attention): Query sets $Q_e$ are extracted from latent-grid locations covered by downsampled masks. The time-propagated attention $A^{v2v}_e$ is required to remain aligned with mask tracks $M_e$ over the entire sequence.

Alignment quality is measured by the Attention Alignment Score (AAS): $$AAS_e^{v2t} = \sum_{f,h,w} (A_e^{v2t} \odot m_e)(f,h,w)$$ (and analogously for $AAS_e^{v2v}$). Empirically, high AAS correlates strongly with successful semantic association and temporal stability (Jin et al., 8 Oct 2025).

3. Interaction Track Alignment via Regularization

MATRIX introduces a targeted regularization scheme to enforce spatial–temporal alignment between model attention and ground-truth interaction tracks:

• Layer selection: Identifies interaction-dominant layers $L_\mathrm{ground}$ (for video-to-text attention, e.g., layers 7 and 11) and $L_\mathrm{prop}$ (for video-to-video attention, e.g., layer 12).
• Attention map decoding: Attention maps $A_e^{v2t}$ and $A_e^{v2v}$ are upsampled via a causal decoder $D_\phi$ to pixel space, producing $\hat{A}_e^{v2t}, \hat{A}_e^{v2v}$.
• Loss composition: Composite loss over mask-track alignment combines BCE, Dice, and $\ell_2$ penalties: $$\ell(X,Y) = \beta_\mathrm{bce}\mathrm{BCE}(X,Y) + \beta_\mathrm{dice}(1-\mathrm{Dice}(X,Y)) + \beta_2 \|X-Y\|_2^2$$.
• Semantic Grounding Alignment (SGA): $$L_\mathrm{SGA} = \sum_{e}\ell(\hat{A}_e^{v2t}, M_e)$$ at $L_\mathrm{ground}$.
• Semantic Propagation Alignment (SPA): $$L_\mathrm{SPA} = \sum_{e}\ell(\hat{A}_e^{v2v}, M_e)$$ at $L_\mathrm{prop}$.
• Overall objective: $$L_\mathrm{total} = L_\mathrm{DM} + \lambda_\mathrm{SGA}L_\mathrm{SGA} + \lambda_\mathrm{SPA}L_\mathrm{SPA}$$; $L_\mathrm{DM}$ is standard diffusion loss; backbone weights remain frozen.

This regularization forcibly teaches the model to tightly bind the semantics of “who does what to whom” and maintain persistent attention alignment to the correct spatial regions, combating common DiT failures such as agent swapping, drift, and hallucination (Jin et al., 8 Oct 2025).

4. Evaluation Protocols for Interaction-Aware Video Generation

MATRIX introduces InterGenEval, a dedicated protocol for quantitatively assessing interaction tracks in generated video. Metrics include:

• KISA (Key Interaction Semantic Alignment): Proportion of correct responses to 6 stage-specific questions per triplet.
• SGI (Semantic Grounding Integrity): Accuracy of subject–object localization based on 4 yes/no questions.
• SPI (Semantic Propagation Integrity): Penalizes framewise emergence/disappearance events via mask coloring and GPT-5 assessment.
• Interaction Fidelity (IF): Defined as $(\mathrm{KISA} + \mathrm{SGI})/2$, integrating grounding, propagation, and key event correctness.

Systematic evaluation on both synthetic (60 cases) and real-domain (58 cases) prompts, with baselines from CogVideoX, Open-Sora, and TaVid, demonstrates superior interaction-fidelity, semantic alignment, and reduction in drift/hallucination when using interaction-aware tracks (Jin et al., 8 Oct 2025).

Method	KISA	SGI	IF	HA	MS	IQ
CogVideoX-2B	0.42	0.47	0.445	0.937	0.993	69.69
Open-Sora-11B	0.453	0.508	0.48	0.891	0.992	63.32
TaVid	0.465	0.522	0.494	0.917	0.991	68.90
MATRIX (Ours)	0.546	0.641	0.593	0.954	0.994	69.73

5. Broader Contexts: Tracking, Simulation, and Physical Interaction

Interaction tracks are foundational to diverse domains beyond generative modeling:

• Multi-object tracking (MOT): Approaches such as MLS-Track (Ma et al., 2024) and SAM-Track (Cheng et al., 2023) extend tracking with natural language queries, semantic modules, and multimodal cues to robustly follow instances specified by prompt, class, or user annotation.
• Physical simulation: In vehicle–track–structure interaction (VTSI) (Fedorova et al., 2024, Fedorova et al., 2024), interaction tracks manifest as dynamic trajectories coupled via kinematic constraints and Lagrange multipliers to enforce physical consistency (wheel–rail contact, object manipulation).
• Human-object/agent interaction: Recent hand-object interaction trackers (Shao et al., 2023) explicitly discover and temporally associate objects manipulated by hands, under severe occlusion and object switching, using geometric and learned spatio-temporal cues.
• Testing frameworks: Real-world Troublemaker (Zhang et al., 20 Feb 2025) leverages interaction tracks to orchestrate adversarial closed-loop scenarios for autonomous driving systems, using 5G-cloud-controlled vehicles and dynamic game-theoretic controllers to probe safety-critical multi-agent events.

6. Key Technical Challenges and Future Directions

• Annotation and dataset curation: Reliably extracting high-quality interaction tracks at scale requires multi-stage pipelines combining advanced detectors (e.g., GroundingDINO, SAM2), LLM-based semantic parsing, and rigorous human QC.
• Temporal identity preservation: Models must prevent identity drift, agent swapping, and hallucination both in generated video and physical tracks, implying explicit alignment mechanisms (mask loss, attention regularization).
• Cross-modal alignment: Ensuring that linguistic descriptions or prompts remain grounded in the spatial–temporal evolution of instance masks and their interactions is critical for semantic integrity, explainability, and usability.
• Scalability and efficiency: Selective layer-wise regularization (interaction-dominant layers) and frozen backbone finetuning reduce compute cost during adaptation, but scaling to more complex interactions, higher frame rates, or multi-agent systems remains a significant challenge.

7. Impact and Significance

The systematic modeling, enforcement, and evaluation of interaction tracks has transformed the capacity of generative, tracking, and simulation systems to reliably capture and reproduce multi-instance event semantics. MATRIX demonstrates that binding video transformer attention maps to precise mask tracks across selectively regularized layers not only improves interaction fidelity, but also yields sharper semantic alignment, stable temporal propagation, and robust resistance to common forms of failure—without degrading overall content quality (Jin et al., 8 Oct 2025). These principles increasingly underpin next-generation video generation, autonomous agent interaction, multi-object tracking, and physical simulation tools operating across a wide array of modalities and domains.