Entity-Centric Video NCT

• Entity-centric video NCT is a paradigm that decomposes video into discrete, trackable scene entities, enabling detailed temporal and semantic analysis.
• The methodology employs hierarchical grouping, slot attention, and contrastive losses to effectively align multi-grained visual features with language.
• This approach enhances video-language models by providing interpretable tracking and improved performance in action recognition, QA, and detail localization.

Entity-centric video NCT (Neural Compositional Tracking) refers to a paradigm in video understanding that structures and organizes video representation, tracking, and reasoning around discrete, dynamically evolving scene entities—such as persons, objects, or regions—over time. Rather than treating video as a flat grid of features or learning only global scene encodings, this approach explicitly discovers, composes, and temporally tracks entities, enabling interpretable, fine-grained alignment with language, robust multi-object tracking, and compositional reasoning about events and actions. The recent literature integrates this methodology with architectures built upon hierarchical grouping, contrastive multigranular objectives, dynamic relational graphs, and spatial-temporal slot attention to advance video-LLMs and tracking systems.

1. Conceptual Foundations and Motivation

Entity-centric video NCT is motivated by limitations in instance-level video-language alignment and conventional frame-level feature representations, particularly in first-person and egocentric contexts. Traditional models lack the compositional and temporally coherent structure aligned with human perception and cognition, often failing to capture how entities evolve, interact, or influence language narrations through time (Vo et al., 2024). NCT formalizes the discovery and tracking of multiple entities as the primary axis for structuring video understanding, supporting:

• Scene decomposition into temporally consistent entities.
• Fine-grained alignment between noun phrases, actions, and visual regions.
• Interpretability by explicit entity-level grounding.
• Compositional and multi-hop reasoning over action sequences and interactions.

2. Hierarchical Grouping and Slot Attention Mechanisms

The realization of entity-centric video NCT begins with the decomposition of video frames into a hierarchy of patch-, segment-, and entity-level representations. Methods such as HENASY (Vo et al., 2024) adopt a multi-stage framework:

1. Patch Tokenization: Each video clip is split into patch tokens across frames, mapped by a linear projection to a tensor $\mathbf{z}^0 \in \mathbb{R}^{T \times K \times D}$.
2. Bootstrapping via Cross-Attention: $G$ learnable group tokens serve as queries in a cross-attention operation, clustering patch tokens into $G$ temporally local segment tokens per frame.
3. Temporal-Aware Grouping: A Gumbel-softmax assignment mechanism compares the similarity between segment tokens and a second set of slots, ensuring temporal smoothing and coherence to yield $G'$ segment slots.
4. Entity Token Assembly: Another cross-attention + grouping cycle aggregates the $G'$ segment slots into $E$ entity tokens per frame, with decomposed space-time transformer (DST) blocks mediating inter-entity temporal interactions.
5. Entity-Aware Decoding: Entity tokens are fused with global context via cross-attention and multi-head blocks, resulting in a single video embedding.

This hierarchical slot attention yields temporally stable, interpretable, trackable entity slots, with their formation governed by differentiable group assignment and permutation-invariant pooling, as demonstrated in both supervised and self-supervised variants for tracking and segmentation tasks (Zhao et al., 2023).

3. Multi-Grained Contrastive Learning and Semantic Alignment

Central to effective entity-centric video NCT are multi-grained contrastive losses that align video, language, and entity representations:

• Global Video-Narration Contrast (EgoNCE): InfoNCE loss over video-caption pairs explicitly encourages alignment on both global and temporally local levels, using noun/verb overlap for positive set construction.
• Noun-Entity Contrastive Loss (NEC): Implements InfoNCE over noun taxonomy by matching ground-truth noun embeddings to discovered entities via the Hungarian algorithm, enforcing that each linguistic noun corresponds to a unique persistent visual entity.
• Verb-Entities Contrastive Loss (VEC): Embeds verb-phrases (from LLMs) as action descriptors, computing per-entity similarity and combining entity embeddings for contrastive matching to actions spanning multiple objects.
• Projection Loss for Weak Box Supervision: Weak external detections are matched via IoU with entity masks, and a Dice loss on projected masks provides auxiliary supervision for spatial precision.

Collectively, these objectives ensure that tracked entities correspond to semantically meaningful concepts and support direct, text-driven grounding and actions over time (Vo et al., 2024).

4. Extensions to Neural Compositional Tracking and High-Fidelity Tracking

Adapting hierarchical entity-centric architectures to NCT requires integration with tracking-specific objectives:

• Tracking Head: Each entity slot regresses frame-level bounding boxes, optimized via $\ell_1$ or GIoU loss against ground-truth trajectories.
• Temporal Smoothness Constraint: The assignment matrix $\alpha^{t}_{p,i}$ is regularized with a temporal difference penalty,

$$\mathcal{L}_{smooth} = \sum_{t=2}^{T} \|\alpha^{t}_{p,\cdot} - \alpha^{t-1}_{p,\cdot}\|_1 \quad \forall p,$$

to enforce slot identity over adjacent frames.

• Identity Stability Objective: A one-to-many contrastive loss between entity embeddings at $t$ and $t+1$ further encourages persistence and disambiguation under occlusion or visual drift.
• Semantic Grounding: Existing noun-entity and verb-entity contrastive heads are leveraged to couple tracklets with language categories, unifying detection, tracking, and semantic parsing within the same framework.

Object-centric methods such as OC-MOT (Zhao et al., 2023) extend slot-based pipelines with index-merge modules (multi-head attention for slot-to-buffer assignments) and object memory modules (autoregressive prototype rollout), enabling robust association and identity maintenance even with sparse mask supervision and no ID labels.

5. Entity-Relation Graph Representations and LLM Integration

Modern approaches generalize entity-centric NCT to handle extended temporal context and richer compositional reasoning by constructing dynamic entity-relation graphs. In GraphVideoAgent (Chu et al., 27 Jan 2025):

• Node Construction: Each entity node maintains a tuple of appearance times, visual features, textual captions, and state transitions; relations are encoded with type, phrase embedding, and timestamp.
• Graph Building: As new frames are processed, entity-relation edges are updated, with message passing propagating context through the graph.
• Reasoning Agent: An LLM agent prompts over the current entity graph and selected frames, choosing additional frames to sample and refining answers through iterative chain-of-thought reasoning guided by confidence assessments.
• NCT Query: Next-entity prediction is realized by querying which node is most likely to reappear post $t$, scored using recency and relation persistence metrics.

This explicit decomposition supports highly efficient frame selection (mean 8.2 per video) and improved accuracy on long-form QA and next-character tracking tasks, outperforming prior sequential or non-entity-centric models (Chu et al., 27 Jan 2025).

6. Evaluation Protocols and Empirical Insights

Entity-centric NCT systems are benchmarked with a mixture of zero-shot retrieval, classification, localization, and QA tasks:

• Downstream Performance: HENASY, equipped with multi-level contrastive objectives, maintains strong zero-shot transfer on tasks such as video/text retrieval, action recognition, multi-choice QA, and moment localization, with EgoMCQ inter-video accuracy at 94.1% and EpicKitchen-100 MIR mAP (V→T) of 35.5% (Vo et al., 2024).
• Ablation Studies: Removal of entity- or verb-centric components leads to 1–2% accuracy drops, and flattening the entity hierarchy degrades tracking and alignment quality.
• Empirical NCT Metrics: On EgoSchema and NExT-QA, entity-centric graph approaches achieve consistent improvements (≈2.2 points over baselines) with substantially reduced frame requirements (Chu et al., 27 Jan 2025).
• Object-centric Tracking: OC-MOT achieves IDF1 ≈88.6% and MOTA ≈82.4% on CATER with no ID labels, narrowing the gap with fully supervised methods (Zhao et al., 2023).

7. Future Directions and Limitations

The trajectory of entity-centric video NCT research highlights several frontiers:

• Scaling to Long-Horizon Streams: Approaches such as EGAgent (Rege et al., 26 Jan 2026) and GraphVideoAgent embed global, time-aware entity scene graphs for multi-day, multi-entity tracking and compositional reasoning, planning, and retrieval over longitudinal egocentric video with high accuracy (EgoLifeQA 57.5%) and efficient memory usage.
• End-to-End Learning and Joint Optimization: Currently, many systems decouple detection, tracking, and semantic reasoning; full end-to-end differentiable pipelines remain an open challenge, especially for integrating LLM-based planning and graph construction (Chu et al., 27 Jan 2025, Rege et al., 26 Jan 2026).
• Entity Identity and Temporal Coherence: Robustness to identity switches under occlusion, complex multi-object interactions, and fine-grained attribute changes is an ongoing area—entity-centric objectives, temporal smoothness, and contrastive task design are actively explored (Vo et al., 2024, Zhao et al., 2023, Ha et al., 3 Jan 2026).
• Richer Multi-Modal Integration: Future systems will require the fusion of audio, knowledge graph context, and multi-source perceptual streams, with benchmarks such as NarrativeTrack (Ha et al., 3 Jan 2026) exposing fundamental trade-offs between perceptual grounding and temporal reasoning.

A plausible implication is that further advances in NCT will hinge on tightening the fusion of entity-centric slot attention, dynamic interaction graph modeling, and LLM-driven compositional reasoning, guided by multigranular semantic alignment to language and robust, interpretable tracking objectives.