Entity Tracking (ET)

• Entity Tracking is a dynamic modeling task that continuously updates an entity’s state, attributes, and location as events unfold.
• It employs sequential inference techniques like BiLSTM-CRF and transformer-based models with structured inputs to enforce temporal consistency.
• Recent benchmarks and frameworks such as OpenPI2.0, MeeT, and ChaCAL underscore the benefits of explicit entity features in improving tracking accuracy.

Entity tracking (ET) denotes a family of problems in which a model must maintain coherent, evolving representations of entities as text, actions, or observations unfold over time. In natural language processing, ET commonly refers to monitoring how an entity’s state, attributes, or location change through a discourse or procedure; in this setting, ET is broader than mention grouping or coreference alone, because it requires updating entity-linked state under temporally ordered events and often under explicit structural constraints (Singh et al., 2022). Other literatures use closely related formulations: open-domain state tracking over entity–attribute–state tuples, multimodal state tracking across text and images, long-document memory-based tracking of entities and their attributes, and, in distinct research communities, “entity detection and tracking” or “extended target tracking” for document-level coreference or geometric object tracking (2305.14603).

1. Definitions and conceptual scope

In procedural-text ET, the input is a procedural text $P$ consisting of steps $\{s_1, s_2, \dots, s_T\}$ and a query entity $e$, and the goal is to predict the entity’s state and location at each timestamp of the procedure (Singh et al., 2022). This formulation emphasizes that ET is not just a sequence labeling problem: predictions must satisfy structural constraints over time, such as the requirement that an entity that has been destroyed should not later be moved (Singh et al., 2022).

Other textual formulations make the state representation more explicit. OpenPI2.0 represents each state change as a 4-tuple consisting of entity, attribute, pre-state, and post-state; the task is to predict, for each step in a procedure, which entities change, which attributes change, and the corresponding before/after values (2305.14603). TechTrack defines ET over technical procedures as identifying the value of each property of each entity at each step, with a distinction between persistent state-based properties and transient event-based properties (Goyal et al., 2021). In synthetic box-world settings, ET is formulated as inferring the final contents of a queried box from an initial description plus a sequence of Put, Remove, and Move operations (Kim et al., 2023).

These definitions converge on a common point: ET concerns dynamic world modeling rather than static reference resolution. The long-document perspective makes this explicit by defining ET as maintaining a coherent, evolving record of entities in a discourse, coupling three subproblems: introducing new entities, resolving later mentions back to earlier entities, and updating entity attributes or state as the narrative progresses (Toshniwal, 2022). MET-Bench extends the same idea to multimodal sequences by formalizing ET as sequential state estimation,

$S_T = f(S_0, A_1, A_2, \ldots, A_T),$

where intermediate actions may be textual or visual and the task is to infer the final state $S_T$ (Cohen et al., 15 Feb 2025).

2. Textual ET in procedures, open domains, and discourse

Procedural text has been a central testbed because it requires stepwise reasoning over creation, movement, destruction, persistence, and implicit transformations. In ProPara, a scientific process paragraph is annotated so that, for each participant entity, the model can answer whether the entity is created, moved, or destroyed, when this happens, and where it happens (Gupta et al., 2019). In Recipes, the task measures location changes of each ingredient throughout the recipe, and intermediate compositions make the problem harder because an ingredient may persist under a different surface form such as “mixture” or “batter” (Gupta et al., 2019).

Open-domain ET makes the ontology itself open-ended. KIEST frames the task as generating arbitrary numbers of state changes of the form “[attribute] of [entity] was [before_state] and [after_state] afterwards” from action descriptions, with entities, attributes, and states all drawn from open vocabularies (Li et al., 2023). OpenPI2.0 retains the original OpenPI procedures but adds canonicalized entities and attributes together with salience annotations, thereby making evaluation fairer and downstream reasoning more targeted (2305.14603).

Technical-procedure ET introduces a different kind of structure. TechTrack focuses on WikiHow procedures from the “Computers and Electronics” category, where entities are typed into categories such as hardware devices and software components and are tracked through properties such as isPowered, isConnected, isOpened, and isInstalled (Goyal et al., 2021). Because some properties persist and others reset after a step, the task requires explicit temporal propagation rather than one-step classification (Goyal et al., 2021).

Entity tracking also appears as a bottleneck in downstream comprehension. In cloze-style reading comprehension, errors on LAMBADA and CBT-NE were linked to failures to track speakers, repeated mentions, and entity-reference structure across a passage; adding explicit entity features and auxiliary entity-tracking objectives improved performance, especially on dialogue-heavy and entity-heavy cases (Hoang et al., 2018). This suggests that ET is not only a standalone benchmark problem but also a latent requirement for broader discourse understanding.

3. Modeling strategies and structured inference

A recurrent theme in ET modeling is the combination of local contextual encoding with explicit structure over time. NCET uses a contextual BiLSTM to encode the paragraph, an entity-specific recurrent state tracker, and a CRF over a 6-tag scheme $(O_B, O_A, C, D, E, M)$ to impose global consistency on discrete state trajectories (Gupta et al., 2019). The CRF is used to enforce impossibility constraints such as “an entity cannot be destroyed before it exists” and “an entity cannot move after being destroyed” (Gupta et al., 2019).

Transformer-based procedural ET initially showed that generic post-conditioning is insufficient. Lightweight GPT- and BERT-based approaches that encoded the process without making the target entity visible to the transformer underperformed even simple rule-based baselines on Recipes, whereas entity-conditioned input templates that place the target entity at the beginning or end of the input substantially improved performance (Gupta et al., 2019). The strongest variants treated the entity as part of the transformer’s intrinsic computation rather than as an after-the-fact classifier input (Gupta et al., 2019).

MeeT pushes this line further by reformulating ET subtasks as question answering for T5. State prediction becomes multi-choice QA, location prediction becomes extractive QA with cues such as “Other locations: none, unknown,” and decoding is separated into an offline CRF-style inference step with Viterbi decoding (Singh et al., 2022). Transition scores are initialized from training statistics, unseen transitions receive $-\inf$, and emission scores are reweighted according to whether the entity is explicitly mentioned in a step, reflecting the empirical finding that explicit steps are much easier than implicit ones (Singh et al., 2022).

Open-domain ET has also motivated retrieval-augmented generation. KIEST first retrieves candidate entities and attributes from ConceptNet, filters them with learned selectors, encodes the resulting entity–attribute subgraph with an RGCN, and then uses a Dynamic Knowledge Grained Encoder-Decoder in which attribute positions attend to attribute knowledge, entity positions attend to entity knowledge, and state words attend to the input text (Li et al., 2023). It further adds constrained decoding based on token relevance and a coherence reward derived from a classifier trained to distinguish coherent from corrupted state changes (Li et al., 2023).

For long documents, memory-based models replace mention-centric storage with compact entity-level memory. PeTra maintains a fixed number of memory cells, each with a content vector and a usage scalar, and updates these cells incrementally as new mentions are introduced, linked, or evicted (Toshniwal, 2022). The same thesis argues for integrating ET into LLMs directly through state-augmented training, so that state descriptions can be generated by prompting rather than recovered only through post hoc probes (Toshniwal, 2022).

4. Benchmarks, evaluation regimes, and empirical findings

The main procedural benchmarks reflect different views of ET. ProPara emphasizes document-level procedural understanding: the final score is the macro average of F1 over four questions concerning input entities, output entities, entity conversions with when/where, and entity movements with when/where (Singh et al., 2022). Recipes focuses on location changes of ingredients across recipe steps (Singh et al., 2022). OpenPI2.0 adds sub-tasks for schemata prediction, state prediction, and full-sentence generation, evaluated with exact-match F1 or accuracy and BERTScore (2305.14603). TechTrack evaluates positive state-change detection with Precision, Recall, and F1, concentrating on transitions such as a property becoming True (Goyal et al., 2021).

On ProPara and Recipes, MeeT reports state-of-the-art results without task-specific architecture design or in-domain pretraining. On ProPara test it achieves $P=80.3$, $R=67.1$, $F1=73.1$, slightly surpassing CGLI’s $\{s_1, s_2, \dots, s_T\}$0 F1; on Recipes test it reaches $\{s_1, s_2, \dots, s_T\}$1, $\{s_1, s_2, \dots, s_T\}$2, $\{s_1, s_2, \dots, s_T\}$3, outperforming LEMON’s $\{s_1, s_2, \dots, s_T\}$4 F1 by 4.9 points (Singh et al., 2022). Its analysis attributes part of this gain to multitask pretraining and to mention-guided decoding, motivated by a large explicit/implicit gap: sentence-level F1 is 70.2 on explicit steps versus 28.3 on implicit ones (Singh et al., 2022).

OpenPI2.0 instead shows that contemporary LMs remain far from competent under a fairer evaluation regime. On global schemata prediction, exact-match F1 ranges from 0.129 for LLaMA 65B to 0.362 for text-davinci-003, and local schemata prediction is harder still; even with gold entity-attribute pairs, state prediction accuracy remains limited, at 0.074 for gpt-3.5-turbo, 0.225 for text-davinci-003, and 0.102 for LLaMA 65B (2305.14603). The same benchmark, however, shows that salient entity state changes can be useful as a compressed reasoning trace: on CREPE, gpt-3.5-turbo improves from 0.348 dev / 0.362 test with direct prediction to 0.402 / 0.370 when provided with state changes for entities of salience $\{s_1, s_2, \dots, s_T\}$5, while using fewer entities per step (2305.14603).

TechTrack reveals a substantial human–model gap in realistic technical procedures. The dataset contains 1351 procedures, more than 1200 unique entities, and an average of 4.7 entities per procedure (Goyal et al., 2021). A BERT-based baseline generally outperforms ProLocal, and a combined BERT model improves especially sparse properties such as isConnected, isUsed, and isSetup; nevertheless, both baselines remain well below human annotation performance (Goyal et al., 2021). In reading comprehension, ET-oriented augmentations also translated into measurable task gains: on LAMBADA, AttSum-Feat + $\{s_1, s_2, \dots, s_T\}$6 reaches 59.23% test accuracy, improving over the base Attention Sum Reader and outperforming the previous state of the art reported in that table (Hoang et al., 2018).

5. LLMs, pretraining, and mechanistic accounts

A major line of work uses synthetic state-tracking tasks to isolate ET ability in LLMs. The box benchmark evaluates accuracy separately by the number of operations affecting a target box, rather than only by aggregate accuracy, because cases with zero affecting operations are trivial copying problems (Kim et al., 2024). In this setting, text-only pretraining does not reliably surface ET: Flan-T5 and GPT-3 davinci mostly repeat the initial state, whereas GPT-3.5 models exhibit non-trivial tracking, and the original study argues that pretraining on code is the critical difference (Kim et al., 2023).

Minimal-pair comparisons later strengthened that claim. Across Llama 2 $\{s_1, s_2, \dots, s_T\}$7 Code Llama, DeepSeek $\{s_1, s_2, \dots, s_T\}$8 DeepSeek-Coder, and Gemma $\{s_1, s_2, \dots, s_T\}$9 CodeGemma, continued training on code reliably improves ET, especially on cases with NumOps ≥ 1; by contrast, additional math training yields at best limited, inconsistent benefits, and alignment tuning helps base models more than code-pretrained models (Kim et al., 2024). The best alignment result reported there is Code Llama 70B Instruct with 64.9 accuracy on NumOps ≥ 1 (Kim et al., 2024).

Mechanistic studies complicate the behavioral picture. One line identifies a sparse four-group circuit—Structure Reader, Position Detector, Position Transmitter, and Value Fetcher—and argues that ET in Llama-style models is implemented by tracking the position of the correct entity rather than transporting object identity directly (Prakash et al., 2024). Fine-tuning on arithmetic or instruction data enlarges and improves this existing circuit rather than replacing it, with gains attributed primarily to better handling of augmented positional information (Prakash et al., 2024).

A later mechanistic account rejects incremental world-state maintenance altogether. In more realistic multi-operation settings, models do not incrementally track global states across tokens or query-relevant states across layers; instead, they consolidate relevant information in parallel at the last token when the query becomes evident (Tang et al., 28 May 2026). The same study reports that REMOVE is implemented with a fragile global suppression tag rather than a proper local update, predicting failure modes such as no-op removes, shared-label objects, and re-introductions of removed objects, some of which can be partially repaired by nullifying the identified tag (Tang et al., 28 May 2026).

A related study of swap-based dynamic tracking proposes a retrieval-conditioned rebinding circuit. Rather than rebuilding a fully updated world state after each swap, models preserve original bindings and use swap information at readout time to redirect retrieval to the object associated with the post-swap binding ID (Oh et al., 7 Jun 2026). Across Gemma and Llama families, this rebinding behavior is supported by compact attention-head circuits, but Gemma expresses the binding signature more clearly in query/key subspaces, whereas Llama carries it primarily in key vectors (Oh et al., 7 Jun 2026). Taken together, these results suggest that successful ET behavior in LLMs need not imply incremental, globally organized state computation.

6. Multimodal, long-horizon, and efficiency-oriented ET

Multimodal ET extends the problem from textual updates to mixed text–image sequences. MET-Bench evaluates final-state prediction in two structured domains, Chess and the Shell Game, with initial and final states represented in text and intermediate actions given either as text or images (Cohen et al., 15 Feb 2025). The main finding is a large performance gap between text-based and image-based tracking: for example, in Chess zero-shot evaluation, Claude 3.5 Sonnet scores 96.8% on text versus 66.2% on image, and GPT-4o scores 89.6% versus 73.3% (Cohen et al., 15 Feb 2025). The benchmark attributes this gap primarily to visual reasoning and state updating rather than to perception, because image-action classification is strong and cascaded inference that first converts images to text restores text-like performance (Cohen et al., 15 Feb 2025). Chain-of-thought prompting can help substantially—GPT-4o on Shell Game improves from 36.0% to 99.8% on text and from 32.4% to 84.2% on image—but substantial limitations remain, especially on long sequences (Cohen et al., 15 Feb 2025).

A separate research direction treats ET as a depth-efficiency problem for transformers. ChaCAL proves that, if each attention layer has receptive field 1 in the computational graph of state dependencies, then a transformer needs at least

$e$0

layers to solve the instance (Fagnou et al., 2024). It replaces one-hop attention with a resolvent-style operator,

$e$1

allowing multi-hop propagation within a single layer (Fagnou et al., 2024). On the toy ET dataset, ChaCAL reaches 100% accuracy with one layer, whereas standard transformers require 4–5 layers; on the advanced Boxes dataset, a 2-layer ChaCAL model reaches 99.1% exact match, outperforming deeper standard baselines (Fagnou et al., 2024).

Structured-sparse extensions then exploit the observation that, in ET settings, learned attention is mostly block-diagonal with a small cross-block residue (Zhao et al., 21 May 2026). Block-ChaCAL computes exact within-block propagation and routes cross-block interactions through a reduced system, achieving subquadratic sequence complexity $e$2 and matching the dense operator’s accuracy on controlled benchmarks while reducing wall-clock time by 12–29% (Zhao et al., 21 May 2026). A stated limitation is that performance collapses when the number of simultaneously evolving properties exceeds the number of attention heads (Zhao et al., 21 May 2026).

7. Other research traditions using the ET label

The abbreviation also appears in distinct tracking literatures with different observables and outputs. In document-level information extraction, “entity detection and tracking” (EDT) is the joint problem of finding entity mentions, assigning entity and mention types, and linking mentions into coreference chains (0907.0807). A joint LaSO-based model decodes left to right, makes detection and coreference decisions simultaneously, and benefits from non-local feature classes such as count-based, history-based, knowledge-based, and inference-based features (0907.0807). On ACE 2004, the reported joint model reaches 79.4 ACE, compared with 78.1 for a pipelined model (0907.0807).

In sensing and control, ET often means extended target tracking. A Gaussian Process based Extended Target Tracker represents an object contour as a star-convex radial function $e$3, uses a recursive GP approximation inside an EKF over joint kinematic and contour state, and passes the recovered contour to a neural classifier built on six geometric shape features (Tuncer et al., 2020). In an ISAC setting, ISACTrackNet instead estimates extended target state directly from communication echoes using a denoising module, an encoder for instantaneous coarse state estimation, and a KalmanNet refinement module under a constant-velocity model (Wang et al., 1 Apr 2025).

Computer-vision ET in industrial environments provides yet another formulation. The TOMIE framework supports continuous tracking of industrial entities such as pallets, crates, barrels, and forklifts over six RGB cameras, using a 52-camera motion-capture system to create semi-automated annotations (Rutinowski et al., 2023). The resulting TOMIE dataset contains 112,860 frames and 640,936 entity instances, and standard MOT methods such as ByteTrack, Bot-SORT, and SiamMOT achieve proof-of-concept results, with Bot-SORT performing best on the reported validation metrics (Rutinowski et al., 2023).

Across these traditions, ET consistently names problems in which identity must be preserved while state, attributes, location, or extent evolve. What differs is the representational substrate: symbolic state tuples in procedural text, latent memory cells in long-document LLMs, multimodal action sequences in vision-language benchmarks, mention chains in EDT, and geometric contours or multi-camera trajectories in tracking systems.