EntityMem: Entity-Centric Memory Systems

• EntityMem is an entity-centric memory paradigm that organizes persistent state around discrete entities rather than tokens or documents.
• It underpins diverse systems across video generation, language modeling, and customer data explainability by storing verified visual and textual references.
• Evaluations reveal enhanced identity preservation, efficient token usage, and improved long-range consistency in multi-modal tasks.

EntityMem denotes an entity-centric memory paradigm in which persistent state is organized around entities rather than around undifferentiated tokens, frames, or documents. In the literature surveyed here, the name refers most directly to a memory-augmented multi-shot video generation system that stores verified per-entity visual references in a persistent memory bank before generation begins (He et al., 14 May 2026). Closely related systems instantiate the same design principle as an “entity-indexed bank of latent patches” for video diffusion (Vandersanden et al., 22 May 2026), as dynamically updated vectors for discourse entities in neural language modeling (Ji et al., 2017), as a large latent entity table in encoder-decoder generation (Zhang et al., 2022), as graph-structured entity clusters in Customer 360 explainability (Jaitly et al., 2022), and as persistent entity embeddings learned from semi-structured text without constructing a traditional knowledge graph (Gunaratna et al., 2021). This suggests a broader technical category: explicit entity memory mechanisms that store, retrieve, update, and explain entity state across time, modalities, and tasks.

1. Core abstraction and scope

The central abstraction is a memory indexed by entity identity. In the EntityBench baseline, memory is a per-episode bank of “verified per-entity references” containing visual references and textual descriptions for characters, selected objects, and locations, and it is used to condition shot generation throughout an episode (He et al., 14 May 2026). In EM-Vid, the memory is an entity-indexed collection of latent visual patches, with separate entries for characters, objects, and scenes (Vandersanden et al., 22 May 2026). In EntityNLM, each coreference cluster receives its own persistent vector, updated only when that entity is mentioned (Ji et al., 2017). In EDMem, the memory is a dense table of entity embeddings pre-trained on Wikipedia and queried from an encoder-decoder model at entity mention boundaries (Zhang et al., 2022). In xEM, the underlying “entity memory” is a graph of records linked into connected components by a probabilistic matching engine, with a GNN surrogate layered on top for explanation (Jaitly et al., 2022).

A useful technical distinction runs between explicit and implicit memory. PromptEM explicitly states that it “doesn’t maintain an explicit external memory table” and instead uses “implicit memory within the LM,” with continuous prompts acting as task-specific query patterns into that parametric store (Wang et al., 2022). UMEM occupies a different point in the design space: it treats an external memory bank $B = \{(k_i, v_i)\}_{i=1}^n$ as the evolvable parameter set of an LLM-based agent, making memory content itself the object of optimization (Ye et al., 11 Feb 2026).

System	Memory form	Domain
EntityMem	persistent per-entity memory bank	multi-shot video generation
EM-Vid	entity-indexed bank of latent patches	multi-shot video generation
EntityNLM	vector per entity	neural language modeling
EDMem	latent entity representations in memory	encoder-decoder QA and generation
xEM	entity graph / connected components	Customer 360 entity matching
ECG	persistent entity embeddings	semi-structured web text
UMEM	external memory bank of key–value pairs	LLM-based agents

2. Representational forms and memory addressing

EntityMem systems differ primarily in what they store and how they are addressed. The EntityBench baseline stores segmented character portraits with name labels, panoramic location images cropped into left, center, and right angle variants, selected object references, cleaned textual descriptions, and per-shot keyframes composed from those references (He et al., 14 May 2026). EM-Vid stores VAE-space latent patches associated with entity masks, spatial coordinates, source-frame indices, DINOv2 region embeddings, CLIP region-text similarity scores, and token counts for each candidate memory entry; memory lookup is driven by a structured script whose abstract_prompt explicitly names entity IDs such as [CH_01], [OB_01], and [SC_01] (Vandersanden et al., 22 May 2026).

In sequence models, memory addressing is typically mention-triggered. EDMem marks mentions with $[E_s]$ and $[E_e]$, then uses the hidden state at the start marker to attend over a 1M-entity memory:

$$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$

$$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$

The resulting entity vector is merged back into the encoder-decoder stack and also supports an entity-linking loss (Zhang et al., 2022). EntityNLM uses a more explicitly structured latent state: $R_t$ marks whether the current token is inside a mention, $E_t$ identifies the entity, and $L_t$ gives the remaining mention length, so the model can allocate and revisit entity-specific memory slots during generation (Ji et al., 2017).

Graph-based formulations represent entity memory as topology plus features. xEM converts relational Customer 360 data into a record graph in which each record becomes a node, edges connect matched records within an entity, and entities are connected components, often organized around a representative record (Jaitly et al., 2022). ECG similarly treats entities as persistent nodes, but places free-form textual relationship contexts on edges rather than relying on a fixed relation vocabulary, allowing the learned embedding space to function as a KG-free entity memory (Gunaratna et al., 2021).

3. Update, maintenance, and management

A defining property of EntityMem systems is whether memory is static, incrementally updated, or jointly optimized with its controller. The EntityBench baseline constructs its memory bank before generation begins and then reuses that bank across the episode; consistency comes from persistent, quality-verified references rather than from updating memory with generated outputs (He et al., 14 May 2026). This choice is explicitly motivated by avoiding error accumulation from noisy generated frames.

EM-Vid adopts a fully incremental strategy. After each shot, it extracts candidate entity entries from selected keyframes, evaluates them with DINOv2 similarity thresholds $T_{\min}^{\text{match}}$ and $T_{\text{redundant}}$, and prunes them under a per-entity budget $[E_s]$0 using the relevance-to-cost score

$[E_s]$1

with the first accepted entry never removed (Vandersanden et al., 22 May 2026). This yields a compact, evolving memory that accumulates new views while bounding token growth.

EntityNLM updates memory through a gated recurrence. When entity $[E_s]$2 is mentioned, its state is revised as

$[E_s]$3

so only the active entity slot is updated, while all other entity vectors remain unchanged (Ji et al., 2017). This is a sparse-update memory regime aligned with coreference structure.

UMEM makes update policy itself trainable. Its Mem-Optimizer emits both memory content and an operation such as ADD or UPDATE index, and these are jointly optimized with GRPO under a neighborhood-level marginal utility reward defined over semantically related queries (Ye et al., 11 Feb 2026). xEM, by contrast, trains in batch mode on the outputs of a Probabilistic Matching Engine and uses the learned GNN only as a surrogate explainer, not as the mechanism that updates the core Customer 360 entity clusters (Jaitly et al., 2022).

4. Functional roles across modalities

In multi-shot video generation, EntityMem serves as an identity-preserving conditioning substrate. The EntityBench baseline uses portraits, panoramas, descriptions, and keyframe composites to condition a StoryMem-like two-stage backbone, with continuation shots additionally receiving the previous shot’s last frame as a temporal seed (He et al., 14 May 2026). EM-Vid pushes the same idea into latent space: only entity-relevant memory tokens are retained for DiT attention, and selective noise injection is used to suppress unwanted background or outdated local attributes, preserving shot-specific prompt controllability (Vandersanden et al., 22 May 2026).

In text generation and question answering, EDMem uses entity memory to enrich both encoding and decoding. The memory is pre-trained jointly with the encoder-decoder on Wikipedia, and three decoding regimes—free-form generation, static entity linking, and dynamic entity linking—control the precision-flexibility trade-off for entity names (Zhang et al., 2022). EntityNLM uses entity memory more directly as discourse state: the current entity vector biases next-token prediction, while $[E_s]$4 and $[E_s]$5 support explicit modeling of coreference and mention spans (Ji et al., 2017).

In entity matching, memory is often structural rather than lexical. xEM treats Customer 360 identity resolution as a graph whose connected components are entities, then explains why a record belongs to an entity by explaining pairwise relations within that cluster through GNN feature masking (Jaitly et al., 2022). PromptEM addresses low-resource generalized entity matching from a different angle: it serializes heterogeneous entities into a schema-agnostic text format and uses prompt-tuning plus self-training, which the paper characterizes as a form of “implicit memory within the LM” rather than an external entity store (Wang et al., 2022).

In representation learning from semi-structured sources, ECG defines an Entity Context Graph in which entity-centric documents generate context triples $[E_s]$6 with free-form textual relation spans, and a TransE-style learner with a CNN relation encoder turns these into persistent entity embeddings usable for classification, link prediction, and KG augmentation (Gunaratna et al., 2021). This suggests that entity memory can be realized either as a runtime retrieval structure or as a learned embedding space that persists across tasks.

5. Empirical behavior and evaluation

Empirical results show that explicit entity memory is most visible when identity persistence, entity precision, or long-range reuse is the evaluation target. On EntityBench, the EntityMem baseline achieves character presence $[E_s]$7, face fidelity $[E_s]$8, and cross-shot llm_face_accuracy $[E_s]$9, compared with StoryMem’s $[E_e]$0, $[E_e]$1, and $[E_e]$2; its long-range LLM-based identity similarity remains $[E_e]$3 even for recurrence gaps of $[E_e]$4–$[E_e]$5 shots (He et al., 14 May 2026). The same benchmark also shows that raw visual quality metrics need not track entity consistency: CineTrans leads imaging quality at $[E_e]$6, while EntityMem ranks second at $[E_e]$7, yet EntityMem is markedly stronger on identity-specific measures (He et al., 14 May 2026).

EM-Vid reports a complementary efficiency-consistency profile. By restricting conditioning to entity-relevant tokens, it uses up to $[E_e]$8 fewer memory tokens and achieves up to $[E_e]$9 speedup, with per-shot inference $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$0–$$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$1 faster than dense full-frame memory. It also records the best ViCLIP score $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$2, CSCCLIP $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$3 versus StoryMem’s $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$4, and BGACLIP $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$5 versus StoryMem’s $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$6 (Vandersanden et al., 22 May 2026).

In language modeling, explicit entity state improves both token prediction and downstream structure prediction. EntityNLM reduces CoNLL-2012 test perplexity to $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$7 from $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$8 for an LSTM RNNLM, improves a strong coreference reranker from $$\alpha_i = \frac{\exp\big(\mathbf{e}_i^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)} {\sum_{j=1}^N \exp\big(\mathbf{e}_j^\top \mathbf{W}_{in} \mathbf{h}^{low}_s\big)}$$9 to $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$0 CoNLL F1, and reaches $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$1 on entity prediction on InScript, compared with $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$2 for Modi and Titov’s model and $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$3 for human prediction (Ji et al., 2017).

In encoder-decoder QA, EDMem’s memory-guided decoding is strongest when answers are entities. On TriviaQA in-house, static entity linking reaches $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$4 EM, compared with $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$5 for EDMem free-form, $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$6 for EncMem, and $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$7 for BART; on WebQuestions it reaches $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$8 EM, and on Natural Questions the free-form variant reaches $$\mathbf{h}^{ent}_s = \mathbf{W}_{out} \Big( \sum_{i=1}^N \alpha_i \cdot \mathbf{e}_i \Big)$$9 EM (Zhang et al., 2022). For entity-intensive generation, dynamic linking substantially improves entity coverage: on ELI5, total entity coverage rises from $R_t$0 for BART to $R_t$1, and unseen entity coverage from $R_t$2 to $R_t$3 (Zhang et al., 2022).

ECG shows that persistent entity memories learned from semi-structured text can be competitive with KG-based alternatives. On FB15K ECG link prediction it reports Hits@10 of approximately $R_t$4, and in joint learning with TransE on FB15K the KG+ECG configuration reaches Hits@10 $R_t$5, compared with $R_t$6 for TransE alone (Gunaratna et al., 2021). PromptEM, although not an explicit external EntityMem system, demonstrates that low-resource generalized entity matching can benefit from prompt-based implicit memory: it is SOTA on $R_t$7 datasets in the default low-resource setting, with REL-HETER F1 $R_t$8 and SEMI-REL F1 $R_t$9 (Wang et al., 2022). UMEM extends the evaluation story from entity memories to self-evolving memory controllers, reporting up to a $E_t$0 improvement in multi-turn interactive tasks together with a monotonic cumulative growth curve during continuous evolution (Ye et al., 11 Feb 2026).

6. Limitations and open directions

Current EntityMem designs expose recurring limitations. The EntityBench baseline improves characters strongly but is weaker on objects: StoryMem surpasses it on llm_object_accuracy $E_t$1 versus $E_t$2, and the paper attributes this to “sticker-like” object composition and the absence of end-to-end training of memory usage (He et al., 14 May 2026). EM-Vid remains training-free and therefore depends on a wrapper around pretrained M2V models rather than a jointly learned memory interface; its more aggressive sparse patchify/unpatchify variant degraded quality because of train-test mismatch and floating-point accumulation (Vandersanden et al., 22 May 2026).

Text-centric systems face different bottlenecks. EDMem is constrained by a fixed 1M-entity Wikipedia vocabulary, by domain specificity, and by the absence of explicit symbolic reasoning over stored facts (Zhang et al., 2022). EntityNLM requires annotated entities and coreference chains during training, scales linearly with the number of entities seen so far in a document, and relies on an LSTM architecture rather than a Transformer (Ji et al., 2017). xEM explains pairwise relations inside already formed clusters but leaves “graph clustering explanations” and “neuro-symbolic evaluation of explanations” as future work, while training remains batch-mode rather than incremental (Jaitly et al., 2022).

Representation-learning and agentic memory systems surface still other issues. ECG depends on entity detection quality, often uses coarse paragraph-level relation extraction, and assumes a single primary entity per document, which complicates transfer to multi-primary sources such as news (Gunaratna et al., 2021). UMEM currently stores memories mainly as natural-language text, does not yet implement richer structured schemas, and does not deeply analyze very large memory-bank scaling (Ye et al., 11 Feb 2026). PromptEM highlights an adjacent limitation: prompt-based implicit memory works well in low-resource generalized entity matching, but numeric-heavy settings such as SEMI-HETER can still favor non-PLM alternatives like TDmatch (Wang et al., 2022).

Across these systems, a plausible synthesis is that the main unresolved problem is not whether to use memory, but how to make entity memory simultaneously explicit, compact, updateable, faithful, and controllable. The literature already supplies the key ingredients—entity-indexed storage, sparse retrieval, verified references, budgeted updates, local explanations, and neighborhood-level utility optimization—yet no single architecture currently unifies all of them.