EntityBench for Consistent Video Generation

• EntityBench is a benchmark that defines long-range multi-shot video generation as an entity-consistency problem across visual narratives.
• It uses explicit per-shot schedules and a three-pillar evaluation suite to separate intra-shot quality, prompt alignment, and cross-shot consistency.
• The accompanying EntityMem baseline employs a persistent memory of per-entity visual references to improve identity continuity in generated videos.

EntityBench is a benchmark for entity-consistent long-range multi-shot video generation. It was introduced to address a specific evaluation gap in multi-shot generation: maintaining consistent characters, objects, and locations across shots over long sequences, under conditions where existing evaluations typically use independently generated prompt sets with limited entity coverage and simple consistency metrics. The benchmark comprises 140 episodes and 2,491 shots derived from real narrative media, includes explicit per-shot entity schedules, and is paired with a three-pillar evaluation suite and a fidelity gate that admits only accurate entity appearances into cross-shot scoring. It is released together with EntityMem, a memory-augmented baseline that stores verified per-entity visual references in a persistent memory bank before generation begins (He et al., 14 May 2026).

1. Scope and problem setting

EntityBench formalizes long-range multi-shot generation as an entity-consistency problem over coherent visual narratives. In this setting, the central difficulty is not only producing visually plausible individual shots, but preserving the visual identity of recurring characters, objects, and locations across shot boundaries and over long recurrence gaps. The benchmark therefore targets cross-shot recurrence explicitly rather than relying on isolated prompt-response evaluation (He et al., 14 May 2026).

The benchmark uses explicit per-shot schedules tracking characters, objects, and locations simultaneously. This design separates entity consistency from prompt construction artifacts by grounding evaluation in scheduled appearances rather than in loosely specified prompts. A notable property is the inclusion of long recurrence gaps spanning up to 48 shots, together with hard episodes of 50 shots each, which directly stress memory and persistence mechanisms in generation systems (He et al., 14 May 2026).

A common misconception is to treat multi-shot generation evaluation as a single scalar notion of “consistency.” EntityBench does not adopt that view. It disentangles intra-shot quality, prompt-following alignment, and cross-shot consistency into separate pillars, and then further restricts cross-shot scoring through a fidelity gate. This means that an entity must first be judged as an accurate appearance before it can contribute to cross-shot consistency metrics (He et al., 14 May 2026).

2. Corpus design and long-range entity scheduling

EntityBench contains 140 episodes totaling 2,491 shots and 1,136 scenes, sourced from real narrative media and refined by LLMs. Episodes are stratified into three difficulty tiers based solely on shot count, specifically to isolate long-range consistency challenges from per-shot complexity (He et al., 14 May 2026).

Tier	Episodes	Shot count
Easy	80	8–12 shots each (mean 10.0)
Medium	40	12–22 shots each (mean 17.8)
Hard	20	50 shots each (fixed length)

The entity composition is explicitly multi-type. Across the benchmark, the mean number of unique entities per episode is 7.05 ± 2.56 characters, 14.84 ± 9.52 objects, and 4.67 ± 1.33 locations. The mean per-shot entity counts are 2.00 characters, 1.61 objects, and 0.98 locations, for 4.59 total. Moreover, 79.1% of shots schedule at least one character, object, and location simultaneously, and 54.3% schedule at least two characters and at least one object simultaneously (He et al., 14 May 2026).

The recurrence structure is a central part of the benchmark’s difficulty. Across all recurring entities ($n = 2{,}026$), 36.1% have a maximum recurrence gap of at least 5 shots, 12.4% have a maximum recurrence gap of at least 10 shots, and the global maximum gap is 48 shots. The benchmark also defines a “memory test signal”: each $(\text{shot}, \text{entity})$ pair is either a first appearance, where a description is provided in the prompt, or a re-appearance that must rely on memory. Across the full benchmark there are 11,445 total appearances, of which 7,852 are re-appearances, corresponding to 68.6%. The re-appearance rate is 80.3% for characters, 73.4% for locations, and 52.9% for objects (He et al., 14 May 2026).

These statistics indicate that EntityBench is designed not merely as a prompt-following testbed, but as a recurrence-heavy benchmark in which the majority of appearances require some form of retained identity representation. A plausible implication is that methods lacking explicit memory or reference management will be systematically stressed by the benchmark’s schedule design.

3. Evaluation architecture and fidelity gating

EntityBench’s evaluation suite is organized into three pillars and uses the same canonical grounding crops for all of them. These crops are selected by a DINOv2+CLIP-gated detector, which makes the evaluation auditable and aligns the scoring pipeline across per-shot and cross-shot metrics (He et al., 14 May 2026).

The first pillar, intra-shot quality, contains 6 metrics. It adopts metrics from VBench with one metric dropped, background consistency. The retained measures are Subject Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree, Aesthetic Quality, and Imaging Quality. For a shot $S_k$ with frames $f_{k,1\ldots F_k}$, Subject Consistency is defined as

$$SC(S_k) = \frac{1}{F_k-1} \sum_{i=1}^{F_k-1} \langle \mathrm{DINO}(f_{k,i}),\,\mathrm{DINO}(f_{k,i+1})\rangle.$$

Episode-level aggregation takes the mean over shots (He et al., 14 May 2026).

The second pillar, intra-shot prompt-following alignment, contains 24 metrics. After detecting each scheduled entity $e$ in shot $k$ and extracting its canonical crop $c^*(k,e)$, the benchmark measures presence, per-entity fidelity, and action fidelity. Presence is computed separately for characters, objects, and locations. Per-entity fidelity uses an LLM to score each present or weak crop with type-specific criteria and an overall fidelity $\phi(k,e)$. Action fidelity uses a tiled $2 \times 3$ grid of 6 labelled frames and asks the LLM to judge overall depiction, subject identity, subject action, object interaction if any, and motion quality (He et al., 14 May 2026).

The third pillar, cross-shot consistency, contains 21 metrics. It is gated by fidelity: only $(\text{shot}, \text{entity})$0 pairs with $(\text{shot}, \text{entity})$1, or unresolved status, enter cross-shot scoring. The gated-in crops $(\text{shot}, \text{entity})$2 are then compared using two signal families. The first is DINOv2 embedding similarity, which computes a centroid and then cosine similarity of each crop to that centroid. The second is LLM pairwise judgment, which compares a centroid-closest anchor crop to the other crops for the same entity and returns binary sameness, a similarity score, and type-specific criteria (He et al., 14 May 2026).

All Pillar 3 metrics use a fidelity-gate–corrected mean that weights zero for gate failures and retains episode counts for coverage. This design is methodologically important because it prevents visually inconsistent or misdetected entity crops from inflating cross-shot scores. It also makes clear that high cross-shot similarity is not meaningful unless the underlying entity instance is itself a faithful realization of the prompt schedule (He et al., 14 May 2026).

4. EntityMem and explicit per-entity memory

EntityMem is the baseline system introduced together with EntityBench. Its central design choice is a persistent memory bank of per-entity visual and textual references assembled before video generation begins. For each entity, the stored visual references differ by type: characters use a single-character portrait on a chroma-key background, segmented and labeled with name; locations use a wide panorama cropped into left, center, and right angles; and objects, restricted to mobile props, use an isolated segmented portrait. The textual reference is the polished registry description from first appearance (He et al., 14 May 2026).

The system is structured as a three-stage multi-agent pipeline. In Stage 1, Entity Reference Generation, a Classification Agent decides which objects need references; a Portrait Agent writes style-aware text-to-image prompts, generates $(\text{shot}, \text{entity})$3 candidates, segments them, and selects the best via an LLM; and a Verification Agent checks segmentation or artifact failures and retries with an alternative key color if needed. In Stage 2, Keyframe Layout and Composition, a Layout Agent uses the shot action, schedule, and prior shot state for continuations to plan 1–3 keyframes with discrete positions on a 7-cell horizontal grid and to select a camera angle from front, left, or right. A compositor then places height-normalized portraits and object sprites onto the location background. In Stage 3, Memory-Augmented Generation, the system assembles memory inputs, injects textual references of recurring entities into the prompt, optionally supplies the previous shot’s last frame as first-frame conditioning for continuations, and runs the video backbone (He et al., 14 May 2026).

The memory interface is summarized by the benchmark in explicit notation:

$(\text{shot}, \text{entity})$4

$(\text{shot}, \text{entity})$5

$(\text{shot}, \text{entity})$6

These equations make explicit that the baseline is not merely conditioned on text. It is conditioned on typed entity references, keyframe layout information, and optionally the immediately preceding shot state. This suggests that EntityBench was designed to reward systems that represent entity identity as an explicit persistent object, rather than as an implicit consequence of prompt engineering alone (He et al., 14 May 2026).

5. Empirical findings and diagnostic conclusions

EntityBench evaluates four methods on representative metrics: EntityMem, StoryMem, HoloCine, and CineTrans. On fidelity-gate–corrected means, EntityMem attains the highest reported values for several character-centered and action-centered metrics. In Pillar 2, its face_fidelity is 0.740, compared with 0.452 for StoryMem, 0.349 for HoloCine, and 0.327 for CineTrans; its char_presence is 0.967, compared with 0.849, 0.882, and 0.796; and its action_overall is 0.618, compared with 0.547, 0.569, and 0.273. In Pillar 3 character LLM metrics, its llm_face_accuracy is 0.406 versus 0.226, 0.228, and 0.091, and its llm_face_mean_score is 0.426 versus 0.234, 0.242, and 0.145 (He et al., 14 May 2026).

The benchmark reports a more nuanced picture for DINOv2-based cross-shot similarity. EntityMem’s cs_face is 0.737, while StoryMem, HoloCine, and CineTrans score 0.792, 0.751, and 0.772 respectively. By contrast, EntityMem leads on cs_transition_boundary with 0.738, compared with 0.663, 0.498, and 0.508. The accompanying interpretation in the benchmark is that DINOv2 similarity shows little gap effect and may reward generic look-alike renderings rather than true identity (He et al., 14 May 2026).

Gap-decay analysis further supports the need for identity-sensitive evaluation. On llm_face_mean_score by recurrence gap, EntityMem records 0.744 for gaps of 1–2 shots, 0.698 for 3–5, 0.646 for 6–10, 0.669 for 11–20, and 0.657 for gaps of at least 21. HoloCine declines from 0.765 to 0.517, 0.614, and 0.420 across the reported intermediate bins, while CineTrans ranges from 0.371 to 0.408, 0.333, 0.600, and 0.457. The benchmark’s stated conclusion is that cross-shot entity consistency degrades sharply with recurrence distance in existing methods, whereas EntityMem maintains stable identity even at large gaps (He et al., 14 May 2026).

Effect-size analysis strengthens that conclusion. Against StoryMem as the strong baseline, EntityMem reaches Cohen’s $(\text{shot}, \text{entity})$7 with $(\text{shot}, \text{entity})$8 on intra-shot character fidelity, and $(\text{shot}, \text{entity})$9 with $S_k$0 on llm_face_accuracy. At the same time, the benchmark also reports that EntityMem trails on object LLM accuracy with $S_k$1, indicating that objects remain a challenge. Tier-stratified analysis shows that its advantage on character presence and action correctness grows with sequence length; for action_overall, $S_k$2 on the easy tier, $S_k$3 on the medium tier, and $S_k$4 on the hard tier (He et al., 14 May 2026).

These results support two methodological cautions. First, character consistency and object consistency are not interchangeable failure modes. Second, embedding similarity alone is insufficient as an entity-identity metric when generic visual resemblance can be scored favorably.

6. Position within entity-centered benchmark literature

The label “EntityBench” sits within a broader landscape of entity-centered benchmarks, but the underlying tasks differ substantially. In the benchmark literature represented here, ESBM is a human-annotated intrinsic benchmark for general-purpose RDF entity summarization, evaluating size-constrained subsets of RDF triples against multiple human ground truths (Liu et al., 2020). Machamp is presented as a benchmark for Generalized Entity Matching across relational, semi-structured, and textual data, with seven tasks spanning format and schema heterogeneity (Wang et al., 2021). A separate entity-centric evaluation framework for entity resolution emphasizes probability sampling of true clusters, summary-statistic monitoring, ratio-of-expectation estimators for performance metrics, and root-cause error analysis (Binette et al., 2024).

Other neighboring benchmarks target different entity problems altogether. BELB addresses biomedical entity linking across 11 corpora and 7 knowledge bases (Garda et al., 2023). IRC-Bench targets implicit entity recognition in first-person reminiscence narratives, where entity-identifying cues are distributed across multiple non-contiguous clauses (Aperstein et al., 7 May 2026). ERBench uses functional dependencies and foreign keys in relational databases to construct automatically verifiable hallucination benchmarks for LLMs (Oh et al., 2024).

This suggests that the 2026 EntityBench belongs to a wider family of benchmarks organized around entity identity, but its contribution is distinct in modality and failure model. Unlike summarization, matching, linking, or implicit recognition, it evaluates whether generated video can preserve visually consistent characters, objects, and locations over long multi-shot narratives under explicit recurrence schedules and fidelity-gated cross-shot scoring (He et al., 14 May 2026).