CogniGPT: Framework for Long Video Understanding

• CogniGPT is a framework for long video understanding modeled on human progressive visual cognition using minimal, task-relevant clues.
• It employs a dual-agent approach: MGPA for multi-granular visual exploration and VERA for verification and strategic planning.
• Empirical evaluations show that CogniGPT achieves superior accuracy and efficiency in long video QA by reducing frame usage while mitigating hallucinations.

Searching arXiv for the specified CogniGPT paper and closely related work to ground the article. CogniGPT is a framework for long video understanding that is explicitly modeled on human progressive visual cognition. It organizes inference as an interactive loop between a Multi-Granular Perception Agent (MGPA), which performs divergent and focused visual exploration, and a Verification-Enhanced Reflection Agent (VERA), which verifies perceived clues, mitigates hallucination, and guides subsequent perception. In this formulation, long-video reasoning is not treated as exhaustive sequential processing, but as the progressive collection of a minimal set of informative and reliable task-related clues for answering a query about a long video (Li et al., 29 Sep 2025).

1. Conceptual basis and problem setting

CogniGPT is designed for long videos, which are described as having temporal complexity and sparse task-relevant information. The framework is motivated by the observation that humans do not process all video content sequentially; instead, they progressively explore, perceive, and reflect on minimal task-relevant clues. This human analogy is operationalized as a Perception–Reflection loop coupling broad clue discovery with explicit verification and planning (Li et al., 29 Sep 2025).

The framework is positioned within long video understanding rather than generic multimodal reasoning. Its specific target is the joint optimization of completeness and efficiency in capturing task-critical information. The data describe this as a setting in which existing LLM-based approaches still struggle to achieve both completeness and efficiency, while CogniGPT aims to explore a minimal set of informative and reliable task-related clues.

A plausible implication is that the framework treats long-video QA as an adaptive evidence acquisition problem rather than a fixed-context summarization problem. That interpretation follows directly from the iterative selection, verification, and termination decisions described for MGPA and VERA.

2. Agent architecture and iterative control loop

CogniGPT consists of two specialized agents. MGPA emulates human perception and operates in both divergent and focused modes to extract multi-granular clues from a video. VERA is modeled after human metacognition and is responsible for verifying clue reliability, identifying hallucinations, assessing sufficiency, and providing strategic guidance for future perception (Li et al., 29 Sep 2025).

The overall procedure begins with an input long video $V$ and query $Q$. Initialization uses a “Quick Preview,” which provides initial context through K-means clustering on video features and produces quick captions that populate a Working Memory $\mathcal{M}$. After initialization, the system proceeds iteratively. At iteration $t$, MGPA selects a perception tool or action $a_t$ conditioned on $Q$, the current working memory $\mathcal{M}_{t-1}$, and VERA’s guidance $g_t$. The resulting observation $o_t$ updates working memory as

$$\mathcal{M}_t = (a_0, o_0, a_1, o_1, ..., a_t, o_t).$$

VERA then performs three functions on the updated state. First, it verifies the new observation. Second, it assesses whether the accumulated information is sufficient and identifies any remaining gaps. Third, it decides whether to continue or terminate. If more evidence is needed, it specifies what information to seek next and which tool should be used; if sufficient evidence has been gathered, it outputs the answer with an explanation.

This architecture assigns perception and reflection to separate agents, but the loop is interactive rather than strictly modular. A plausible implication is that reliability is improved not merely by post hoc checking, but by using verification to alter the future search policy.

3. Multi-Granular Perception Agent

MGPA extracts key clues at multiple temporal and spatial granularities through three tools: Divergent Search, Temporal Focus, and Spatial Focus (Li et al., 29 Sep 2025).

Divergent Search is the broad exploratory mechanism. It first uses an LLM to decompose the query into a sub-query $Q$0 and select a temporal span. For sampled frames $Q$1, it uses EVA-CLIP-8B visual features and cosine similarity to the sub-query:

$Q$2

The similarities are smoothed with a sliding-window average,

$Q$3

and the global mean

$Q$4

is used as a threshold to segment the timeline into peaks and valleys. The method then selects the highest-scoring frame in each peak, a “watershed” strategy intended to maximize contextual diversity rather than choose redundant adjacent maxima. The selected frames are captioned with timestamps by a vision-LLM.

Temporal Focus supports finer-grained sub-event understanding inside a candidate interval. It extracts video features from a temporal window, clusters them with K-means into $Q$5 groups, and selects the clip nearest each centroid as representative. The clustering objective is

$Q$6

These representative clips are then captioned to provide fine-grained semantic coverage.

Spatial Focus targets detailed object, attribute, or relation queries that are not well captured by general captions. Here, the LLM proposes a subquestion for a specific frame or timestamp, and LLaVA-NeXT is used in a VQA mode over the frame and question to return fine-detailed answers.

Taken together, these three tools define the “multi-granular” aspect of perception: broad frame discovery, local temporal refinement, and frame-level spatial interrogation.

4. Verification-Enhanced Reflection Agent and working memory

VERA is responsible for reliability control and strategy revision. Its central mechanism is Cross-Verification, described as Chain-of-Verification. For each critical observation $Q$7, VERA checks whether the observation supports answering the query. If it does, VERA generates two to three direct verification questions about the corresponding visual content and queries a vision-LLM for objective facts. The returned facts are then compared with the original clue; if they align, the clue is retained as reliable, otherwise it is not (Li et al., 29 Sep 2025).

VERA also performs planning and guidance. It determines whether the available information is sufficient for a confident answer, identifies missing information, and chooses between CONTINUE and TERMINATE. When continuing, it provides explicit guidance about what should be sought next and which MGPA tool should be invoked. This makes reflection operational rather than merely diagnostic.

Working Memory $Q$8 stores the growing sequence of actions and observations and serves as the evolving reasoning context. The data describe it as analogous to human working memory. Because only verified clues are retained, the memory is not just a transcript of observations; it is a filtered state supporting later decisions.

This suggests that CogniGPT’s control policy is shaped by evidence quality as well as evidence quantity. In that sense, VERA functions as both a verifier and a search-policy regulator.

5. Empirical performance and efficiency

CogniGPT was evaluated on EgoSchema, Video-MME, NExT-QA, and MovieChat. The reported metrics include accuracy, frames processed, runtime, number of LLM calls, and tokens generated. The most salient result is that the framework is described as superior in both accuracy and efficiency across these benchmarks (Li et al., 29 Sep 2025).

For training-free agent comparisons, the reported results are:

Method	EgoSchema (Acc./Frames)	Video-MME (Acc./Frames)
LLoVi	51.8 / 180	45.4 / 492
VideoAgent*	60.2 / 8.4	46.4 / 24.6
VideoTree	66.2 / 62.4	53.1 / 128
DrVideo	66.4 / >90	51.7 / >492
CogniGPT	69.2 / 11.2	54.7 / 18.3

The same set of experiments also reports 76.8 / 10.9 on NExT-QA and 95.3 / 10.5 on MovieChat for CogniGPT. On EgoSchema, the framework is noted to surpass existing training-free methods using only 11.2 frames and to achieve performance comparable to Gemini 1.5-Pro.

For comparisons with large multimodal models, the reported EgoSchema results include Gemini 1.5-Pro at 70.2 using 16 frames, GPT-4o at 72.2 using 180 frames, Qwen2.5-VL-72B at 76.2 using 180 frames, and CogniGPT with Qwen2.5-VL-7B at 72.4 using 10.7 frames. The data characterize this as accuracy on par with Gemini 1.5-Pro and GPT-4o while processing far less video.

Runtime results are also included. On EgoSchema, CogniGPT is reported to achieve 69.2% in 39.5s, processing 11.2 frames, while DrVideo requires 143.3s for >90 frames. These measurements support the claim that the framework targets not only answer quality but also evidence efficiency.

6. Ablations, failure prevention, and broader context

The ablation results emphasize three internal design choices. First, using all three MGPA tools together produces the best accuracy. Second, the watershed frame-selection strategy in Divergent Search outperforms simple top-$Q$9 selection because it captures more causally diverse frames. Third, removing VERA’s verification improves frame efficiency but reduces answer quality; the reported effect is that verification yields an accuracy +4.4% improvement by reducing hallucination-driven error (Li et al., 29 Sep 2025).

Hallucination control is therefore not treated as an external calibration step. Instead, every supposed key clue can be challenged by new VQA or fact-extraction prompts, and only information that passes cross-verification remains in working memory. This distinguishes the framework from caption-centric pipelines in which early perceptual errors propagate uncorrected into downstream reasoning.

Within the broader family of cognitively inspired LLM systems, CogniGPT belongs to a line of work that imports explicit cognitive process structure into model orchestration. Related examples include CogGPT, which studies lifelong cognitive dynamics through iterative self-refinement and explicit short-term and long-term memory (Lv et al., 2024), and CogDoc, which introduces a unified coarse-to-fine “Fast Reading” and “Focused Thinking” strategy for long document reasoning (Xu et al., 14 Dec 2025). This suggests that CogniGPT’s contribution is specifically the transfer of progressive perception, reflection, and clue verification to long video understanding.

The framework’s significance lies in its specific synthesis: multi-granular perception, explicit clue verification, iterative guidance, and early termination once sufficient evidence has been gathered. In the reported experiments, that synthesis is associated with state-of-the-art training-free performance and unusually low frame usage on multiple long-video benchmarks.