Spatio-Temporal Textual Cognitive Map

• Spatio-Temporal Textual Cognitive Map (ST-TCM) is a structured representation that binds semantic entities to spatial and temporal contexts to model sequential cognition.
• It integrates content information units from transcripts or frame-by-frame summaries to capture evolving scene dynamics for applications in cognitive impairment assessment and dynamic video reasoning.
• Graph-based features derived from ST-TCM, such as path distances and clustering coefficients, provide actionable insights into visual narration and cognitive status.

Spatio-Temporal Textual Cognitive Map (ST-TCM) denotes a structured representation that binds semantic entities to space and time in order to make sequential cognition explicit. In the literature represented by automated spatio-semantic graph extraction for cognitive impairment assessment and by multimodal reasoning over dynamic 4D scenes, ST-TCM is used to encode what is mentioned or observed, where it is located, and how it evolves over time. In one formulation, it extends a spatio-semantic graph of Content Information Units (CIUs) in picture-description transcripts by adding temporal sequencing; in another, it is a frame-by-frame textual summary of object geometry, kinematics, and pairwise relations that is injected into a multimodal LLM (MLLM) as structured context (Ng et al., 2 Feb 2025, Huang et al., 13 Mar 2026).

1. Formal definition and representational scope

In the transcript-grounded formulation, a spatio-semantic graph $G$ encodes both the order in which a speaker mentions CIUs and the 2-D locations of those units in a reference image. Formally,

$G = (V,E,\ell_V,\ell_E),$

where $V = \{v_1,\ldots,v_N\}$ is the set of nodes, one per CIU mention, $E \subseteq V\times V$ is the set of directed edges linking successive mentions, $\ell_V(v_i) = (c_i, x_i, y_i)$ assigns to node $v_i$ the triple of $$(\text{CIU category } c_i, x\text{-coordinate}, y\text{-coordinate})$$, and $\ell_E((v_i\to v_j)) = w_{ij}$ is a weight, typically the Euclidean distance between $(x_i,y_i)$ and $(x_j,y_j)$. Extending this to ST-TCM adds a timestamp $G = (V,E,\ell_V,\ell_E),$0, so that $G = (V,E,\ell_V,\ell_E),$1, with directed edges ordered by increasing $G = (V,E,\ell_V,\ell_E),$2 and weights that can combine spatial and temporal components:

$G = (V,E,\ell_V,\ell_E),$3

This makes the representation explicitly spatio-temporal rather than purely spatio-semantic (Ng et al., 2 Feb 2025).

In the Dyn-Bench formulation, ST-TCM is a frame-by-frame symbolic summary of every object’s 3D pose, velocity, acceleration, plus pairwise relations such as distance and approach/recede, all rendered in concise text. The resulting textual map is prepended to the usual question prompt, or cross-attended in the vision-language transformer, so that the MLLM is anchored by an explicit dynamic summary rather than having to infer geometry and motion from raw pixels alone (Huang et al., 13 Mar 2026).

Formulation	Core elements	Primary role
Transcript-grounded ST-TCM	CIU category $G = (V,E,\ell_V,\ell_E),$4, coordinates $G = (V,E,\ell_V,\ell_E),$5, timestamp $G = (V,E,\ell_V,\ell_E),$6, directed successive-mention edges	automatic characterization of the visual semantic path during picture description
Video-grounded ST-TCM	3D pose, velocity, acceleration, pairwise relations, concise text	dynamics perception and spatio-temporal reasoning in the physical 4D world

A plausible implication is that ST-TCM is best understood as a representational family rather than a single fixed architecture. Across both settings, the invariant idea is the same: a symbolic scaffold is imposed on sequential perception or narration so that downstream inference can operate on explicit spatial and temporal structure.

2. Transcript-grounded ST-TCM for picture description

The transcript-grounded pipeline begins with a CHAT-formatted transcript of a picture description. The preprocessing steps are specified as follows: remove all punctuation and special characters; lemmatize tokens using spaCy; and, for each lemmatized token, look up which CIU dictionary entry it belongs to. The rule-based tagging stage uses a fixed dictionary that maps approximately 300 common words to 23 CIUs, with ambiguity resolved by an ordered list of CIUs, described as the same order used in clinical practice (Ng et al., 2 Feb 2025).

After tagging, each of the 23 CIU categories is assigned a fixed pixel coordinate $G = (V,E,\ell_V,\ell_E),$7 on the 546×290-pixel Cookie Theft image. When a CIU $G = (V,E,\ell_V,\ell_E),$8 is tagged at time $G = (V,E,\ell_V,\ell_E),$9, the node receives spatial coordinates $V = \{v_1,\ldots,v_N\}$0. CIU mentions are then sorted by spoken order or timestamp, and directed edges $V = \{v_1,\ldots,v_N\}$1 are created for each consecutive mention. The resulting graph operationalizes the participant’s narrative path from transcripts alone, addressing a central limitation of approaches that typically require eye tracking to assess the visual narrative path (Ng et al., 2 Feb 2025).

The reported robustness characteristics are also specific. Compared to human annotators on a held-out set, the automatic method recovered over 90 % of manually tagged CIUs and also captured additional mentions that were overlooked by humans. Errors are grouped into three categories: false-positives, in which a word appears but is not intended as a CIU; false-negatives, associated with unseen synonyms; and occasional misalignments in sequence (Ng et al., 2 Feb 2025).

This transcript-based formulation retains the interpretability of CIU analysis while removing the manual tagging bottleneck. Because the coordinates are fixed on a known reference image, temporal order and spatial displacement become directly measurable from language alone.

3. Graph-theoretic features and clinical discrimination

The graph representation supports both structural and spatio-temporal metrics. The adjacency matrix is

$V = \{v_1,\ldots,v_N\}$2

The degree of node $V = \{v_1,\ldots,v_N\}$3 is $V = \{v_1,\ldots,v_N\}$4, and the total path length is

$V = \{v_1,\ldots,v_N\}$5

For the spatial formulation, the Euclidean distance is

$V = \{v_1,\ldots,v_N\}$6

The feature inventory includes Avg. X, Std. X, Avg. Y, Std. Y, total path distance, unique nodes, nodes, self-cycles, cycles, quadrant self-cycles, cross-ratio, and normalized path $V = \{v_1,\ldots,v_N\}$7. The additional ST-TCM features include mean time between CIU mentions, temporal variance, a quadrant transition probability matrix $V = \{v_1,\ldots,v_N\}$8, and the clustering coefficient

$V = \{v_1,\ldots,v_N\}$9

Once $E \subseteq V\times V$0 is built, standard graph-embedding methods, including Node2Vec and spectral embeddings, may be applied to turn the nodes or whole graph into fixed-length vectors for machine learning (Ng et al., 2 Feb 2025).

For cognitive mapping, an ANCOVA covarying age, education, gender, and unique nodes compared cognitively unimpaired speakers against impaired speakers grouped as MCI + dementia. The key findings using automatically extracted CIUs are summarized below (Ng et al., 2 Feb 2025).

Feature	F-value	Means
Total path distance	39.5***	2521 px vs 2952 px
Total path/Unique nodes	37.5***	164 vs 191
Self-cycles (CIU)	4.34**	0.56 vs 0.72
Cycles	45.6***	6.65 vs 9.23
Nodes	47.6***	22.8 vs 26.2
Self-cycles (Quadrants)	39.3***	10.8 vs 12.6

The associated interpretation is explicit: impaired speakers took longer, more erratic paths, repeated CIUs more often, and transitioned across quadrants more. The features achieved highly significant group differences, with F-values greater than 30 in many cases. As a standalone feature set, spatio-semantic and ST-TCM features can be fed into a classifier such as logistic regression, SVM, or gradient boosting to predict cognitive status with accuracy competitive with more opaque embeddings (Ng et al., 2 Feb 2025).

A common misconception is that this class of narrative-path analysis requires gaze instrumentation. The reported pipeline is specifically designed to estimate the visual semantic path from transcripts alone.

4. Video-grounded ST-TCM for dynamic 4D reasoning

In the dynamic-scene formulation, a video is sampled at $E \subseteq V\times V$1 synchronized RGB-D frames with an instance mask set $E \subseteq V\times V$2. For each object $E \subseteq V\times V$3 in frame $E \subseteq V\times V$4, the representation uses the image centroid in pixels $E \subseteq V\times V$5, estimated depth at that pixel, and camera intrinsics $E \subseteq V\times V$6 together with camera extrinsics $E \subseteq V\times V$7 from VIPE. The 3D position is recovered by

$E \subseteq V\times V$8

Velocities and accelerations are then obtained by finite differencing,

$E \subseteq V\times V$9

and each trajectory is smoothed via an exponential moving average to remove depth noise (Huang et al., 13 Mar 2026).

For every object pair $\ell_V(v_i) = (c_i, x_i, y_i)$0 at time $\ell_V(v_i) = (c_i, x_i, y_i)$1, the system computes Euclidean distance

$\ell_V(v_i) = (c_i, x_i, y_i)$2

a relative motion scalar

$\ell_V(v_i) = (c_i, x_i, y_i)$3

described as positive when $\ell_V(v_i) = (c_i, x_i, y_i)$4 is approaching $\ell_V(v_i) = (c_i, x_i, y_i)$5 and negative when receding, and camera-centric bearing expressed as azimuth $\ell_V(v_i) = (c_i, x_i, y_i)$6 and elevation $\ell_V(v_i) = (c_i, x_i, y_i)$7. These real-valued attributes are converted via a small rule-based cognitive mapper $\ell_V(v_i) = (c_i, x_i, y_i)$8 into English phrases. Stacking the frame-level strings yields

$\ell_V(v_i) = (c_i, x_i, y_i)$9

which is treated as a single structured textual prefix to the downstream question-answering model (Huang et al., 13 Mar 2026).

Within the Dyn-Bench framework, ST-TCM sits between raw video + masks + depth, structured text, and the multimodal LLM. Its function is to supply temporal and spatial scaffolding so that the vision-language backbone has stable, metric cues for localized dynamics perception and spatio-temporal reasoning.

5. Architectural integration and empirical performance in MLLMs

The implementation described for dynamic video reasoning uses Qwen3-VL-235B as the base MLLM. A Vision Encoder ingests each raw frame, or a mask-guided composite if using Mask-Guided Fusion. A standard text tokenizer converts ST-TCM text and the question into token embeddings. Within every transformer layer, visual tokens and text tokens cross-attend, so that ST-TCM tokens supply a strong prior when the model attends to pixel features. A small multimodal fusion head merges the attended representations before decoding the final answer. Mask-Guided Fusion injects an extra channel where each pixel’s mask bit highlights moving objects, allowing the backbone to focus on those regions more strongly (Huang et al., 13 Mar 2026).

Training and inference behavior is deliberately constrained. ST-TCM is built offline and used at inference time; no new end-to-end training is required. All models remain in zero-shot or few-shot mode, with their original weights frozen. The only training described is tuning the random-forest regressor in video filtering and crafting the rule-based template library $v_i$0; losses and optimization remain those already baked into Qwen3-VL (Huang et al., 13 Mar 2026).

The quantitative gains reported for structured integration are as follows (Huang et al., 13 Mar 2026).

Setting	Metric	Result
Textual ablation on Qwen3-VL-32B	Average VQA accuracy	59.0% to 69.2% with full T+M+S; motion + spatial cues: 68.4%
Region-level grounding on Sa2VA-InternVL2.5-8B	Overall $v_i$1	75.6 to 77.3
Mask-Guided Fusion	Average VQA	raw video + ST-TCM: 53.8%; only masks: 53.8%; masks + frames: 57.1%

Across Inter-Object, Object-Scene, and Camera-Object categories, these structured integrations outperform nothing, chain-of-thought hints, and caption-based cues. Conventional prompting strategies provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and ST-TCM, significantly enhance dynamics perception and spatio-temporal reasoning in the physical 4D world (Huang et al., 13 Mar 2026).

A second common misconception is that ST-TCM is merely a prompting style. In this formulation it is a structured external representation derived from geometry, depth, masks, and camera pose, then injected into a frozen multimodal system.

6. Limitations, misconceptions, and research directions

The limitations of the transcript-grounded formulation are concrete. The dictionary approach misses novel descriptions and rare synonyms, and temporal resolution is limited to transcript segmentation, described as one token at a time. Future datasets should include explicit time-aligned transcripts or eye-tracking to validate temporal inferences, and longitudinal studies of ST-TCM change over months or years could reveal progressive alterations in cognitive maps (Ng et al., 2 Feb 2025).

The limitations of the video-grounded formulation are different. ST-TCM adds pre-processing overhead through pose estimation and template generation, and the template library must cover diverse object categories. At the same time, the ablation evidence clarifies why the method works: temporal text alone yields only a +0.3% gain; motion or geometry individually can yield +5–7% gains; combined, they yield +10%. The accompanying explanation is that conventional chain-of-thought prompts do not supply grounded, metric information about where and how fast objects are moving, whereas ST-TCM explicitly encodes temporal continuity, spatial layout, and relational dynamics (Huang et al., 13 Mar 2026).

Several extensions are suggested directly. Automatic CIU extraction can be replaced or augmented by large-language-model classifiers to catch paraphrases and synonyms; a static dictionary can be replaced by a sequence tagger such as BERT-CRF; prosodic timestamps including speech rate and pause durations can refine temporal weights; dynamic graph neural networks such as Temporal Graph Networks can learn embeddings over evolving ST-TCM; and acoustic and lexical features can be fused with ST-TCM to improve sensitivity to early impairment (Ng et al., 2 Feb 2025).

Taken together, these formulations position ST-TCM as an interpretable spatio-temporal scaffold. In the clinical setting, it generalizes spatio-semantic graphs by embedding sequence timing and yields graph-based features sensitive to cognitive status. In multimodal video reasoning, it distills 4D scene evolution into textual form that can be consumed by frozen MLLMs. A plausible implication is that the concept’s unifying significance lies less in a specific implementation than in the deliberate externalization of spatial, semantic, and temporal structure for downstream inference.