VTEM: Video-to-Event Mapper

• VTEM is a semantic discretization module that transforms continuous video frames into discrete, interpretable event tokens using spatio-temporal feature extraction, adaptive pooling, and conceptual quantization.
• It ensures temporal coherence and preserves action progression by integrating event coherence bias and contrastive objectives, crucial for fine-grained long-term action recognition.
• VTEM feeds a compact visual event sentence into a large vision-language model, achieving state-of-the-art performance on datasets like NTU RGB+D with 94.1% accuracy.

Video-to-Event Mapper (VTEM) is the front-end module in VT-LVLM-AR that transforms raw video into compact, semantically rich, and temporally coherent “visual event sequences” for downstream reasoning by a Large Vision-LLM. In the formulation of “VT-LVLM-AR: A Video-Temporal Large Vision-LLM Adapter for Fine-Grained Action Recognition in Long-Term Videos” (Li et al., 21 Aug 2025), VTEM addresses long-term video action recognition by replacing high-frame-rate, high-dimensional video with a discrete “visual event sentence” built through lightweight spatio-temporal feature extraction, adaptive temporal pooling, and conceptual quantization with an event coherence bias.

1. Definition and functional role

VTEM is introduced as a critical module designed to bridge continuous video signals and the discrete, semantically meaningful inputs required by Large Vision-LLMs. Its stated mission is to convert raw video into a compact sequence of “visual event tokens,” forming an interpretable “visual event sentence” that captures both the semantic richness and temporal coherence of human actions. The target setting is fine-grained action recognition in long-term videos, where complex backgrounds, subtle action differences, computational overhead, difficulty in capturing long-range temporal dependencies, and limited semantic understanding pose difficulties for conventional deep learning models (Li et al., 21 Aug 2025).

Within VT-LVLM-AR, VTEM is not an auxiliary preprocessor in the narrow sense. It is the mechanism that determines how video content is discretized before LVLM-based reasoning. The paper explicitly presents three stages: spatio-temporal feature extraction, adaptive temporal pooling, and conceptual quantization. The output is then consumed by an Action Reasoning module built on a frozen LLaVA-1.5 model adapted with P-Tuning v2 for action classification (Li et al., 21 Aug 2025).

A common source of ambiguity is the word “event.” In VTEM, an event is a discrete semantic token representing a sub-action or sub-event, not a sensor-level brightness-change spike. This distinction becomes important when VTEM is compared with event-camera literature, where an event is typically a tuple such as $(p, x, y, t)$ (Hannan et al., 2024).

2. Mapping raw video to a visual event sequence

Given an input video

$V = \{f_1, f_2, \dots, f_T\},$

VTEM first applies a lightweight spatio-temporal backbone, such as a sparsely-sampled 3D ResNet or Swin Transformer. The output is a sequence of feature vectors

$$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$

with the formal mapping

$$H = \mathcal{F}(V) = \left\{ \mathcal{F}_\text{seg}(f_t, \dots, f_{t+k}) \right\}_{t=1}^N.$$

This stage captures appearance and dynamics from either individual frames or short clips (Li et al., 21 Aug 2025).

The second stage, Adaptive Temporal Pooling, reduces redundancy and emphasizes long-range dependencies by aggregating features over variable, salient time segments. From the feature sequence $H$, VTEM produces

$$P = \{p_1, p_2, \dots, p_M\}, \qquad p_i \in \mathbb{R}^{D_p},\ M \ll N,$$

with

$$P = \mathcal{P}(H) = \left\{ \mathrm{Pool}(h_j, \dots, h_k) \right\}_{i=1}^M.$$

The pooling function can be learnable or dynamic, adapting window sizes to pool relevant sub-actions. This compression step is central to the mapper’s ability to represent long videos compactly without treating all temporal segments uniformly (Li et al., 21 Aug 2025).

The processing sequence therefore moves from dense frame streams to segment-level latent structure and then to a much shorter event sequence. This suggests that VTEM is designed less as a frame selector than as a structured abstraction module: it preserves action progression while changing the representational type from continuous features to discrete semantic tokens.

3. Conceptual quantization and event coherence

The core operation in VTEM is Conceptual Quantization, which converts pooled features into a discrete vocabulary of learned “event tokens.” The codebook is defined as

$$\mathcal{C} = \{c_1, c_2, ..., c_K\}, \qquad c_k \in \mathbb{R}^{D_e}.$$

Each pooled feature $p_j$ is mapped to the closest codebook entry in terms of Euclidean distance,

$$e_j = \operatorname*{argmin}_{c_k \in \mathcal{C}} \| p_j - c_k \|_2^2,$$

yielding the token sequence

$V = \{f_1, f_2, \dots, f_T\},$0

The paper characterizes each $V = \{f_1, f_2, \dots, f_T\},$1 as a discrete, high-level semantic token corresponding to a sub-action or sub-event, with the codebook learned using straight-through estimators for gradient backpropagation (Li et al., 21 Aug 2025).

VTEM does not treat quantization as mere compression. To ensure that the resulting token sequence is semantically and temporally coherent rather than an unordered collection of event words, training combines a reconstruction objective and a contrastive objective. The reconstruction loss is described as MSE between $V = \{f_1, f_2, \dots, f_T\},$2 and reconstructed $V = \{f_1, f_2, \dots, f_T\},$3 from $V = \{f_1, f_2, \dots, f_T\},$4, while the contrastive term is described as a coherence or temporal consistency objective, for example InfoNCE, that pulls temporally proximate tokens together and pushes apart unrelated ones. The stated effect is a compact, faithful, and temporally ordered “visual event sentence” (Li et al., 21 Aug 2025).

This design positions coherence as part of the representation itself rather than solely a downstream modeling concern. Removing the coherence bias degrades recognition accuracy, which indicates that the quantized sequence is intended to preserve event continuity, not just category-discriminative content.

4. Interface to LVLM-based action reasoning

The output of VTEM, $V = \{f_1, f_2, \dots, f_T\},$5, is the intermediate representation provided to the LVLM. In VT-LVLM-AR, this LVLM is a frozen LLaVA-1.5 model adapted by parameter-efficient Prompt Tuning (P-Tuning v2) for action classification. The overall framework is therefore organized as a video-to-language translation pipeline: VTEM produces a language-like event representation, and the adapted LVLM performs action reasoning over that representation (Li et al., 21 Aug 2025).

The paper attributes several properties to this arrangement. First, it enables robust action reasoning without full finetuning of a large multimodal model. Only approximately $V = \{f_1, f_2, \dots, f_T\},$6M parameters are updated via prompt tuning, compared with approximately $V = \{f_1, f_2, \dots, f_T\},$7B for full finetuning. Second, the event sentences are presented as interpretable and human-readable summaries of action progression. Third, the paper reports that the LVLM alone, or the LVLM supplied with unstructured video features, is dramatically less effective than the full configuration with VTEM (Li et al., 21 Aug 2025).

A plausible implication is that VTEM acts as the semantic contract between video and LVLM reasoning. Rather than asking a frozen LVLM to absorb raw temporal complexity directly, the framework supplies a discrete sequence whose units already encode sub-action structure and temporal ordering.

5. Empirical characterization and ablations

VT-LVLM-AR is evaluated on NTU RGB+D and NTU RGB+D 120, and the abstract reports that it consistently achieves state-of-the-art performance, including $V = \{f_1, f_2, \dots, f_T\},$8 accuracy on NTU RGB+D X-Sub (Li et al., 21 Aug 2025). The ablation results provided for VTEM are especially informative because they isolate the contribution of each mapping stage.

Method Variant	NTU RGB+D X-Sub (%)
VT-LVLM-AR (Full Model)	94.1
w/o Conceptual Quantization (cont. feat.)	91.5
w/o Adaptive Temporal Pooling (fixed)	92.8
w/o Event Coherence Bias	93.3

These results show that directly sending continuous features to the LVLM reduces accuracy from $V = \{f_1, f_2, \dots, f_T\},$9 to $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$0, using naive or flat pooling reduces it from $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$1 to $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$2, and removing the contrastive event-coherence loss reduces it to $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$3. The paper therefore treats conceptual quantization, adaptive pooling, and event coherence bias as critical components rather than interchangeable implementation details (Li et al., 21 Aug 2025).

The representation itself is also tuned empirically. The best token sequence length is reported as $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$4, with shorter sequences losing temporal detail and longer sequences introducing noise or redundancy. The best codebook size is reported as $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$5, with smaller codebooks under-representing events and larger ones risking overfitting or sparsity. Human evaluation further supports the interpretability claim: VTEM event sentences are rated much higher, with mean $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$6 for coherence versus $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$7 for a baseline, in terms of narrative clarity in describing actions (Li et al., 21 Aug 2025).

6. Relation to broader video-to-event mapping research

The VTEM of VT-LVLM-AR belongs to a broader family of systems that map unstructured video into event-centric representations, but the form of the event varies substantially across the literature. In “TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding” (Yang et al., 3 Aug 2025), events are represented as tuples

$$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$8

where $$H = \{h_1, h_2, \dots, h_N\}, \qquad h_i \in \mathbb{R}^{D_h},\ N \leq T,$$9 is timestamp, $$H = \mathcal{F}(V) = \left\{ \mathcal{F}_\text{seg}(f_t, \dots, f_{t+k}) \right\}_{t=1}^N.$$0 is saliency score, and $$H = \mathcal{F}(V) = \left\{ \mathcal{F}_\text{seg}(f_t, \dots, f_{t+k}) \right\}_{t=1}^N.$$1 is caption; the model then performs causal autoregressive event prediction over structured outputs. In “VidCEP: Complex Event Processing Framework to Detect Spatiotemporal Patterns in Video Streams” (Yadav et al., 2020), raw video is converted into a Video Event Knowledge Graph and queried through VEQL for near real-time complex event matching. In “EA-VTR: Event-Aware Video-Text Retrieval” (Ma et al., 2024), event awareness is developed through Event Content Augmentation, Event Temporal Augmentation, and frame-level or video-level contrastive alignment.

A different use of “video-to-event” appears in event-camera research, where the target is not a semantic token sequence but a stream of asynchronous sensor events. “Video to Events: Recycling Video Datasets for Event Cameras” (Gehrig et al., 2019), “v2e: From Video Frames to Realistic DVS Events” (Hu et al., 2020), and “V2CE: Video to Continuous Events Simulator” (Zhang et al., 2023) all address the conversion of conventional video into synthetic DVS-like events, with emphasis on threshold dynamics, timestamp realism, noise, or continuous event sampling. Conversely, “Event-to-Video Conversion for Overhead Object Detection” (Hannan et al., 2024) studies the reverse direction, converting event streams into gray-scale video to better align with pre-trained RGB detectors.

This comparison clarifies a frequent misconception. VTEM in VT-LVLM-AR is not an event-camera simulator, a temporal grounding decoder, or a CEP matcher. It is a semantic discretization module for long-term action understanding. At the same time, the related literature suggests a broader taxonomy of video-to-event mappings: semantic tokenization, structured event tuples, graph-based complex events, cross-modal event-text alignments, and neuromorphic sensor events. A plausible implication is that VTEM occupies the semantic end of this spectrum, where the primary objective is not photometric fidelity or sensor realism but compact, coherent, and LVLM-compatible action abstraction.