LET-US: Long Event-Text Understanding

• LET-US is a framework for long event-stream text comprehension that uses adaptive, query-guided compression and hierarchical clustering to handle ultra-long, asynchronous data.
• It employs a two-stage optimization paradigm, beginning with large-scale visual-language pretraining followed by event-specific fine-tuning on datasets like EIQA-1M for robust event-text alignment.
• Benchmarking on resources such as EVQA-Bench demonstrates that LET-US outperforms traditional multimodal models in classification, captioning, reasoning, and temporal localization tasks.

Searching arXiv for the target paper and closely related event-language work to ground the article. arXiv Search Query: (Chen et al., 10 Aug 2025) LET-US Long Event-Text Understanding of Scenes LET-US, short for Long Event-Text Understanding of Scenes, is a framework for long event-stream–text comprehension designed for event-camera data, which consist of sparse, asynchronous signals with microsecond-level temporal resolution. It is positioned against a setting in which existing Multimodal LLMs (MLLMs) have achieved substantial success on RGB video but either fail to interpret event streams effectively or remain constrained to very short sequences. LET-US addresses this setting through adaptive compression of long event streams, a two-stage optimization paradigm for event–text alignment, and a benchmark spanning reasoning, captioning, classification, temporal localization, and moment retrieval (Chen et al., 10 Aug 2025).

1. Problem setting and scope

Event cameras output event streams as sparse, asynchronous data with microsecond-level temporal resolution, enabling low-latency perception and operation under a high dynamic range. In the formulation associated with LET-US, this modality creates a distinct long-context problem for multimodal modeling: event streams may contain millions of events, reaching up to $10^9$ timestamped events even for short recording durations, and may exceed $10^6$ events per second (Chen et al., 10 Aug 2025).

The framework is motivated by four coupled difficulties. First, data volume exceeds the input or context limits of conventional MLLMs. Second, sparsity and redundancy are highly non-uniform across time, with informative “event bursts” interleaved with relatively uninformative segments. Third, noise and redundancy reduce computational efficiency. Fourth, there is a substantial modality gap between event data—described as binary changes and spatiotemporal bursts—and textual representations (Chen et al., 10 Aug 2025).

Within this problem formulation, LET-US is not merely a long-context variant of video-language modeling. Its central claim is that long event understanding requires both information-adaptive compression and event–language alignment. This suggests that the framework treats temporal scale and cross-modal semantics as inseparable design constraints rather than independent subsystems (Chen et al., 10 Aug 2025).

2. Adaptive compression of long event streams

The core systems contribution of LET-US is a two-stage, query-guided adaptive compression mechanism that reduces input size while preserving visually and semantically salient event content (Chen et al., 10 Aug 2025).

In the first stage, the event stream is partitioned into discrete temporal bins, each containing spatial, temporal, and polarity attributes. Features are extracted for each bin using SigLIP2 + DINOv2. A textual query is encoded, and cosine similarity between the query feature and each bin feature is computed. Bins are retained only when they satisfy a thresholded similarity criterion: $$\widetilde{V} = \left\{ V^S_t\ |\ \mathrm{sim}(V^S_t, q) \geq \tau,\, t=1,\ldots,T \right\}.$$ This stage is described as cross-modal guided compression, because the query is used to decide which temporal regions should survive the first reduction step (Chen et al., 10 Aug 2025).

In the second stage, the retained bins are divided into non-overlapping windows of size $J$. For each window $W_m$, the framework computes the average pairwise cosine distance after $\ell_2$ normalization: $$D_m = \frac{2}{J(J-1)} \sum_{1 \leq i < j \leq J} [1 - \langle V_i, V_j \rangle].$$ This diversity score determines the number of clusters: $$R_m = \max \left( 1,\, \min \left(J,\, \mathrm{round}\left( \frac{D_m}{2} J \right) \right) \right).$$ A bottom-up average-link hierarchical clustering is then applied within each window, and cluster members are averaged to form compressed event features (Chen et al., 10 Aug 2025).

The resulting compressed token sequence is temporally ordered and passed forward as a reduced representation of the original event stream. The stated objective is to preserve the most query-relevant, information-dense event features rather than perform uniform downsampling. This is significant because the compression is adaptive in two senses: it is conditioned on the query and it varies according to within-window feature diversity (Chen et al., 10 Aug 2025).

3. Event–text alignment and model pipeline

LET-US addresses the event–text modality gap through a two-stage optimization paradigm. The first stage is visual-language pretraining on large-scale vision–text data, specifically LLaVA-OneVision and VideoChat2-IT, to establish general scene and streaming-data understanding. The second stage is event-language fine-tuning on a custom large-scale event–text corpus, EIQA-1M, which is used to align event-frame representations with natural language (Chen et al., 10 Aug 2025).

This staged design is described as progressive acquisition of event-scene interpretability. The first stage supplies general visual grounding, while the second introduces event-specific semantic alignment. A plausible implication is that LET-US assumes direct training on event–text pairs alone would be insufficient because of the limited availability of native event-language supervision (Chen et al., 10 Aug 2025).

The paper’s pipeline summary proceeds as follows. Event streams are segmented into bins; features are extracted; cross-modal similarity is used to retain relevant bins; hierarchical clustering compresses the selected bins; the resulting event features are processed through an Event-Language Adapter; these are concatenated with query features; and the fused tokens are fed into Llama3.2-3B to generate textual outputs (Chen et al., 10 Aug 2025).

This architecture distinguishes LET-US from models that simply append event features to an LLM context. The framework instead introduces an explicit intermediate stage for feature reduction and alignment, which is central to its handling of ultra-long event sequences (Chen et al., 10 Aug 2025).

4. Datasets and benchmark construction

LET-US is accompanied by two named data resources: EIQA-1M and EVQA-Bench (Chen et al., 10 Aug 2025).

EIQA-1M, expanded as Event-Image-QA-1M, contains more than one million QA pairs. It is described as using aligned synthetic event images from N-ImageNet, generated with an event simulator such as v2e, together with QA pairs produced automatically by ChatGPT and filtered by humans. Its role is training and fine-tuning for event–text alignment in the second stage of optimization (Chen et al., 10 Aug 2025).

EVQA-Bench, expanded as Event-Video-QA-Benchmark, contains 50K QA pairs and spans multiple domains including driving, cinema, and human action. It is defined as the principal evaluation benchmark and covers classification, captioning, reasoning, moment retrieval, temporal localization, and dialogue, with timestamp spans from $10^5$ to $10^9$ (Chen et al., 10 Aug 2025).

Resource	Scale	Role
EIQA-1M	$10^6$01M QA pairs	Event–text alignment training
EVQA-Bench	50K QA pairs	Evaluation benchmark

The benchmark design is important because it does not limit evaluation to captioning or closed-set recognition. Instead, it tests open-ended and temporally grounded tasks over long event streams. This suggests that the framework is intended as a general event-language understanding system rather than a task-specific recognizer (Chen et al., 10 Aug 2025).

5. Benchmark tasks and empirical results

The evaluation suite for LET-US covers classification, captioning, reasoning, temporal localization, and moment retrieval. The task descriptions include classification for human action and object categories, captioning for both “sparse” and “dense” event streams, reasoning over event-based scene content, and temporal grounding of specified moments in long streams (Chen et al., 10 Aug 2025).

The reported empirical finding is that LET-US outperforms prior state-of-the-art MLLMs in both descriptive accuracy and semantic comprehension on long-duration event streams. The summary specifies that LET-US outperforms baselines, including state-of-the-art MLLMs and EventGPT, across nearly every task and timestamp span, while using fewer parameters in those comparisons (Chen et al., 10 Aug 2025).

One stated example concerns action classification on N-Caltech101, where LET-US is reported at 0.44/0.70 compared with VideoLLaMA3 at 0.41/0.70, while also being described as the only model capable of handling the full range of event durations up to $10^6$1 timestamps (Chen et al., 10 Aug 2025). More generally, the framework is said to maintain superiority on reasoning, moment retrieval, and captioning, including dense long-stream settings (Chen et al., 10 Aug 2025).

Ablation evidence is also reported. The component analysis indicates that the full LET-US design outperforms alternatives based on random, interval, or solely clustering-based sampling, and that cross-modal filtering together with hierarchical clustering contributes to performance, especially for dense captioning and temporal localization (Chen et al., 10 Aug 2025). In methodological terms, this supports the claim that query guidance and adaptive temporal aggregation are complementary rather than interchangeable.

6. Interpretation, significance, and limitations

LET-US is described as the first MLLM to achieve robust, semantically aligned understanding of ultra-long event streams up to $10^6$2 timestamps (Chen et al., 10 Aug 2025). Within the paper’s framing, its significance lies in establishing a viable route from raw event-camera streams to text-conditioned reasoning and description at durations beyond the reach of short-sequence event models.

The framework also has a broader methodological significance. By combining query-guided selection, hierarchical clustering, event-language adaptation, and staged alignment, it defines a blueprint for treating event data as a first-class multimodal input rather than as an RGB surrogate. This suggests a transition from event recognition pipelines toward event-native multimodal inference (Chen et al., 10 Aug 2025).

Several boundaries remain implicit in the formulation. LET-US depends on query-conditioned compression, so its retained representation is partly shaped by the textual prompt. It also relies on curated event–text alignment data and synthetic event-image QA generation for training. These choices indicate that long event understanding, in this framework, is mediated by both dataset design and compression policy rather than by raw long-context scaling alone (Chen et al., 10 Aug 2025).

The paper states that all datasets, codes, and models will be publicly available (Chen et al., 10 Aug 2025). If that release proceeds as described, LET-US and its associated resources would function not only as a model but also as an evaluation standard for long event-stream–text research, particularly in reasoning, captioning, classification, temporal localization, and moment retrieval over high-temporal-resolution asynchronous visual data.