LUST: User-Defined Video Significance Tracker

• The paper introduces LUST, a framework that uses user-defined thematic anchors to generate segment-wise relevance scores by fusing visual cues and ASR-derived speech.
• It employs a two-stage hierarchical scoring mechanism where direct relevance is first computed and then refined with temporal context to capture narrative progression.
• The system outputs an annotated video and detailed analytical logs, enabling nuanced, user-driven video summarization and thematic tracking.

Learned User Significance Tracker (LUST) is a multi-modal, LLM-based framework for thematic significance tracking in video. Given a video $V$ and a user-provided textual description of significance, the reference summary $R_{sum}$, it analyzes visual content and ASR-derived speech, assigns each segment a local relevance estimate, then refines that estimate with temporal context to produce a contextual relevance time series over the video. The resulting system is intended to quantify how important each part of a video is to a user-defined theme, and to output both an annotated video and detailed analytical logs (Luiz, 6 Aug 2025).

1. Definition, problem formulation, and semantic scope

LUST addresses thematic significance tracking: for a video segmented over time, it scores each segment according to how strongly that segment matches a user-defined theme. The user supplies $R_{sum}$, a natural-language description of what counts as significant. The paper gives examples such as “Tracking moments of escalating tension followed by a resolution” and “Identifying instances of collaborative problem-solving.” The authors treat $R_{sum}$ as the semantic anchor or “significance vector,” and define thematic relevance operationally through the scores produced by the LLM-based pipeline rather than through an independent formal semantics (Luiz, 6 Aug 2025).

The framework takes as inputs a video $V$ with audio, of duration $T_{vid}$, and the reference summary $R_{sum}$. Its intermediate representations are segmented video windows, one representative frame $I_i$ per window, a speech context snippet $\mathcal{C}_{S,i}$ aggregated from ASR output around that window, and two scalar scores for each segment: the direct relevance score $S_{d,i} \in [0,1]$ and the contextual relevance score $R_{sum}$0. Its outputs are an annotated video in which $R_{sum}$1 is visually overlaid over time, together with analytical logs containing configuration data, the full transcript $R_{sum}$2, per-segment records, and saved representative frames.

A central feature of the formulation is that “significance” is not predefined by a fixed ontology. It is conditioned on $R_{sum}$3, repeatedly injected into prompts, and interpreted by the LLM in context. This makes LUST explicitly user-driven: what is significant in one run may be peripheral in another if the reference summary changes.

2. Multi-modal preprocessing and internal representations

The architecture begins by dividing the video into contiguous fixed-length windows of length $R_{sum}$4, with an example value of $R_{sum}$5 second. For a window $R_{sum}$6, the start time, end time, duration, and center time are defined by

$R_{sum}$7

$R_{sum}$8

$R_{sum}$9

$R_{sum}$0

From each window, the system extracts the frame closest to $R_{sum}$1,

$R_{sum}$2

then converts it to a PIL image and encodes it as a base64 data URI $R_{sum}$3 for the multi-modal LLM interface. The paper explicitly notes that there is no deep vision backbone or learned visual encoder separate from the LLM; the multi-modal LLM consumes the image directly (Luiz, 6 Aug 2025).

Audio is demultiplexed from the video and standardized to WAV at 16 kHz, mono, 16-bit PCM. ASR is performed with an efficient reimplementation of OpenAI Whisper, using configurable model size such as a medium English model. The resulting transcript is a set of time-stamped utterances

$R_{sum}$4

Voice Activity Detection is also used to clean up the transcripts.

For each window, LUST constructs a local speech context by gathering all utterances whose time spans intersect a temporal radius $R_{sum}$5 around the window center: $R_{sum}$6 The speech context is then

$R_{sum}$7

where $R_{sum}$8 denotes string concatenation, with each utterance formatted with timestamps. If no utterances fall in the specified radius, a placeholder is used to indicate the absence of speech.

Fusion is implicit rather than feature-level. In Stage 1, the LLM receives text and image jointly and performs its own internal cross-modal reasoning. In Stage 2, only text is used. The design therefore differs from architectures based on explicit cross-modal transformers or separately trained visual and textual encoders.

3. Hierarchical LLM-based scoring mechanism

The core of LUST is a two-stage hierarchical scoring procedure in which local segment evidence is separated from temporal reinterpretation (Luiz, 6 Aug 2025).

Stage	Inputs	Output
Stage 1: direct relevance	$R_{sum}$9, $R_{sum}$0, optionally $R_{sum}$1	$R_{sum}$2
Stage 2: contextual relevance	$R_{sum}$3, $R_{sum}$4, $R_{sum}$5, optionally $R_{sum}$6	$R_{sum}$7

In Stage 1, LUST assigns a direct relevance score to each window based only on that segment’s immediate content. Two prompt templates are defined: $R_{sum}$8 when audio context is present, and $R_{sum}$9 when it is absent. The multi-modal LLM $V$0 receives a textual prompt $V$1, the image $V$2, and a system prompt $V$3, and outputs a scalar that is then clamped into $V$4: $V$5 Conceptually, this stage is intentionally myopic: it asks how relevant the current segment is to $V$6 without reference to wider narrative progression.

In Stage 2, LUST computes a contextual relevance score. This uses a truncated theme description

$V$7

with an example value $V$8 characters, and a truncated history of direct scores

$V$9

$T_{vid}$0

The prompt varies depending on whether $T_{vid}$1, whether audio is present, and whether older history had to be dropped. The Stage 2 score is

$T_{vid}$2

The paper does not define an explicit algebraic combination rule beyond this two-stage process. Stage 2 uses $T_{vid}$3 and $T_{vid}$4 to produce $T_{vid}$5, and for downstream use the final thematic significance per segment is $T_{vid}$6. This hierarchical organization is therefore not a scene-level decomposition but a conceptual progression from direct to contextual relevance. At video level, the sequence $T_{vid}$7 forms a time series, and the paper notes an average contextual score for logging.

A recurring interpretive point is that LUST does not contain an explicit neural temporal model such as an RNN or Transformer over segment embeddings. Temporal modeling is implemented through prompt construction: the LLM is given recent score trajectories and local speech and is expected to infer buildup, climax, repetition, continuity, or resolution from that textualized history.

4. Prompts, models, and implementation status

LUST uses LLMs purely for scoring. The system prompt $T_{vid}$8 instructs the model to act as an analytical scoring tool, to produce only a numeric score in $T_{vid}$9, and to avoid free-form explanations. Stage 1 uses the model’s multi-modal interface; Stage 2 uses the same model in text-only mode. The paper mentions $R_{sum}$0 as, for example, “a model from the Mistral-Small series or similar,” and describes low temperature, with an example value of $R_{sum}$1, to promote deterministic and stable outputs (Luiz, 6 Aug 2025).

A notable terminological issue concerns the word “learned.” LUST is not trained end-to-end on labeled thematic relevance data. The paper explicitly states that there is no supervised training of the LUST pipeline, no loss function for the framework as a whole, and no labeled dataset of thematic significance. In this usage, “learned” refers to the use of pre-trained components—Whisper for ASR and pre-trained LLMs for scoring—and to the fact that significance is learned by the LLM in context from $R_{sum}$2 and the prompts rather than hard-coded.

Several parameters are configurable. The window length $R_{sum}$3 is configurable, with $R_{sum}$4 second given as an example. The speech radius $R_{sum}$5 is configurable, with an example of approximately $R_{sum}$6 seconds. The history length $R_{sum}$7 caps the number of prior direct scores injected into Stage 2. Windows are non-overlapping in time, although their speech contexts may overlap because of the radius-based aggregation around window centers.

The implementation also produces a substantial audit trail. Logs include the configuration and average contextual score, the full transcript $R_{sum}$8, per-segment CSV or JSON records with $R_{sum}$9, $I_i$0, $I_i$1, $I_i$2, and $I_i$3, and representative frames saved with filenames encoding scores. The annotated video overlays a semi-transparent plot along the frame bottom, normalizes $I_i$4 to vertical coordinates, draws a smooth curve with cubic Bézier interpolation, and marks the current time with a moving dot.

5. Evaluation, qualitative demonstration, and interpretive claims

The paper is primarily methodological rather than empirical. It does not provide a formal quantitative evaluation section, a benchmark dataset, explicit baselines, or standard retrieval or prediction metrics such as Pearson correlation, Spearman correlation, precision, or recall. It also does not present ablation studies. Evaluation is therefore demonstrative rather than comparative (Luiz, 6 Aug 2025).

The main qualitative example is a math lecture with a theme roughly described as “Explanation of an example about calculating integrals using a circle.” In the illustrated output, high contextual relevance values correspond to segments in which the lecturer is actively explaining that example, while lower values correspond to tangents, pauses, or off-topic segments. The paper further distinguishes how the two stages behave in such a case. Direct relevance $I_i$5 may spike when the segment contains locally salient lexical or visual cues such as “circle” or “integral.” Contextual relevance $I_i$6, by contrast, may preserve elevated scores across a coherent explanatory passage or reduce isolated mentions that do not fit the surrounding history.

The interpretive value claimed for the framework lies in this distinction between local matching and narrative positioning. A segment may be only moderately locally relevant but still receive a high contextual score if it resolves a buildup or materially advances the user-defined theme. Conversely, a repetitive or isolated locally relevant segment may be down-weighted when it does not move the thematic narrative forward.

The paper does not report a user study, inter-annotator agreement, or structured feedback regarding interpretability. It argues instead that the visual overlay makes the output intuitively understandable. That claim should therefore be read as a design rationale rather than as a validated human-factors result.

6. Limitations, assumptions, and future extensions

The framework has several explicit limitations. It depends on ASR quality, so transcription errors arising from accents, noise, or domain jargon can mislead the LLM. It inherits LLM-specific problems, including hallucinations, biases in relevance estimation, and prompt-sensitive consistency. It uses fixed windowing, which may not align with natural scene or topic boundaries, and it truncates score history to $I_i$7, which can discard long-range dependencies. Most importantly, it lacks human-annotated thematic relevance labels and any formal empirical validation (Luiz, 6 Aug 2025).

The system also assumes that $I_i$8 is sufficiently clear and specific. The paper states that vague descriptions may produce noisy scores. Since the semantic criterion is entirely anchored in the user’s natural-language formulation, ambiguity in $I_i$9 can propagate through both scoring stages.

Several extensions are proposed. These include adaptive windowing, such as shot detection or content-based segmentation instead of fixed $\mathcal{C}_{S,i}$0; advanced temporal models, including RNNs or Transformers over segment embeddings rather than reliance solely on in-prompt temporal reasoning; interactive refinement, in which users provide feedback on scores and prompts or weightings are updated; additional modalities, such as on-screen text, subtitles, gestures, or more fine-grained visual semantics; and interpretability tools that explain which phrases or frames drove the assigned scores.

Two common misconceptions are directly addressed by the paper’s design. First, LUST is not a supervised significance predictor trained on labeled examples; it is a rule-based orchestration of pre-trained models. Second, its “hierarchy” does not denote separate scene-level and video-level learned modules. The hierarchy is the direct-to-contextual progression in scoring.

7. Relation to adjacent user-centric LLM scoring research

LUST is positioned at the intersection of video understanding and summarization, multi-modal analysis, and LLM applications in multimedia. Its distinctive emphasis is user-defined, high-level thematic tracking rather than predefined salience or generic summarization criteria. The paper identifies four novelties: user-driven thematic tracking through $\mathcal{C}_{S,i}$1, hierarchical LLM-based scoring, joint use of frame images and speech transcripts in Stage 1, and narrative modeling through prompt-based score history in Stage 2 (Luiz, 6 Aug 2025).

A related user-centric line of work is CLUE, a cascade LLM-based usefulness evaluation framework for search documents. CLUE treats usefulness as an ordinal, user-centered construct conditioned on query, task, context, and behavioral evidence, and it uses a cascading judgment structure with multi-voter stabilization to generate multi-level usefulness labels (Wang et al., 10 Jun 2025). Although the domain is document retrieval rather than video, the methodological parallel is clear: both systems rely on pre-trained LLMs as judges of subjective, task-conditioned significance rather than as generators of text.

This suggests a broader research trend in which LLMs are used as scoring engines for user-perceived importance. In CLUE, the conditioning signals are task description, document content, click sequence, dwell time, and session context; in LUST, they are the reference summary $\mathcal{C}_{S,i}$2, representative frames, ASR speech context, and score history. The comparison is not an identity claim—the two frameworks solve different problems and use different input structures—but it clarifies the conceptual space into which LUST falls: user-centric evaluation by prompt-conditioned LLM judgment.

Within that space, LUST’s contribution is specifically temporal and multimedia. Rather than assigning a single usefulness label to a document, it produces a segment-wise contextual relevance trajectory over a video, intended to surface evolving thematic significance as the narrative unfolds.