SLFG: Scene-Localized Frame Grouping

• SLFG is a training-free method that localizes semantically coherent scenes instead of isolated frames to improve long-video question answering.
• It employs a language-mediated abstraction and dynamic frame reassembly process to overcome the limitations of traditional keyframe selection.
• Empirical benchmarks show significant accuracy gains by providing context-rich scene-level reasoning in lengthy videos.

Searching arXiv for the primary SLFG paper and a few related long-video QA works for citation support. SLFG, short for Scene-Localized Frame Grouping, is a training-free, plug-and-play method for long video question answering that redefines the basic unit of retrieval from isolated frames to semantically coherent scenes. It was introduced to address the failure mode of multimodal LLMs (MLLMs) on hour-scale videos, where visual token budgets and context windows typically permit only tens to hundreds of frames, while the source video may contain tens of thousands. Rather than optimizing for “finding a single frame,” SLFG localizes a relevant scene, reconstructs a compact temporally ordered set of frames around it, and feeds that set to an unchanged base video-LLM for final reasoning (Yang et al., 5 Aug 2025).

1. Conceptual definition and problem setting

SLFG was proposed in the context of long video question answering, where feeding all frames to an MLLM is computationally and contextually infeasible. The method starts from the observation that existing training-free pipelines often emphasize sparse keyframe selection aligned to a query, but that this framing is poorly matched to questions requiring integrated understanding of a coherent scene rather than an isolated moment. In this formulation, a scene is treated as the fundamental unit of reasoning, because scene-level semantics preserve multiple interacting elements and the short temporal substrate needed for causality and narrative (Yang et al., 5 Aug 2025).

This design is explicitly motivated by a cognitive analogy: people skim a long video, detect scene transitions, and then zoom in on the relevant scene’s details. SLFG formalizes that intuition through a language-mediated abstraction-and-retrieval pipeline followed by rule-based dynamic frame reassembly. The final “scene frames” are not a new visual primitive inside the model architecture; they are the reassembled, temporally ordered frames from the top-ranked frame groups after merging and window extension, chosen to fill the MLLM’s visual context budget.

A common misunderstanding is to treat SLFG as a temporal segmentation model in the conventional sense. It is not. The localization mechanism is text-centric rather than boundary-centric: no explicit shot boundary detection, temporal segmentation via optimization, audio cues, or supervised labels are used. Scene relevance is inferred through similarity between language descriptions of grouped frames and a language description of the question’s target scene.

2. SceneQA and the LVSQA evaluation setting

SLFG is tied to a reformulation of long-video evaluation called SceneQA. SceneQA requires a model to localize a semantically coherent scene and answer using fine-grained visual cues such as character actions, object states, interactions, and short-term dynamics. Its purpose is to measure detailed perception and integrated scene reasoning, rather than isolated keyframe lookup (Yang et al., 5 Aug 2025).

To support this setting, the same work introduces LVSQA, a dataset built from 100 selected LVBench videos, each over 30 minutes in duration. Videos heavily reliant on subtitles were filtered out in order to concentrate evaluation on visual content. The dataset contains 500 high-quality question-answer pairs generated with MLLM assistance and then extensively refined by humans and filtered by experts. Two question types are central:

• Scene-based Detail Recognition: precise perception of local cues within a scene.
• Scene-based Causal Reasoning: inference of cause-effect relationships within a coherent scene.

Accuracy is reported as the evaluation metric. The dataset design is significant because it targets scene-level detail recognition and scene-level causal reasoning in very long videos, thereby exposing failure modes that can be hidden in “needle-in-a-haystack” benchmarks.

3. Pipeline and mathematical formulation

SLFG operates in four stages: frame grouping and description, scene generation, scene localization, and dynamic frame reorganization. The entire method is external to the base model and requires no modification of the video encoder or LLM backend (Yang et al., 5 Aug 2025).

In the first stage, the video is densely sampled at a fixed interval $\Delta t$:

$$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$

Sampled frames are organized into fixed-granularity groups of size $N$:

$$G_k = \{f_k, f_{k+1}, \ldots, f_{k+N-1}\}, \quad k = 0,1,2,\ldots$$

An MLLM $\mathcal{M}$ then produces fine-grained textual descriptions for each group:

$$D_k = \mathcal{M}(G_k) = \{\text{object}_i, \text{action}_j, \text{relation}_{ij}, \ldots\}$$

In the second stage, an LLM $\mathcal{L}$ aggregates these detailed descriptions into scene-level summaries:

$$S_m = \mathcal{L}(D_1, D_2, \ldots, D_k), \quad m = 1,2,\ldots,M$$

This language abstraction reduces redundancy and surfaces coherent scene semantics. In the implementation reported in the paper, Qwen2.5-7B-Instruct is used for scene generation, while BGE-M3 provides text embeddings for retrieval.

The third stage localizes question-relevant scenes by embedding the scene summaries and a question scene description, then computing cosine similarity:

$$\text{Sim}(A_i, B) = \cos\big(\text{emb}(A_i),\; \text{emb}(B)\big)$$

Group-level relevance is defined by the maximum scene-summary match within a group:

$\text{Score}_G = \max_i \text{Sim}(A_i, B)$

Groups are ranked by $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$0. The fourth stage then reorganizes frames dynamically. Adjacent groups are merged if their score difference is below a threshold, 10% by default, and low-relevance tails are discarded. If the selected groups occupy fewer than the base model’s maximum frame budget $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$1, the remaining budget is distributed across groups:

$$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$2

where $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$3 is the remaining frame budget. If $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$4, the remainder is allocated proportionally using group time spans, with extra frames sampled before and after each selected window at interval $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$5. Final reasoning is then performed by the original MLLM on the reassembled input:

$$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$6

The important architectural point is that weighting and attention are entirely rule-based at this stage: there is no learned reweighting or differentiable attention in the frame aggregation step.

4. Integration with MLLMs and empirical behavior

SLFG is explicitly designed as a compatibility layer for existing video-LLMs. The final reassembled frames are provided through the base model’s native visual input interface, with native inference prompts, and without fine-tuning or architectural revision. The reported demonstrations use LLaVA-OneVision with a 32-frame context and LLaVA-Video with a 64-frame context (Yang et al., 5 Aug 2025).

The paper reports consistent gains on several long-video benchmarks:

Benchmark	LLaVA-OneVision	LLaVA-Video
VideoEval-Pro	40.1% → 42.6% (+2.5)	45.8% → 48.7% (+2.9)
LVBench	38.7% → 42.5% (+3.8)	41.8% → 45.3% (+3.5)
VideoMME (w/o long)	43.6% → 46.9% (+3.3)	47.6% → 49.5% (+1.9)
LVSQA	56.2% → 61.0%	59.8% → 63.4% (+3.6)

On VideoEval-Pro macro-categories, SLFG attains 54.6 in Local Perception, 48.3 in Local Reasoning, 33.2 in Holistic Perception, 39.6 in Holistic Reasoning, and 48.7 overall average, compared with 38.3 overall for QuoTA and 47.7 for AKS. The reported interpretation is that keyframe retrieval is strong at “finding,” whereas scene-centric grouping more effectively supports integrated reasoning.

This distinction matters because SLFG does not reject frame-level retrieval outright; rather, it changes the analysis unit. A plausible implication is that the method is particularly useful where the answer depends on temporally adjacent evidence distributed across a scene, not merely on the presence of a single discriminative frame.

5. Hyperparameters, efficiency, and computational profile

The reported implementation uses dense sampling at 10 seconds per frame, frame-group granularity $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$7, and a default merge threshold of 10%, with ablations at 5% and 20%. For context size, the method uses 32 frames for LLaVA-OneVision and 64 frames for LLaVA-Video. Hardware is reported as 80G A800 GPUs (Yang et al., 5 Aug 2025).

The main computational burden lies in precomputation. Let video length be $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$8, sampled every $$f_i = F(t_0 + i \cdot \Delta t), \quad i = 0,1,2,\ldots$$9, so that the number of sampled frames is approximately $N$0, and the number of groups is approximately $N$1. The method performs $N$2 MLLM calls for group descriptions and $N$3 LLM processing for scene abstraction. Per question, localization costs $N$4 for embedding similarity, $N$5 for sorting, and $N$6 for reassembly over selected groups. Memory at final inference remains bounded by the base model’s frame budget $N$7 regardless of video length.

Because group descriptions and scene summaries are cached once per video, the average cost per question decreases as the number of questions per video increases. The paper gives the amortized form

$N$8

which explains why average per-question inference time approaches direct inference as $N$9 grows and can continue to decrease through amortization. In the authors’ efficiency analysis, many videos contain about five questions, while a ten-question setting is simulated by repeated inference.

Ablation results on LVSQA with LLaVA-Video further support the reassembly design: Top-1 only gives 61.2, Top-$$G_k = \{f_k, f_{k+1}, \ldots, f_{k+N-1}\}, \quad k = 0,1,2,\ldots$$0 without merging or padding gives 63.0, a 5% threshold gives 63.2, the default 10% threshold gives 63.4, and a 20% threshold returns to 63.0. This indicates that both dynamic merging and window extension contribute, with 10% the most effective among the tested settings.

6. Relation to prior approaches, limitations, and scope

SLFG is positioned against keyframe-selection approaches such as AKS, VideoTree, and Q-Frame, as well as query-oriented token assignment such as QuoTA. Those methods operate primarily at frame granularity. The central claim of SLFG is not that frame retrieval is obsolete, but that frame-only sparsification can discard contextual continuity needed for scene-level reasoning (Yang et al., 5 Aug 2025).

Several limitations are explicit. First, localization depends on language mediation: the quality of MLLM-generated descriptions and LLM abstractions directly affects retrieval. Second, merging decisions rely on a simple percentage threshold, which may under-merge or over-merge when score distributions are nonuniform. Third, the evaluation setting is visually focused: no audio, OCR, or subtitle-dependent reasoning is integrated, and LVSQA deliberately filters out subtitle-reliant videos. Fourth, the fixed sampling interval $$G_k = \{f_k, f_{k+1}, \ldots, f_{k+N-1}\}, \quad k = 0,1,2,\ldots$$1 and fixed group size $$G_k = \{f_k, f_{k+1}, \ldots, f_{k+N-1}\}, \quad k = 0,1,2,\ldots$$2 may be suboptimal for highly dynamic scenes. Fifth, embedding quality is tied to BGE-M3; domain shift or multilingual content may affect similarity quality.

These constraints clarify what SLFG is and is not. It is a training-free external orchestration layer for long-video reasoning under strict context budgets. It is not an end-to-end learned temporal segmentation system, not a multimodal retrieval stack that incorporates audio or OCR, and not a replacement for architecture-level changes to video-LLMs. This suggests a broader research direction: scene-level localization may be combined with richer multimodal signals or adaptive grouping strategies, while preserving the plug-and-play property that makes SLFG operationally attractive.

In that sense, SLFG’s main contribution is methodological reorientation. By moving the retrieval unit from frames to scenes, it provides a concrete recipe for scene-based detail perception and scene-based causal reasoning in long videos, and it does so while leaving the underlying MLLM unchanged (Yang et al., 5 Aug 2025).