HiMu: Training-Free Long-Form Video QA

• HiMu is a training-free framework that decomposes complex video queries into hierarchical logic trees, leveraging lightweight experts (e.g., CLIP, ASR) for precise temporal analysis.
• It introduces PASS (Peak-And-Spread Selection) to dynamically select diverse, high-value frames, ensuring robust temporal coherence under strict frame budgets.
• HiMu advances the efficiency-accuracy Pareto front in long-form video QA by preserving cross-modal bindings and precise temporal relations while reducing computational overhead.

HiMu is a training-free framework for long-form video question answering that performs hierarchical multimodal frame selection under strict frame budgets. It addresses the selection bottleneck faced by large vision-LLMs (LVLMs) on long videos by replacing flat similarity scoring and iterative agentic search with a single text-only LLM decomposition of the query into a hierarchical logic tree, lightweight expert evaluation across vision and audio, fuzzy-logic temporal composition, and a peak-diversity frame selector. In the reported evaluations, HiMu advances the efficiency-accuracy Pareto front on Video-MME, LongVideoBench, and HERBench-Lite, especially at small budgets such as $K=16$ frames (Ben-Ami et al., 19 Mar 2026).

1. Problem setting and design rationale

Long-form video QA requires reasoning over temporal contexts that may span minutes to hours, while LVLMs remain bounded by finite context windows. Under this constraint, the decisive issue is not only answer generation but whether the relevant evidence is present in the selected frames. HiMu is designed for precisely this setting: query-aware, multimodal selection over long videos, with explicit handling of temporal ordering and cross-modal bindings (Ben-Ami et al., 19 Mar 2026).

The framework is motivated by a sharp trade-off in prior selectors. Similarity-based selectors are efficient, but they reduce a compositional query to a single dense representation and score each frame with a single similarity value; this collapses sub-event ordering and cross-modal relations. Agent-based systems recover such structure by repeated LVLM-mediated reasoning, verification, and tool use, but incur much higher latency. HiMu occupies the middle ground by moving compositional reasoning into a single text-only parse and then evaluating lightweight experts over cached multimodal features, without iterative LVLM-in-the-loop selection.

A central misconception is that long-video selection can be treated as a pure retrieval problem. HiMu is explicitly built around the opposite view: many questions require decomposing the query into atomic predicates, aligning heterogeneous modalities, and enforcing temporal relations such as sequence or adjacency before any final answering call. This is why its core object is not a frame embedding but a continuous satisfaction curve $T(t)\in[0,1]$ over the sampled video timeline (Ben-Ami et al., 19 Mar 2026).

2. Hierarchical logic tree and expert routing

HiMu begins with a single text-only LLM call that parses the question, and multiple-choice options when present, into a hierarchical logic tree in JSON. Internal nodes are logical or temporal operators, and leaves are atomic predicates of the form $\ell=(\text{expert}, \text{query})$. The leaf expert is chosen from $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$, and the query is a short atomic phrase such as ovd("red car") or asr("reaction") (Ben-Ami et al., 19 Mar 2026).

The internal operators are And, Or, Seq, and RightAfter. And encodes co-occurrence in time, Or disjunction, Seq ordered sub-events, and RightAfter tight temporal adjacency. For multiple-choice questions, the paper uses the pattern And(shared_context, Or(option_1, …, option_n)), with shared components factored out before routing. A worked example is “After the doorbell rings, who opens the door?” with options “A man” and “A woman”: [CLAP](https://www.emergentmind.com/topics/contrastive-latent-action-pretraining-clap)("doorbell ringing") supplies the cause, while [CLIP](https://www.emergentmind.com/topics/contrastive-language-image-pre-trained-clip-models)("opens the door") and [OVD](https://www.emergentmind.com/topics/open-vocabulary-object-detection-ovd)("man"/"woman") describe the effect, and Seq(cause, effect) imposes the temporal order.

The routing policy is modality-specific rather than generic. Physical objects and people are sent to open-vocabulary detection, actions and scenes to CLIP, on-screen text to OCR, spoken references to ASR, environmental sounds to CLAP, and object-conditioned actions to an AND composition of OVD and CLIP. This decomposition preserves cross-modal structure that would be lost in a single-vector representation.

Expert	Role	Signal source
CLIP	Actions, scenes, states	Per-frame image-text similarity
OVD	Physical objects, people	Detection confidence
OCR	On-screen text	Text recognition with fuzzy matching
ASR	Spoken references, dialogue	Timestamped transcript matching
CLAP	Environmental sounds	Audio-text similarity

The experts are deliberately lightweight. CLIP uses CLIP-dfn ViT-L/14, OVD uses YOLO-World v2, OCR uses docTR, ASR uses faster-whisper large-v3-turbo, and CLAP uses LAION-CLAP. Each leaf produces a raw temporal signal $u_i(t)$ over sampled frames, which is later normalized, smoothed, and composed (Ben-Ami et al., 19 Mar 2026).

3. Signal processing and fuzzy temporal composition

For each leaf $i$, HiMu converts the raw score $u_i(t)$ into a calibrated signal in $[0,1]$ using robust median/MAD sigmoid normalization: $$\tilde{u}_i(t)=\sigma\!\left(\gamma \cdot \frac{u_i(t)-\mathrm{med}(u_i)}{\mathrm{MAD}(u_i)+\delta}\right).$$ When multiple leaves share an expert, the median and MAD are computed jointly across their scores to preserve relative confidence magnitudes between predicates. This prevents independent calibration from destroying inter-predicate comparability (Ben-Ami et al., 19 Mar 2026).

The normalized signals are then temporally smoothed by modality-matched Gaussian kernels: $$\hat{u}_i(t)=\sum_{t'=1}^{T}\tilde{u}_i(t')\,\mathcal{G}(t-t';\sigma_m), \qquad \mathcal{G}(\Delta;\sigma)=\frac{1}{\sqrt{2\pi}\sigma}\exp\!\left(-\frac{\Delta^2}{2\sigma^2}\right).$$ Visual experts use narrow kernels, while audio experts use wider kernels. The purpose is not mere denoising: smoothing aligns bandwidth and latency differences across modalities, increasing temporal overlap for later conjunctions.

Composition proceeds bottom-up on the logic tree by continuous fuzzy operators. For logical conjunction and disjunction, HiMu uses a product t-norm and a probabilistic sum: $T(t)\in[0,1]$0 No [NOT](https://www.emergentmind.com/topics/neural-organ-transplantation-not) operator is used in the reported system.

The temporal operators are the distinctive part of the method. For a sequence over children $T(t)\in[0,1]$1, HiMu defines

$T(t)\in[0,1]$2

and then

$T(t)\in[0,1]$3

This permits frames associated with each ordered step to activate, rather than only the terminal event. RightAfter is defined through exponentially decayed cause-effect coupling with decay rate $T(t)\in[0,1]$4, favoring short delays. The root node produces the final satisfaction curve $T(t)\in[0,1]$5, which is deterministic, weight-free except for $T(t)\in[0,1]$6, $T(t)\in[0,1]$7, and $T(t)\in[0,1]$8, and interpretable because each selected frame retains its per-leaf activations (Ben-Ami et al., 19 Mar 2026).

A plausible implication is that HiMu is best understood as a programmatic selector rather than a learned scorer: the query is compiled into a temporal logic-like structure, and the resulting curve is the execution trace over the video.

4. PASS frame selection and runtime profile

HiMu converts the satisfaction curve into a frame set through PASS, short for Peak-And-Spread Selection. PASS is designed to avoid the failure mode of naive top-$T(t)\in[0,1]$9 ranking, which tends to concentrate the budget on a single temporal region and misses both diverse events and short-range motion context (Ben-Ami et al., 19 Mar 2026).

PASS has three stages. It first detects local maxima of $\ell=(\text{expert}, \text{query})$0, sorts them, and selects $\ell=(\text{expert}, \text{query})$1 peaks subject to a minimum inter-peak distance $\ell=(\text{expert}, \text{query})$2. It then adds $\ell=(\text{expert}, \text{query})$3 high-scoring neighbors inside a window $\ell=(\text{expert}, \text{query})$4 around each peak. Finally, it greedily fills any remaining slots with the highest-scoring unselected frames. In the reported implementation,

$\ell=(\text{expert}, \text{query})$5

The computational profile is one of the main contributions. Composition over $\ell=(\text{expert}, \text{query})$6 leaves and $\ell=(\text{expert}, \text{query})$7 sampled frames is $\ell=(\text{expert}, \text{query})$8; smoothing is $\ell=(\text{expert}, \text{query})$9 with finite-support kernels, or $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$0 via FFT if needed; peak detection is linear. In measurement, scoring, composition, and PASS take less than $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$1 seconds CPU time. The dominant per-query costs are the single LLM parse and the per-query open-vocabulary detection pass.

For a 10-minute video at 1 FPS, giving 600 candidate frames with $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$2, and using $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$3A100 GPUs, the reported latency breakdown is:

System type or component	Latency
Similarity-based selectors	1.8–3.0 s E2E
Structured detector systems	$$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$4–13.3 s
Agentic/per-frame LVLM systems	16–60 s
HiMu first query	13.3 s E2E
HiMu subsequent queries	9.0 s amortized

HiMu’s preprocessing branch, which caches CLIP, OCR, ASR, and CLAP features, takes 4.3 s wall clock. Per query, the LLM parse is 6.7 s, OVD is 2.1 s, and scoring/composition/PASS is under 0.3 s. This caching regime is critical: HiMu is not as cheap as pure similarity selection on a first query, but it is substantially less expensive than agentic search while retaining explicit compositional structure (Ben-Ami et al., 19 Mar 2026).

5. Empirical performance and ablations

The reported evaluations cover Video-MME, LongVideoBench$$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$5, and HERBench-Lite. Under controlled comparisons at $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$6 frames with Qwen3-VL 8B, HiMu reaches 73.22% on Video-MME overall, 64.19% on LongVideoBench$$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$7, and 43.22% on HERBench-Lite. The corresponding values for Uniform are 66.36%, 55.74%, and 41.70%; for BOLT, 68.74%, 54.55%, and 42.20%; for AKS, 67.98%, 57.14%, and 40.25%; and for T$$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$8, 69.77%, 57.49%, and 39.10% (Ben-Ami et al., 19 Mar 2026).

The generalization experiments across downstream LVLMs retain the same selection method and change only the answering model. On Video-MME overall at $$\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}$$9, the reported Uniform $u_i(t)$0 HiMu gains are 66.63% $u_i(t)$1 71.35% for InternVL-3.5-8B, 62.57% $u_i(t)$2 67.09% for Qwen2.5-VL 7B, 63.55% $u_i(t)$3 67.65% for LLaVA-OV-1.5-8B, 62.99% $u_i(t)$4 68.28% for Gemma-3-12B, 68.95% $u_i(t)$5 76.11% for Gemini-2.5-Flash, and 73.81% $u_i(t)$6 78.18% for GPT-4o.

The asymmetric budget comparisons are especially notable. On a Video-MME subset with Qwen2.5-VL 7B, HiMu at 16 frames reaches 69.70%, compared with 62.57% for VideoZoomer at 128 frames and 65.20% for VideoChat-A1 at 512 frames. With GPT-4o, HiMu at 16 frames reaches 78.18%, surpassing VSLS at 32 frames with 67.09% and VideoChat-A1 at 384 frames with 62.99%. The paper further states that on Video-MME, HiMu achieves 78.18% accuracy with GPT-4o while requiring roughly $u_i(t)$7 fewer FLOPs than state-of-the-art agentic systems operating at 32–512 frames.

The ablations isolate the source of these gains. Replacing the tree with flat fusion causes a $u_i(t)$8 percentage-point drop, the largest single decline, indicating that explicit structure matters more than simply having access to the expert signals. Leave-one-out expert ablations show $u_i(t)$9 pp without ASR, $i$0 pp without CLIP, $i$1 pp without CLAP, $i$2 pp without OCR, and $i$3 pp without OVD. This makes audio a first-class modality rather than an auxiliary cue. PASS adds $i$4 pp over vanilla top-$i$5 on a 50% subset, and the maximum deviation across smoothing bandwidths, $i$6, and $i$7 is $i$8 pp, suggesting moderate hyperparameter robustness (Ben-Ami et al., 19 Mar 2026).

A recurring empirical pattern is that HiMu is most advantageous under tight budgets. The paper reports that HiMu at $i$9 on Video-MME, with 73.22%, outperforms Uniform at $u_i(t)$0, with 71.68%, indicating that the benefit is not merely better ranking within a large budget but a substantial reduction in wasted frames.

6. Implementation details, limitations, and extensions

HiMu is explicitly training-free. It uses robust calibration by median/MAD normalization and fixed fuzzy operators, rather than learned score calibration. The default hyperparameters are $u_i(t)$1 and $u_i(t)$2 for normalization; $u_i(t)$3, $u_i(t)$4, $u_i(t)$5, $u_i(t)$6, and $u_i(t)$7 for smoothing; and $u_i(t)$8 for RightAfter. Feature caching is performed once per video for CLIP, OCR, ASR, and CLAP, while OVD runs per query (Ben-Ami et al., 19 Mar 2026).

The system prompt constrains the parser to a JSON schema with LEAF, AND, OR, SEQ, and RIGHT_AFTER nodes. The reported spread across different decomposition LLMs is within 0.95%, which the paper attributes to the stability induced by the JSON-constrained prompt. This suggests that parser quality matters, but within the tested set it is not the dominant source of variance.

The limitations are largely structural rather than purely computational. ASR and OCR noise can degrade leaf signals, especially under multilingual, low-resource, or low-legibility conditions. CLAP may misfire in complex audio scenes. If the text-only parser constructs shallow or malformed trees, predicates may be misrouted. Open-vocabulary detection may miss rare categories or fine-grained attributes. Temporal misalignment between transcription chunks and frames is only partially mitigated by smoothing. Finally, Seq and RightAfter do not express very intricate temporal constraints such as deeply nested temporal dependencies or richer operators like until and within (Ben-Ami et al., 19 Mar 2026).

The paper presents several extension directions. HiMu can be paired with token compression or pruning methods such as LongVU and FastV for still longer videos, augmented with additional experts such as visual grounding, motion descriptors, face or identity recognition, or richer audio-event classifiers, and used as a guide for hybrid agentic systems that restrict expensive LVLM tool calls to regions near the satisfaction peaks. Another proposed direction is learned score calibration that preserves the existing compositional scheme, as well as richer program synthesis with more expressive temporal logic operators.

Taken as a system design, HiMu replaces end-to-end learned frame scoring with a modular pipeline whose main representational object is a hierarchical logic tree evaluated against cached multimodal evidence. Its significance lies not only in the reported accuracy gains, but in showing that explicit compositional structure can shift the efficiency-accuracy Pareto front for long-video QA under tight context budgets (Ben-Ami et al., 19 Mar 2026).