---
title: 'HiMu: Training-Free Long-Form Video QA'
url: https://www.emergentmind.com/topics/himu
type: topic
---

# HiMu: Training-Free Long-Form Video QA

HiMu is a training-free framework for long-form video question answering that performs hierarchical multimodal frame selection under strict frame budgets. It addresses the selection bottleneck faced by large vision-language models (LVLMs) on long videos by replacing flat similarity scoring and iterative agentic search with a single text-only LLM decomposition of the query into a hierarchical logic tree, lightweight expert evaluation across vision and audio, fuzzy-logic temporal composition, and a peak-diversity frame selector. In the reported evaluations, HiMu advances the efficiency-accuracy Pareto front on Video-MME, LongVideoBench, and HERBench-Lite, especially at small budgets such as \(K=16\) frames [2603.18558].

## 1. Problem setting and design rationale

Long-form video QA requires reasoning over temporal contexts that may span minutes to hours, while LVLMs remain bounded by finite context windows. Under this constraint, the decisive issue is not only answer generation but whether the relevant evidence is present in the selected frames. HiMu is designed for precisely this setting: query-aware, multimodal selection over long videos, with explicit handling of temporal ordering and cross-modal bindings [2603.18558].

The framework is motivated by a sharp trade-off in prior selectors. Similarity-based selectors are efficient, but they reduce a compositional query to a single dense representation and score each frame with a single similarity value; this collapses sub-event ordering and cross-modal relations. Agent-based systems recover such structure by repeated LVLM-mediated reasoning, verification, and tool use, but incur much higher latency. HiMu occupies the middle ground by moving compositional reasoning into a single text-only parse and then evaluating lightweight experts over cached multimodal features, without iterative LVLM-in-the-loop selection.

A central misconception is that long-video selection can be treated as a pure retrieval problem. HiMu is explicitly built around the opposite view: many questions require decomposing the query into atomic predicates, aligning heterogeneous modalities, and enforcing temporal relations such as sequence or adjacency before any final answering call. This is why its core object is not a frame embedding but a continuous satisfaction curve \(T(t)\in[0,1]\) over the sampled video timeline [2603.18558].

## 2. Hierarchical logic tree and expert routing

HiMu begins with a single text-only LLM call that parses the question, and multiple-choice options when present, into a hierarchical logic tree in JSON. Internal nodes are logical or temporal operators, and leaves are atomic predicates of the form \(\ell=(\text{expert}, \text{query})\). The leaf expert is chosen from \(\{\text{CLIP}, \text{OVD}, \text{OCR}, \text{ASR}, \text{CLAP}\}\), and the query is a short atomic phrase such as `ovd("red car")` or `asr("reaction")` [2603.18558].

The internal operators are `And`, `Or`, `Seq`, and `RightAfter`. `And` encodes co-occurrence in time, `Or` disjunction, `Seq` ordered sub-events, and `RightAfter` tight temporal adjacency. For multiple-choice questions, the paper uses the pattern `And(shared_context, Or(option_1, …, option_n))`, with shared components factored out before routing. A worked example is “After the doorbell rings, who opens the door?” with options “A man” and “A woman”: `CLAP("doorbell ringing")` supplies the cause, while `CLIP("opens the door")` and `OVD("man"/"woman")` describe the effect, and `Seq(cause, effect)` imposes the temporal order.

The routing policy is modality-specific rather than generic. Physical objects and people are sent to open-vocabulary detection, actions and scenes to CLIP, on-screen text to OCR, spoken references to ASR, environmental sounds to CLAP, and object-conditioned actions to an `AND` composition of OVD and CLIP. This decomposition preserves cross-modal structure that would be lost in a single-vector representation.

| Expert | Role | Signal source |
|---|---|---|
| CLIP | Actions, scenes, states | Per-frame image-text similarity |
| OVD | Physical objects, people | Detection confidence |
| OCR | On-screen text | Text recognition with fuzzy matching |
| ASR | Spoken references, dialogue | Timestamped transcript matching |
| CLAP | Environmental sounds | Audio-text similarity |

The experts are deliberately lightweight. CLIP uses CLIP-dfn ViT-L/14, OVD uses YOLO-World v2, OCR uses docTR, ASR uses faster-whisper large-v3-turbo, and CLAP uses LAION-CLAP. Each leaf produces a raw temporal signal \(u_i(t)\) over sampled frames, which is later normalized, smoothed, and composed [2603.18558].

## 3. Signal processing and fuzzy temporal composition

For each leaf \(i\), HiMu converts the raw score \(u_i(t)\) into a calibrated signal in \([0,1]\) using robust median/MAD sigmoid normalization:
\[
\tilde{u}_i(t)=\sigma\!\left(\gamma \cdot \frac{u_i(t)-\mathrm{med}(u_i)}{\mathrm{MAD}(u_i)+\delta}\right).
\]
When multiple leaves share an expert, the median and MAD are computed jointly across their scores to preserve relative confidence magnitudes between predicates. This prevents independent calibration from destroying inter-predicate comparability [2603.18558].

The normalized signals are then temporally smoothed by modality-matched Gaussian kernels:
\[
\hat{u}_i(t)=\sum_{t'=1}^{T}\tilde{u}_i(t')\,\mathcal{G}(t-t';\sigma_m),
\qquad
\mathcal{G}(\Delta;\sigma)=\frac{1}{\sqrt{2\pi}\sigma}\exp\!\left(-\frac{\Delta^2}{2\sigma^2}\right).
\]
Visual experts use narrow kernels, while audio experts use wider kernels. The purpose is not mere denoising: smoothing aligns bandwidth and latency differences across modalities, increasing temporal overlap for later conjunctions.

Composition proceeds bottom-up on the logic tree by continuous fuzzy operators. For logical conjunction and disjunction, HiMu uses a product t-norm and a probabilistic sum:
\[
\text{And}(A,B)(t)=A(t)\cdot B(t),
\qquad
\text{Or}(A,B)(t)=A(t)+B(t)-A(t)B(t).
\]
No `NOT` operator is used in the reported system.

The temporal operators are the distinctive part of the method. For a sequence over children \(u_1,\dots,u_L\), HiMu defines
\[
H_j(t)=\max_{s<t}u_j(s),\qquad F_j(t)=\max_{s>t}u_j(s),
\]
and then
\[
\text{Seq}(t)=\max_{\ell\in\{1,\dots,L\}}
\left[
u_\ell(t)\cdot
\prod_{j<\ell}H_j(t)\cdot
\prod_{j>\ell}F_j(t)
\right].
\]
This permits frames associated with each ordered step to activate, rather than only the terminal event. `RightAfter` is defined through exponentially decayed cause-effect coupling with decay rate \(\kappa\), favoring short delays. The root node produces the final satisfaction curve \(T(t)\in[0,1]\), which is deterministic, weight-free except for \(\kappa\), \(\gamma\), and \(\sigma\), and interpretable because each selected frame retains its per-leaf activations [2603.18558].

A plausible implication is that HiMu is best understood as a programmatic selector rather than a learned scorer: the query is compiled into a temporal logic-like structure, and the resulting curve is the execution trace over the video.

## 4. PASS frame selection and runtime profile

HiMu converts the satisfaction curve into a frame set through PASS, short for Peak-And-Spread Selection. PASS is designed to avoid the failure mode of naive top-\(K\) ranking, which tends to concentrate the budget on a single temporal region and misses both diverse events and short-range motion context [2603.18558].

PASS has three stages. It first detects local maxima of \(T(t)\), sorts them, and selects \(N_p\) peaks subject to a minimum inter-peak distance \(\Delta\). It then adds \(N_n\) high-scoring neighbors inside a window \(w\) around each peak. Finally, it greedily fills any remaining slots with the highest-scoring unselected frames. In the reported implementation,
\[
N_p=\lfloor \sqrt{K}\rfloor,\qquad
N_n=\lfloor \sqrt{K}/2\rfloor,\qquad
w=\lfloor \sqrt{K}\rfloor,\qquad
\Delta=\lfloor \sqrt{K}\rfloor.
\]

The computational profile is one of the main contributions. Composition over \(L\) leaves and \(T\) sampled frames is \(O(L\cdot T)\); smoothing is \(O(T\cdot w_k)\) with finite-support kernels, or \(O(T\log T)\) via FFT if needed; peak detection is linear. In measurement, scoring, composition, and PASS take less than \(0.3\) seconds CPU time. The dominant per-query costs are the single LLM parse and the per-query open-vocabulary detection pass.

For a 10-minute video at 1 FPS, giving 600 candidate frames with \(K=16\), and using \(8\times\)A100 GPUs, the reported latency breakdown is:

| System type or component | Latency |
|---|---|
| Similarity-based selectors | 1.8–3.0 s E2E |
| Structured detector systems | \(\approx 13.0\)–13.3 s |
| Agentic/per-frame LVLM systems | 16–60 s |
| HiMu first query | 13.3 s E2E |
| HiMu subsequent queries | 9.0 s amortized |

HiMu’s preprocessing branch, which caches CLIP, OCR, ASR, and CLAP features, takes 4.3 s wall clock. Per query, the LLM parse is 6.7 s, OVD is 2.1 s, and scoring/composition/PASS is under 0.3 s. This caching regime is critical: HiMu is not as cheap as pure similarity selection on a first query, but it is substantially less expensive than agentic search while retaining explicit compositional structure [2603.18558].

## 5. Empirical performance and ablations

The reported evaluations cover Video-MME, LongVideoBench\(_\text{val}\), and HERBench-Lite. Under controlled comparisons at \(K=16\) frames with Qwen3-VL 8B, HiMu reaches 73.22% on Video-MME overall, 64.19% on LongVideoBench\(_\text{val}\), and 43.22% on HERBench-Lite. The corresponding values for Uniform are 66.36%, 55.74%, and 41.70%; for BOLT, 68.74%, 54.55%, and 42.20%; for AKS, 67.98%, 57.14%, and 40.25%; and for T\(^*\), 69.77%, 57.49%, and 39.10% [2603.18558].

The generalization experiments across downstream LVLMs retain the same selection method and change only the answering model. On Video-MME overall at \(K=16\), the reported Uniform \(\rightarrow\) HiMu gains are 66.63% \(\rightarrow\) 71.35% for InternVL-3.5-8B, 62.57% \(\rightarrow\) 67.09% for Qwen2.5-VL 7B, 63.55% \(\rightarrow\) 67.65% for LLaVA-OV-1.5-8B, 62.99% \(\rightarrow\) 68.28% for Gemma-3-12B, 68.95% \(\rightarrow\) 76.11% for Gemini-2.5-Flash, and 73.81% \(\rightarrow\) 78.18% for GPT-4o.

The asymmetric budget comparisons are especially notable. On a Video-MME subset with Qwen2.5-VL 7B, HiMu at 16 frames reaches 69.70%, compared with 62.57% for VideoZoomer at 128 frames and 65.20% for VideoChat-A1 at 512 frames. With GPT-4o, HiMu at 16 frames reaches 78.18%, surpassing VSLS at 32 frames with 67.09% and VideoChat-A1 at 384 frames with 62.99%. The paper further states that on Video-MME, HiMu achieves 78.18% accuracy with GPT-4o while requiring roughly \(10\times\) fewer FLOPs than state-of-the-art agentic systems operating at 32–512 frames.

The ablations isolate the source of these gains. Replacing the tree with flat fusion causes a \(-5.49\) percentage-point drop, the largest single decline, indicating that explicit structure matters more than simply having access to the expert signals. Leave-one-out expert ablations show \(-1.99\) pp without ASR, \(-1.43\) pp without CLIP, \(-1.00\) pp without CLAP, \(-1.04\) pp without OCR, and \(-0.76\) pp without OVD. This makes audio a first-class modality rather than an auxiliary cue. PASS adds \(+0.72\) pp over vanilla top-\(K\) on a 50% subset, and the maximum deviation across smoothing bandwidths, \(\kappa\), and \(\gamma\) is \(\pm 1.04\) pp, suggesting moderate hyperparameter robustness [2603.18558].

A recurring empirical pattern is that HiMu is most advantageous under tight budgets. The paper reports that HiMu at \(K=16\) on Video-MME, with 73.22%, outperforms Uniform at \(K=64\), with 71.68%, indicating that the benefit is not merely better ranking within a large budget but a substantial reduction in wasted frames.

## 6. Implementation details, limitations, and extensions

HiMu is explicitly training-free. It uses robust calibration by median/MAD normalization and fixed fuzzy operators, rather than learned score calibration. The default hyperparameters are \(\gamma=3.0\) and \(\delta=10^{-6}\) for normalization; \(\sigma_{\text{clip}}=0.5\), \(\sigma_{\text{ovd}}=0.5\), \(\sigma_{\text{ocr}}=0.5\), \(\sigma_{\text{asr}}=1.5\), and \(\sigma_{\text{clap}}=2.0\) for smoothing; and \(\kappa=2.0\) for `RightAfter`. Feature caching is performed once per video for CLIP, OCR, ASR, and CLAP, while OVD runs per query [2603.18558].

The system prompt constrains the parser to a JSON schema with `LEAF`, `AND`, `OR`, `SEQ`, and `RIGHT_AFTER` nodes. The reported spread across different decomposition LLMs is within 0.95%, which the paper attributes to the stability induced by the JSON-constrained prompt. This suggests that parser quality matters, but within the tested set it is not the dominant source of variance.

The limitations are largely structural rather than purely computational. ASR and OCR noise can degrade leaf signals, especially under multilingual, low-resource, or low-legibility conditions. CLAP may misfire in complex audio scenes. If the text-only parser constructs shallow or malformed trees, predicates may be misrouted. Open-vocabulary detection may miss rare categories or fine-grained attributes. Temporal misalignment between transcription chunks and frames is only partially mitigated by smoothing. Finally, `Seq` and `RightAfter` do not express very intricate temporal constraints such as deeply nested temporal dependencies or richer operators like `until` and `within` [2603.18558].

The paper presents several extension directions. HiMu can be paired with token compression or pruning methods such as LongVU and FastV for still longer videos, augmented with additional experts such as visual grounding, motion descriptors, face or identity recognition, or richer audio-event classifiers, and used as a guide for hybrid agentic systems that restrict expensive LVLM tool calls to regions near the satisfaction peaks. Another proposed direction is learned score calibration that preserves the existing compositional scheme, as well as richer program synthesis with more expressive temporal logic operators.

Taken as a system design, HiMu replaces end-to-end learned frame scoring with a modular pipeline whose main representational object is a hierarchical logic tree evaluated against cached multimodal evidence. Its significance lies not only in the reported accuracy gains, but in showing that explicit compositional structure can shift the efficiency-accuracy Pareto front for long-video QA under tight context budgets [2603.18558].

Source: https://www.emergentmind.com/topics/himu