NeuS-QA: Neuro-Symbolic LVQA Pipeline

• The paper introduces a four-stage pipeline that converts natural language queries into temporal logic and constructs a video automaton for precise segment extraction.
• It applies probabilistic model checking to verify temporal sequences, improving accuracy on event ordering, causality, and compositional reasoning tasks.
• Empirical results on LongVideoBench and CinePile demonstrate gains of up to 9.65 percentage points over baseline VLMs, validating its effectiveness.

NeuS-QA is a training-free, plug-and-play neuro-symbolic pipeline for long-form video question answering (LVQA) that translates a natural-language question into temporal logic, constructs a finite-state video automaton from frame-level semantic propositions, applies probabilistic model checking to identify the video interval satisfying the query, and sends only those logic-verified frames to an off-the-shelf vision-LLM (VLM) for final answer generation (Shah et al., 22 Sep 2025). It is motivated by the failure modes of vanilla uniform-frame VLM prompting on long videos, where token-budget constraints force severe downsampling and thereby obscure fine-grained visual structure, subtle event transitions, and compositional temporal dependencies, especially for questions involving event ordering, causality, and multi-step reasoning (Shah et al., 22 Sep 2025).

1. Problem formulation and pipeline structure

NeuS-QA addresses LVQA as a temporal grounding problem rather than as direct end-to-end prompting. The central claim is that long-form queries often demand explicit representations of event order, persistence, and causal succession, whereas query-adaptive frame sampling, hierarchical keyframe selection, and agent-based iterative querying remain fundamentally heuristic because they lack explicit temporal representations and cannot enforce or verify logical event relationships (Shah et al., 22 Sep 2025). NeuS-QA therefore introduces a four-stage architecture: natural language to temporal logic, video automaton construction, probabilistic model checking, and VLM-based answer generation.

At a high level, the pipeline first parses a free-form question into a finite set of atomic propositions and a single temporal formula. It then discretizes the video into windows, estimates proposition truth values for each window with a vision-language proposition detector, and assembles these detections into a deterministic forward-transition automaton. Model checking identifies a satisfying interval, and only that compact clip is forwarded to a downstream VLM. This division of labor is intended to improve interpretability, reduce hallucinations, and enable compositional reasoning without modifying or fine-tuning the underlying VLM (Shah et al., 22 Sep 2025).

2. Temporal-logic representation of questions

NeuS-QA adopts a fragment of linear-time temporal logic (LTL), embedded in the Probabilistic Computation Tree Logic (PCTL) framework used by the STORM model checker. The formula grammar is

$$\phi ::= p \mid \neg \phi \mid (\phi \wedge \psi) \mid (\phi \vee \psi) \mid X\phi \mid F\phi \mid G\phi \mid (\phi U \psi),$$

where $p \in P$ is an atomic proposition. The operators have the standard LTL readings: $X$ for “next,” $F$ for “eventually,” $G$ for “globally/always,” and $U$ for “until” (Shah et al., 22 Sep 2025).

Semantics are defined over infinite paths $\pi = q_0 q_1 q_2 \dots$ through a labeled transition system with labeling function $L(q_i) \subseteq P$. Thus, $\pi,i \models p$ iff $p \in L(q_i)$; $p \in P$0 iff $p \in P$1; $p \in P$2 iff there exists $p \in P$3 such that $p \in P$4; $p \in P$5 iff for all $p \in P$6, $p \in P$7; and $p \in P$8 iff there exists $p \in P$9 such that $X$0 and for all $X$1 with $X$2, $X$3 (Shah et al., 22 Sep 2025).

In practice, NeuS-QA uses a two-shot prompt to GPT-o1-mini to parse a question $X$4 into a finite proposition set $X$5 and a temporal formula $X$6. The prompt includes schemas such as “A before B” $X$7 and “A and then B” $X$8. The output is produced in LaTeX-style syntax and checked for well-formedness by a small grammar validator (Shah et al., 22 Sep 2025). This suggests that NeuS-QA treats query interpretation as a constrained formalization task rather than as unrestricted chain-of-thought generation.

3. Video automaton construction and proposition grounding

NeuS-QA discretizes a video $X$9 into overlapping windows of $F$0 frames, with the paper giving $F$1 at 3 fps as an example. One state is created per window, yielding $F$2. For each window $F$3 and atomic proposition $F$4, the pipeline estimates

$F$5

using a vision-language proposition detector $F$6, exemplified by InternVL2-8B, which returns calibrated “Yes/No” logits for each proposition (Shah et al., 22 Sep 2025).

The resulting video automaton is

$F$7

with deterministic forward transitions

$F$8

and state labels

$F$9

where $G$0 is a per-proposition calibration threshold chosen to control false positives (Shah et al., 22 Sep 2025). The automaton is therefore not learned as a latent dynamical system; it is explicitly assembled from proposition detections over fixed-stride windows. The paper notes that hierarchical segmentation or key-frame clustering can optionally be applied at this stage, but that simple fixed-stride windows suffice in the reported experiments (Shah et al., 22 Sep 2025).

This construction is significant because it converts long-form video from a dense unstructured token stream into a symbolic transition system amenable to formal verification. A plausible implication is that the automaton becomes an intermediate representation supporting auditability: every later decision can be traced back to proposition thresholds and temporal states rather than only to a final VLM output.

4. Probabilistic model checking and verified segment extraction

For each time step $G$1, NeuS-QA builds the partial automaton $G$2 over states $G$3 and computes

$G$4

using probabilistic model checking under PCTL (Shah et al., 22 Sep 2025). Internally, STORM converts $G$5 to an $G$6-automaton, constructs the product with $G$7, and solves a linear system for reachability probabilities. The worst-case complexity is reported as $G$8, although the paper states that formulas and automata remain small in practice (Shah et al., 22 Sep 2025).

The raw satisfaction probability is smoothed by

$G$9

where $U$0 is the satisfaction threshold, given as $U$1 as an example, and $U$2 controls steepness. Once $U$3, NeuS-QA stops and backtracks to find the minimal index $U$4 such that the sub-path $U$5 satisfies $U$6. This yields a satisfying interval $U$7 of windows that witnesses $U$8 (Shah et al., 22 Sep 2025).

The segment is then expanded to a compact clip $U$9, where $\pi = q_0 q_1 q_2 \dots$0 and $\pi = q_0 q_1 q_2 \dots$1 extend a few seconds before and after for context. From $\pi = q_0 q_1 q_2 \dots$2, NeuS-QA uniformly samples $\pi = q_0 q_1 q_2 \dots$3 frames, with $\pi = q_0 q_1 q_2 \dots$4 or $\pi = q_0 q_1 q_2 \dots$5, converts them to image patches, and concatenates them with the original question into a prompt for a VLM. The reported VLMs are GPT-4o-2024-08-06, Qwen2.5-VL-7B-Instruct, and LLaVA-OneVision-Qwen2-7B; typical settings are temperature $\pi = q_0 q_1 q_2 \dots$6, top-p $\pi = q_0 q_1 q_2 \dots$7, and max_tokens $\pi = q_0 q_1 q_2 \dots$8. A system prompt is prepended to enforce “Answer only using information visible in the frames. Do not invent new objects or events.” (Shah et al., 22 Sep 2025)

5. Empirical performance on long-form benchmarks

NeuS-QA is evaluated on LongVideoBench, which contains videos up to 60 minutes, and CinePile, which contains narrative movie scenes. The evaluation target is multiple-choice accuracy on questions requiring temporal ordering and causality (Shah et al., 22 Sep 2025). On LongVideoBench, the reported baseline VLM accuracies with 32 uniformly sampled frames are 50.44% for Qwen2.5-VL-7B and 49.56% for GPT-4o. Structured baselines are VideoTree at 50.47% and LVNet at 45.59%. NeuS-QA raises Qwen2.5-VL to 60.09% for a gain of +9.65 percentage points and GPT-4o to 58.33% for a gain of +8.77 percentage points (Shah et al., 22 Sep 2025).

The category-level breakdown on LongVideoBench shows particularly strong gains on temporally structured question types: T3E rises from 39.62% to 54.72%, E3E from 58.11% to 62.16%, T3O from 54.39% to 66.67%, and O3O from 45.46% to 54.55% (Shah et al., 22 Sep 2025). On CinePile, the Qwen2.5-VL baseline is 50.73%, whereas NeuS-QA + Qwen2.5-VL reaches 53.66% for a gain of +2.93 percentage points (Shah et al., 22 Sep 2025). The paper reports that all improvements are statistically significant under a paired t-test with $\pi = q_0 q_1 q_2 \dots$9.

These results support the paper’s more general claim that NeuS-QA is especially effective on questions involving event ordering, causality, and multi-step compositional reasoning. They also indicate that the gains do not arise merely from substituting a stronger answer model, since the same downstream VLMs are evaluated both with and without the neuro-symbolic front end (Shah et al., 22 Sep 2025).

6. Interpretability, limitations, and subsequent extensions

NeuS-QA is positioned as an explicit alternative to heuristic clip-selection pipelines. Its stated benefits are interpretability, because the automaton and satisfying run form an explicit “proof” of why selected frames answer the query; formal guarantees, because model checking ensures no spurious ordering or event relationships slip through; and compositionality, because queries with arbitrary nesting of $L(q_i) \subseteq P$0 are handled out-of-the-box (Shah et al., 22 Sep 2025). This framing distinguishes NeuS-QA from systems that retrieve frames adaptively but do not represent temporal structure symbolically.

The limitations are equally explicit. Proposition-detector errors can cause false negatives: if an event is not detected, $L(q_i) \subseteq P$1 may never be satisfied. Automaton construction and probabilistic model checking can also be computationally heavier than simple frame retrieval (Shah et al., 22 Sep 2025). The paper identifies several extensions: richer logics such as Signal Temporal Logic and Metric Temporal Logic, unsupervised proposition learning, agentic iteration that refines multiple queries and caches automaton states, and pre-computing a single video automaton for thousands of queries to amortize builder cost (Shah et al., 22 Sep 2025).

A direct extension is LE-NeuS, which preserves the four logical stages of NeuS-QA while introducing CLIP-guided two-stage adaptive sampling, batched proposition detection, and Multi-Segment FoI Retrieval to reduce inference latency (Liang et al., 26 Feb 2026). On NVIDIA H100 GPUs, LE-NeuS reduces the global average latency from 553.7 s for dense NeuS-QA to 44.2 s, a 12.53× speedup, while improving accuracy on temporally complex subsets: on LongVideoBench, NeuS-QA overall is 61.89% and LE-NeuS overall is 67.10%; on the Video-MME Temporal Reasoning subset, NeuS-QA is 55.17% and LE-NeuS is 67.24% (Liang et al., 26 Feb 2026). Theoretical analysis in LE-NeuS isolates the dominant bottleneck in dense sequential proposition grounding and gives the end-to-end bound

$L(q_i) \subseteq P$2

thereby formalizing the latency-accuracy tradeoff inherent in the original NeuS-QA design (Liang et al., 26 Feb 2026). This suggests that NeuS-QA established the core neuro-symbolic formulation, while later work focused on mitigating its computational overhead without discarding temporal logic or model checking.