---
title: 'NeuS-QA: Neuro-Symbolic LVQA Pipeline'
url: https://www.emergentmind.com/topics/neus-qa
type: topic
---

# NeuS-QA: Neuro-Symbolic LVQA Pipeline

NeuS-QA is a training-free, plug-and-play neuro-symbolic pipeline for long-form video question answering (LVQA) that translates a natural-language question into temporal logic, constructs a finite-state video automaton from frame-level semantic propositions, applies probabilistic model checking to identify the video interval satisfying the query, and sends only those logic-verified frames to an off-the-shelf vision-language model (VLM) for final answer generation [2509.18041]. It is motivated by the failure modes of vanilla uniform-frame VLM prompting on long videos, where token-budget constraints force severe downsampling and thereby obscure fine-grained visual structure, subtle event transitions, and compositional temporal dependencies, especially for questions involving event ordering, causality, and multi-step reasoning [2509.18041].

## 1. Problem formulation and pipeline structure

NeuS-QA addresses LVQA as a temporal grounding problem rather than as direct end-to-end prompting. The central claim is that long-form queries often demand explicit representations of event order, persistence, and causal succession, whereas query-adaptive frame sampling, hierarchical keyframe selection, and agent-based iterative querying remain fundamentally heuristic because they lack explicit temporal representations and cannot enforce or verify logical event relationships [2509.18041]. NeuS-QA therefore introduces a four-stage architecture: natural language to temporal logic, video automaton construction, probabilistic model checking, and VLM-based answer generation.

At a high level, the pipeline first parses a free-form question into a finite set of atomic propositions and a single temporal formula. It then discretizes the video into windows, estimates proposition truth values for each window with a vision-language proposition detector, and assembles these detections into a deterministic forward-transition automaton. Model checking identifies a satisfying interval, and only that compact clip is forwarded to a downstream VLM. This division of labor is intended to improve interpretability, reduce hallucinations, and enable compositional reasoning without modifying or fine-tuning the underlying VLM [2509.18041].

## 2. Temporal-logic representation of questions

NeuS-QA adopts a fragment of linear-time temporal logic (LTL), embedded in the Probabilistic Computation Tree Logic (PCTL) framework used by the STORM model checker. The formula grammar is

$$
\phi ::= p \mid \neg \phi \mid (\phi \wedge \psi) \mid (\phi \vee \psi) \mid X\phi \mid F\phi \mid G\phi \mid (\phi U \psi),
$$

where $p \in P$ is an atomic proposition. The operators have the standard LTL readings: $X$ for “next,” $F$ for “eventually,” $G$ for “globally/always,” and $U$ for “until” [2509.18041].

Semantics are defined over infinite paths $\pi = q_0 q_1 q_2 \dots$ through a labeled transition system with labeling function $L(q_i) \subseteq P$. Thus, $\pi,i \models p$ iff $p \in L(q_i)$; $\pi,i \models X\phi$ iff $\pi,i+1 \models \phi$; $\pi,i \models F\phi$ iff there exists $j \ge i$ such that $\pi,j \models \phi$; $\pi,i \models G\phi$ iff for all $j \ge i$, $\pi,j \models \phi$; and $\pi,i \models \phi U \psi$ iff there exists $k \ge i$ such that $\pi,k \models \psi$ and for all $j$ with $i \le j < k$, $\pi,j \models \phi$ [2509.18041].

In practice, NeuS-QA uses a two-shot prompt to GPT-o1-mini to parse a question $q$ into a finite proposition set $P = \{p_1,\dots,p_n\}$ and a temporal formula $\phi$. The prompt includes schemas such as “A before B” $\rightarrow (\neg B \ U \ A) \wedge F B$ and “A and then B” $\rightarrow A \wedge (A \ U \ B) \wedge F B$. The output is produced in LaTeX-style syntax and checked for well-formedness by a small grammar validator [2509.18041]. This suggests that NeuS-QA treats query interpretation as a constrained formalization task rather than as unrestricted chain-of-thought generation.

## 3. Video automaton construction and proposition grounding

NeuS-QA discretizes a video $V$ into overlapping windows of $\kappa$ frames, with the paper giving $\kappa = 4$ at 3 fps as an example. One state is created per window, yielding $Q = \{q_0,q_1,\dots,q_T\}$. For each window $t$ and atomic proposition $p_i$, the pipeline estimates

$$
Z_{t,i} = \text{probability that } p_i \in P \text{ holds in window } t,
$$

using a vision-language proposition detector $\mathcal{M}_{\text{prop}}$, exemplified by InternVL2-8B, which returns calibrated “Yes/No” logits for each proposition [2509.18041].

The resulting video automaton is

$$
A_V = (Q, q_0, \delta, L),
$$

with deterministic forward transitions

$$
\delta(q_t,q_{t+1}) = 1,
$$

and state labels

$$
L(q_t) = \{\,p_i \in P \mid Z_{t,i} \ge \theta_i\,\},
$$

where $\theta_i$ is a per-proposition calibration threshold chosen to control false positives [2509.18041]. The automaton is therefore not learned as a latent dynamical system; it is explicitly assembled from proposition detections over fixed-stride windows. The paper notes that hierarchical segmentation or key-frame clustering can optionally be applied at this stage, but that simple fixed-stride windows suffice in the reported experiments [2509.18041].

This construction is significant because it converts long-form video from a dense unstructured token stream into a symbolic transition system amenable to formal verification. A plausible implication is that the automaton becomes an intermediate representation supporting auditability: every later decision can be traced back to proposition thresholds and temporal states rather than only to a final VLM output.

## 4. Probabilistic model checking and verified segment extraction

For each time step $t$, NeuS-QA builds the partial automaton $A_{V,t}$ over states $q_0,\dots,q_t$ and computes

$$
P_t = \Pr[A_{V,t} \models \phi]
$$

using probabilistic model checking under PCTL [2509.18041]. Internally, STORM converts $\phi$ to an $\omega$-automaton, constructs the product with $A_{V,t}$, and solves a linear system for reachability probabilities. The worst-case complexity is reported as $O(|Q| \cdot 2^{|\phi|})$, although the paper states that formulas and automata remain small in practice [2509.18041].

The raw satisfaction probability is smoothed by

$$
\hat F(c) = \frac{1}{1 + \exp(-\gamma(c-\tau))},
$$

where $\tau$ is the satisfaction threshold, given as $\tau = 0.5$ as an example, and $\gamma$ controls steepness. Once $\hat F(P_t) \ge \tau$, NeuS-QA stops and backtracks to find the minimal index $s \le t$ such that the sub-path $q_s \dots q_t$ satisfies $F(\phi) \wedge G_{[0,s)} \neg(\neg \phi)$. This yields a satisfying interval $[t_{\text{start}}, t_{\text{end}}]$ of windows that witnesses $\phi$ [2509.18041].

The segment is then expanded to a compact clip $V' = V[t_{\text{start}}-\alpha : t_{\text{end}}+\beta]$, where $\alpha$ and $\beta$ extend a few seconds before and after for context. From $V'$, NeuS-QA uniformly samples $F$ frames, with $F=32$ or $48$, converts them to image patches, and concatenates them with the original question into a prompt for a VLM. The reported VLMs are GPT-4o-2024-08-06, Qwen2.5-VL-7B-Instruct, and LLaVA-OneVision-Qwen2-7B; typical settings are temperature $= 0.0$, top-p $= 0.9$, and max\_tokens $= 512$. A system prompt is prepended to enforce “Answer only using information visible in the frames. Do not invent new objects or events.” [2509.18041]

## 5. Empirical performance on long-form benchmarks

NeuS-QA is evaluated on LongVideoBench, which contains videos up to 60 minutes, and CinePile, which contains narrative movie scenes. The evaluation target is multiple-choice accuracy on questions requiring temporal ordering and causality [2509.18041]. On LongVideoBench, the reported baseline VLM accuracies with 32 uniformly sampled frames are 50.44% for Qwen2.5-VL-7B and 49.56% for GPT-4o. Structured baselines are VideoTree at 50.47% and LVNet at 45.59%. NeuS-QA raises Qwen2.5-VL to 60.09% for a gain of +9.65 percentage points and GPT-4o to 58.33% for a gain of +8.77 percentage points [2509.18041].

The category-level breakdown on LongVideoBench shows particularly strong gains on temporally structured question types: T3E rises from 39.62% to 54.72%, E3E from 58.11% to 62.16%, T3O from 54.39% to 66.67%, and O3O from 45.46% to 54.55% [2509.18041]. On CinePile, the Qwen2.5-VL baseline is 50.73%, whereas NeuS-QA + Qwen2.5-VL reaches 53.66% for a gain of +2.93 percentage points [2509.18041]. The paper reports that all improvements are statistically significant under a paired t-test with $p < 0.01$.

These results support the paper’s more general claim that NeuS-QA is especially effective on questions involving event ordering, causality, and multi-step compositional reasoning. They also indicate that the gains do not arise merely from substituting a stronger answer model, since the same downstream VLMs are evaluated both with and without the neuro-symbolic front end [2509.18041].

## 6. Interpretability, limitations, and subsequent extensions

NeuS-QA is positioned as an explicit alternative to heuristic clip-selection pipelines. Its stated benefits are interpretability, because the automaton and satisfying run form an explicit “proof” of why selected frames answer the query; formal guarantees, because model checking ensures no spurious ordering or event relationships slip through; and compositionality, because queries with arbitrary nesting of $U/F/G$ are handled out-of-the-box [2509.18041]. This framing distinguishes NeuS-QA from systems that retrieve frames adaptively but do not represent temporal structure symbolically.

The limitations are equally explicit. Proposition-detector errors can cause false negatives: if an event is not detected, $\phi$ may never be satisfied. Automaton construction and probabilistic model checking can also be computationally heavier than simple frame retrieval [2509.18041]. The paper identifies several extensions: richer logics such as Signal Temporal Logic and Metric Temporal Logic, unsupervised proposition learning, agentic iteration that refines multiple queries and caches automaton states, and pre-computing a single video automaton for thousands of queries to amortize builder cost [2509.18041].

A direct extension is LE-NeuS, which preserves the four logical stages of NeuS-QA while introducing CLIP-guided two-stage adaptive sampling, batched proposition detection, and Multi-Segment FoI Retrieval to reduce inference latency [2602.23553]. On NVIDIA H100 GPUs, LE-NeuS reduces the global average latency from 553.7 s for dense NeuS-QA to 44.2 s, a 12.53× speedup, while improving accuracy on temporally complex subsets: on LongVideoBench, NeuS-QA overall is 61.89% and LE-NeuS overall is 67.10%; on the Video-MME Temporal Reasoning subset, NeuS-QA is 55.17% and LE-NeuS is 67.24% [2602.23553]. Theoretical analysis in LE-NeuS isolates the dominant bottleneck in dense sequential proposition grounding and gives the end-to-end bound

$$
L_{\text{total}}(L,P,s) \le L_{\text{fixed}} + L \cdot L_{\text{CLIP}} + \left\lceil \frac{sL}{\kappa} \right\rceil \cdot L_{\text{VLM}},
$$

thereby formalizing the latency-accuracy tradeoff inherent in the original NeuS-QA design [2602.23553]. This suggests that NeuS-QA established the core neuro-symbolic formulation, while later work focused on mitigating its computational overhead without discarding temporal logic or model checking.

Source: https://www.emergentmind.com/topics/neus-qa