NoSense Baseline: Unmasking Benchmark Shortcuts

• NoSense Baselines are models that intentionally discard task-specific cues to test if benchmark performance relies on shallow shortcuts.
• They process each input element independently—ignoring temporal and spatial structures in video tasks and premise context in NLI—thereby exposing dataset artifacts.
• Empirical results show near-optimal accuracy in video benchmarks and significant success in NLI tasks, highlighting critical flaws in benchmark and dataset design.

NoSense Baseline refers to a class of models that intentionally omit core task-relevant structures or cues—such as temporal coherence, spatial relationships, or input context—demonstrating the extent to which benchmarks can be solved without the high-level reasoning or modeling they ostensibly measure. The NoSense approach exposes critical weaknesses in dataset construction by showing the sufficiency of shallow or shortcut-based strategies for achieving near-optimal performance. In recent years, the paradigm has emerged across vision–language video tasks (notably VSI-Super-Recall) and in natural language inference (hypothesis-only classification). NoSense baselines are recommended as mandatory reporting for benchmarking, as their strong results reveal flaws and superficialities in task design.

1. Conceptual Overview

NoSense baselines operate by discarding the cues, structures, or modeling principles that a benchmark purports to require. In the VSI-Super-Recall (VSR) task, this is instantiated by ignoring any temporal dependencies and spatial world modeling; instead, each video frame is processed independently to retrieve frames most semantically similar to a query object and assign an answer based solely on shallow similarities. In natural language inference (NLI), the hypothesis-only baseline predicts entailment labels without consulting the premise, revealing label-conditional artifacts and statistical irregularities in the dataset (Udandarao et al., 20 Nov 2025, Poliak et al., 2018).

The fundamental goal of the NoSense approach is not high task performance per se but the critical evaluation of whether benchmark performance truly reflects progress on the intended underlying capability.

2. Algorithms and Mathematical Structure

VSI-Super-Recall (VSR) Case

The NoSense baseline for VSR is defined formally as follows. For a video $x_1, \dots, x_N$ (sampled at 1 FPS), target object $o$, auxiliary environment labels $a_1, \dots, a_4$, and answer permutations $\{\pi^{(k)}\}_{k=1}^4$:

1. Encode the object prompt(s) into a unit vector $O \in \mathbb{R}^d$.
2. Iterate through video frames $x_t$:
   • Compute image feature $F_t = \operatorname{norm}(f_{\text{img}}(x_t))$.
   • Compute relevance $s_t = \langle F_t, O \rangle$.
   • Maintain a buffer of the 4 frames with highest $s_t$.
3. For each of the top-4 frames, encode “object+environment” joint prompts $A_i = \operatorname{norm}(f_{\text{txt}}(\text{"a photo of a \{o\} in a \{a_i\}"}))$ for $i=1\dots4$.
4. Construct $R_{u,i} = \langle F_{t_u}, A_i \rangle$ for $u,i = 1\dots4$.
5. For each answer permutation $\pi^{(k)}$, compute $score(k) = \sum_{u=1}^4 R_{u, \pi^{(k)}_u}$.
6. Output $\hat{k} = \arg\max_{k=1\dots4} score(k)$.

All operations are performed in a streaming, single-pass manner, storing only four image feature vectors and necessary text embeddings (Udandarao et al., 20 Nov 2025).

NLI (Hypothesis-only) Case

For NLI, the hypothesis-only classifier predicts the label $y$ given only the hypothesis $h$, i.e., $p(y|h)$. The model structure:

• Encodes hypothesis $h$ via bidirectional LSTM with max-pooling: $$v(h) = \max\text{-pool}( \mathrm{BiLSTM}(\mathrm{Embeddings}(h)))$$.
• Label probabilities: $p(y|h) = \mathrm{Softmax}(W v(h) + b)$.
• Loss: standard multi-class cross-entropy over all hypotheses and labels.

No context from the premise is ever accessed, and all signal comes from the hypothesis alone (Poliak et al., 2018).

3. Discarding Temporal and Structural Information

Central to the NoSense philosophy is the explicit removal of structural cues:

• VSR: NoSense makes no use of temporal aggregation or long-horizon memory. Each frame is processed independently; the only “memory” is the top-4 most relevant frame buffer. Sequence or scene flow, spatial layout, and object tracking are not modeled.
• NLI: NoSense predicts using only the hypothesis statement, omitting the premise entirely—disqualifying any genuine semantic or logical inference linking the premise and hypothesis.

This design ensures that if benchmark accuracy remains high, the task must be solvable using superficial or confounded signals present in the individual inputs alone.

4. Empirical Results and Practical Implications

Notable benchmark results quantifying NoSense efficacy include (Udandarao et al., 20 Nov 2025, Poliak et al., 2018):

Task / Dataset Split	NoSense Acc.	SOTA/System	SOTA Acc.
VSR (10/30/60min)	98.3–96.7%	Cambrian-S	40–45%
VSR (2hr/4hr)	~95%
SNLI (NLI, test)	69.0%	SOTA (InferSent, etc.)	89.3%
MNLI-M (NLI)	55.5%	SOTA	80.6%
SPR (NLI, recast)	86.6%	SOTA	80.6%

For VSR, NoSense achieves near-perfect accuracy, saturating benchmark performance and outstripping more elaborate world modeling approaches. In NLI, hypothesis-only models consistently double or triple majority-class baseline accuracy and reach a substantial fraction of full-system SOTA performance.

These outcomes demonstrate that benchmark construction can embed statistical or semantic shortcuts and that reported task gains may be decoupled from the underlying cognitive or algorithmic capabilities nominally under test.

5. Shortcut Exploitation and Dataset Irregularities

Analysis of NoSense successes reveals several mechanisms for shortcut exploitation (Udandarao et al., 20 Nov 2025, Poliak et al., 2018):

• Sparse Signal Targeting: VSR as implemented is a “needle-in-a-haystack” retrieval task, with exactly four distinctive “object + environment” frames dominating the answer.
• Unambiguous Auxiliary Cues: The auxiliary environment labels are strongly textually and visually aligned; contrastive embeddings suffice to reliably identify correct frames.
• Permutation Structure: The finite answer set (all 24 possible orderings) enables brute-force scoring by frame-label similarity.

For NLI, artifact patterns include:

• Label-Correlated Words (“Give-away” Words): Words such as “Nobody” or “because” have highly skewed label conditional probabilities, allowing trivial prediction.
• Grammaticality and Synthetic Artifacts: Synthetically generated or ungrammatical hypotheses correlate with certain classes.
• World Knowledge/Proto-Role Bias: Lexical semantics alone can often predict entailment without need for premise integration.

The presence of such artifacts undermines benchmark validity as a measure of the intended property (e.g., schematic generalization, world modeling, inference).

6. Computational Efficiency and Resource Profile

NoSense baselines are computationally minimal:

• Memory: For VSR, the algorithm requires storage only for four $d$-dimensional feature vectors plus prompt text features.
• Operation Count: Processing is a single forward pass per frame via a pretrained VLM (e.g., SigLIP or CLIP) at 1 FPS—$\mathcal{O}(N(\text{Cost}_{\text{imgEnc}}+d))$ for $N$ video frames.
• Contrasts: Cambrian-S (for VSR) invokes multimodal LLMs, segmentation, and dynamic memory, with significantly higher FLOP and memory requirements.

This computational profile highlights the inefficiency of elaborate pipelines when simple shallow retrieval suffices under flawed benchmark conditions (Udandarao et al., 20 Nov 2025).

7. Limitations, Caveats, and Recommendations

NoSense efficacy demonstrates benchmark fragility but should not be misconstrued as evidence that advanced modeling is unnecessary across all instances or datasets:

• Results are task- and dataset-specific; other benchmarks may be less susceptible to shortcut exploitation.
• Findings do not generalize to tasks with more naturalistic data, randomized structure, or invariance checks (e.g., repeated scene visits, shuffled environments).
• Authors recommend routine inclusion of NoSense-style baselines in benchmark reporting to surface and remediate dataset flaws.
• Until benchmarks robustly foil atemporal and structure-ignorant baselines, high accuracy should not be cited as evidence of genuine spatial, temporal, or inferential reasoning (Udandarao et al., 20 Nov 2025, Poliak et al., 2018).

The broader lesson is that rigorous benchmark construction—resistant to shallow shortcuts, with greater structural diversity and invariance—remains essential for credible measurement of progress in computational cognition and world modeling.