Video Spatial Knockout: Methods & Insights

• Video Spatial Knockout is an intervention that disables intra-frame self-attention, isolating spatial interactions to analyze model behavior.
• It is applied across interpretability, controllable video generation, and spatial reasoning to selectively manage content location and occlusion.
• Empirical studies show minimal performance loss in VideoQA with spatial knockout, underscoring the dominant role of language-guided retrieval.

Video Spatial Knockout denotes a family of interventions that suppress, restrict, or strategically withhold spatial information in video systems in order to probe model internals, control generation, or localize editing. In its most explicit published formulation, it is an attention-ablation operator for Video-LLMs: intra-frame spatial self-attention among video tokens is disabled while language-to-video attention and inter-frame video attention remain active (Gou et al., 21 Aug 2025). In adjacent video-generation, editing, and spatial-reasoning settings, the same phrase is used more loosely to describe mechanisms that enforce where content may appear, where it may be removed, or whether spatial structure can still be inferred under occlusion, masking, missing frames, or ambiguous viewpoints (Bhat et al., 17 Jun 2026, Liu et al., 22 May 2026, Ouyang et al., 2 Apr 2025).

1. Terminological scope

Across the cited works, “Video Spatial Knockout” is not a single standardized task. Its meaning depends on the operational level at which spatial information is being manipulated. The literature therefore separates into a formal interpretability usage and several applied extensions.

Context	Formal status	Operational meaning
Video-LLM interpretability	Explicit	Disable intra-frame video-token self-attention within each frame
Controllable generation and editing	Interpretive extension	Permit, suppress, or preserve content in selected spatiotemporal regions
Spatial reasoning robustness	Interpretive extension	Test whether spatial relations survive missing or ambiguous visual cues

The clearest formalization appears in "An Empirical Study on How Video-LLMs Answer Video Questions" (Gou et al., 21 Aug 2025). There, Video Spatial Knockout, abbreviated VS-K, is a pre-softmax attention mask applied at inference time. By contrast, the supplied scope for (Ouyang et al., 2 Apr 2025) states that the paper does not explicitly define or use a task or benchmark named “Video Spatial Knockout”; instead, the phrase is mapped to robust video spatial reasoning under difficult visibility and temporal conditions. Likewise, "LooseControlVideo: Directorial Video Control using Spatial Blocking" (Bhat et al., 17 Jun 2026) and "Occlusion-Aware Physics-Semantic Keyframe Selection for Robust Video Editing" (Liu et al., 22 May 2026) use the idea in an application-oriented sense: spatial knockout becomes the selective enforcement or suppression of content through 3D blocking or mask tubes.

A recurrent misconception is therefore to treat the term as if it named one benchmark. The evidence supports a narrower claim: it is a general intervention pattern whose exact semantics depend on whether the target is transformer attention, generation control, object-level editing, or geometric reasoning.

2. Formal definition in Video-LLM interpretability

In the interpretability setting, the multimodal sequence is written as $MMs = [F_1,\dots,F_N,T]$, where each $F_i$ is the token sequence for frame $i$ and $T$ is the text-token sequence. Standard attention is

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$

Video Spatial Knockout inserts an additional mask that nullifies intra-frame video-to-video edges. If $\mathcal{V}_i$ denotes the index set of video tokens from frame $i$, then the spatial knockout mask is defined as (Gou et al., 21 Aug 2025)

$$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$

The modified attention becomes

$$\tilde{A}=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}+M_{\mathrm{spatial}}\right).$$

Operationally, VS-K prevents a video token from attending to other video tokens in the same frame while preserving two pathways: inter-frame video-to-video attention and language-to-video attention. This places it alongside two companion interventions in the same study: Video Temporal Knockout, which blocks inter-frame video attention, and Language-to-Video Knockout, which blocks attention from text queries to video keys (Gou et al., 21 Aug 2025).

The implementation is deliberately simple. Videos are uniformly sampled to $N=32$ frames, each frame is patch-tokenized with a ViT-like encoder, and typical LLaVA-family settings use $F_i$0 tokens per frame. The study applies hard $F_i$1 masks uniformly across all heads in designated layers, either across the full depth or within a sliding window of four layers. The evaluated backbones include LongVA, InternVideo2.5, LLaVA-OneVision-7B, and LLaVA-Video-7B, with a 34B model reported in the appendix as showing similar trends (Gou et al., 21 Aug 2025).

This formulation matters because it isolates spatial self-attention as a distinct computational substrate. It asks whether frame-internal token interactions are actually responsible for VideoQA behavior, or whether such behavior is driven primarily by cross-modal retrieval from language tokens.

3. Empirical behavior and efficiency implications

The main empirical result is that applying VS-K across all layers causes only minimal performance degradation on Video-MME, MVBench, and EgoSchema, whereas Language-to-Video Knockout causes a large drop; Video Temporal Knockout also causes only small drops (Gou et al., 21 Aug 2025). The study therefore concludes that current Video-LLMs rely more on language-guided retrieval than on either intra-frame or inter-frame video self-attention for VideoQA.

The layerwise picture is more nuanced. In the fine-grained setting, a knockout is applied within a sliding window of four layers. Most windows exhibit small sensitivity to VS-K, but certain intermediate windows act as “critical outliers” with a larger impact. The paper further reports a two-stage processing pattern: lower layers focus on perceptual encoding, while higher layers handle abstract reasoning (Gou et al., 21 Aug 2025). This suggests that the small average effect of VS-K does not imply uniform irrelevance across depth; rather, sensitivity is concentrated in a small subset of intermediate layers.

The complexity analysis explains why the efficiency benefits of spatial knockout are limited. Full video self-attention scales as $F_i$2. Decomposed by frame structure, intra-frame pairs contribute $F_i$3, whereas inter-frame pairs contribute $F_i$4. With $F_i$5 and $F_i$6, inter-frame pairs are approximately $F_i$7 times more numerous than intra-frame pairs, matching the paper’s observation that temporal attention incurs a computational cost $F_i$8 times higher than spatial attention (Gou et al., 21 Aug 2025). Consequently, pruning intra-frame edges yields modest savings relative to pruning temporal edges or exiting video tokens entirely.

The efficiency experiments in Table 1 of (Gou et al., 21 Aug 2025) do not directly optimize with VS-K alone; instead, they combine early-layer temporal knockout with later token exit. For LLaVA-Video, attention FLOPs drop from $F_i$9 to $i$0 to $i$1, while MME changes from $i$2 to $i$3 to $i$4, MVBench from $i$5 to $i$6 to $i$7, and EgoSchema from $i$8 to $i$9 to $T$0. A plausible implication is that VS-K is most useful as diagnostic evidence: it shows that large portions of spatial video self-attention can be pruned in non-critical layers without substantial loss, even if the largest savings come from temporal pruning rather than spatial pruning.

4. Explicit spatial knockout in controllable video generation

In controllable video generation, the term denotes the ability to explicitly permit or suppress content and motion in space and time. "LooseControlVideo: Directorial Video Control using Spatial Blocking" (Bhat et al., 17 Jun 2026) implements this through sparse, oriented 3D boxes that act as a blocking proxy. Each box specifies center, size, orientation, and trajectory; rendered with the target camera, the boxes expose perspective, relative depth, and occlusion in the 2D conditioning space used by the diffusion backbone.

The control representation is DNOCS, which jointly encodes orientation and depth ordering. Orientation is mapped from the local intersection point on the box to RGB through a spherical color wheel, while depth modulates brightness. During compositing, nearer surfaces overwrite farther ones by a $T$1-test, so occlusion ordering is embedded directly into the control video (Bhat et al., 17 Jun 2026). The output per frame is an RGB DNOCS control image together with a binary hit mask and depth map.

Architecturally, LooseControlVideo uses a Wan 2.2 Diffusion Transformer with VACE modules in a ControlNet-style path. The DNOCS control video feeds the VACE path, whose residuals are added to the frozen Wan 2.2 backbone. Training uses LoRA fine-tuning of the VACE modules with rank $T$2, for $T$3k iterations on $T$4 H100 80GB GPUs. Control videos are randomly mixed as $T$5 box-only DNOCS renders and $T$6 partial real frames with occlusion-aware composited mixtures (Bhat et al., 17 Jun 2026).

This design makes “knockout” literal at the level of content allocation. A director can enforce where content may appear, where it may not appear, prescribe trajectories, order occlusions, and locally refine only selected spatiotemporal regions while leaving the global scene context intact. Evaluation is correspondingly geometric. The paper measures Containment, Trajectory Error, Occlusion Accuracy, Rigid Motion Consistency, Global Motion Field Agreement, and Global Overlap Winner, and reports a $T$7 to $T$8 improvement in Trajectory Error, about $T$9 improvement in Rigid Motion Consistency, and a $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$0 to $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$1 increase in Occlusion Accuracy over 2D-box and flow-based baselines on nuScenes, HO-3D, and BEHAVE, while maintaining VBench quality comparable to VACE baselines (Bhat et al., 17 Jun 2026).

The central significance of this line of work is methodological. Instead of knocking out attention links, it knocks out admissible regions of scene occupancy. Spatial control is lifted into 3D, then projected back into the 2D domain where the video model operates.

5. Mask tubes, cutout, and object-level removal

A second applied interpretation of Video Spatial Knockout is object-level localization and removal through temporally consistent masks. "Occlusion-Aware Physics-Semantic Keyframe Selection for Robust Video Editing" (Liu et al., 22 May 2026) addresses this by selecting a single reliable anchor frame using three signals: structural completeness, cycle-consistent tracking stability, and vision-language attribute visibility. Candidate frames are first ranked by a base score derived from the detector’s text-alignment score and a border-distance completeness term; the top $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$2 are then rescored using cycle consistency with $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$3 and attribute visibility from Qwen2.5-VL, with fusion weights $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$4, $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$5, and $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$6 (Liu et al., 22 May 2026). The chosen keyframe is propagated bidirectionally by tracking to form a dense spatiotemporal mask tube $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$7, and the downstream diffusion editor is trained with a region-aware noise-prediction objective

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$8

On Open-VE Bench, the method reaches an average score of $$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M_{\mathrm{causal}}\right)V.$$9, improving its ReCo backbone from $\mathcal{V}_i$0 to $\mathcal{V}_i$1; on Occlusion-Bench it reports strong instruction-fidelity, perceptual, and semantic scores across replacement, removal, and addition (Liu et al., 22 May 2026).

The older interactive segmentation literature provides the mask-propagation substrate for this viewpoint. "Selective Video Object Cutout" (Wang et al., 2017) formulates cutout as label propagation from a small number of annotated frames. It combines a pyramid histogram confidence map with geodesic-distance dynamic modeling and uncertainty-driven local classifiers. Frames are oversegmented with SLIC into about $\mathcal{V}_i$2 superpixels at $\mathcal{V}_i$3, RGB is quantized into $\mathcal{V}_i$4 histogram bins, and foreground masks are propagated bidirectionally before being merged and morphologically cleaned. The method is explicitly linked to spatial knockout through compositing such as $\mathcal{V}_i$5 for foreground removal. On DAVIS it reports average $\mathcal{V}_i$6 and $\mathcal{V}_i$7, with an average runtime of $\mathcal{V}_i$8 s/frame on CPU (Wang et al., 2017).

Once masks are available, background completion becomes the next problem. "VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal" (Chang et al., 2019) fuses optical-flow-based warping from neighboring frames with an image-inpainting backbone and a ConvLSTM refinement network. FlowNet2 estimates raw flow, foreground-corrupted flow is removed and bilinearly inpainted, previous object-removed outputs are warped into the current frame for $\mathcal{V}_i$9, and the best candidate is selected by LPIPS distance to the previous output before refinement (Chang et al., 2019). On the SVOR dataset, VORNet reports MSE $i$0, SSIM $i$1, and LPIPS $i$2, outperforming the compared patch-based and image-based baselines.

Taken together, these works define a full object-level knockout pipeline: select or annotate a reliable anchor, propagate masks through time, then inpaint or regenerate the removed region with temporal coherence.

6. Relation to video spatial reasoning and robustness evaluation

The supplied clarification for (Ouyang et al., 2 Apr 2025) maps “Video Spatial Knockout” to a robustness-oriented evaluation of video spatial reasoning rather than to a formally named benchmark. Under that interpretation, a knockout-style test would examine whether a model can still infer spatial layouts, geometric relations, sizes, and appearance order when visibility is impaired by occlusion, masking, missing frames, or ambiguous viewpoints (Ouyang et al., 2 Apr 2025).

The reported system is described as Spatial-R1, built from ScanNet videos resampled at $i$3 FPS and a dataset of approximately $i$4k QA pairs after filtering. The task space comprises seven verifiable categories: object relative distance, object size estimation, room size estimation, object relative direction, object appearance order, object absolute distance, and object counting. Answers are grounded in axis-aligned segmented scene point clouds, Alpha Shape floor footprints, and per-frame semantic segmentation, yielding programmatically verifiable numeric or categorical outputs (Ouyang et al., 2 Apr 2025).

Training uses Qwen2.5-VL-7B-Instruct with Group Relative Policy Optimization and three verifiable rewards: Numerical Accuracy Reward, Multi-Choice Accuracy Reward, and a Format Reward enforcing the output structure > <answer> </answer> (Ouyang et al., 2 Apr 2025). On VSI-Bench, the model reports $i$5 overall accuracy, compared with $i$6 for Qwen2.5-VL-7B zero-shot and $i$7 for GPT-4o, while Gemini-2.0 Flash reaches $i$8. The largest reported gains over the zero-shot baseline occur in Object Counting, $i$9 versus $$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$0; Object Abs Distance, $$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$1 versus $$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$2; and Object Size Estimation, $$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$3 versus $$(M_{\mathrm{spatial}})_{u\to v}= \begin{cases} -\infty, & u\in \mathcal{V}_i,\ v\in \mathcal{V}_i,\ u\neq v,\ 0, & \text{otherwise}. \end{cases}$$4. Declines appear in Room Size Estimation, Object Rel Direction, and Route Planning (Ouyang et al., 2 Apr 2025).

This evidence does not define Video Spatial Knockout as a benchmark in the strict sense. It does, however, motivate a robustness interpretation: if spatial answers are verifiable from 3D scene geometry, then masking or otherwise degrading the visible evidence provides a natural stress test of whether a model has learned grounded spatial computation rather than superficial pattern matching. The supplied notes explicitly state that extensions such as occlusion or frame-drop variants would be additional engineering beyond what is reported in (Ouyang et al., 2 Apr 2025).

In aggregate, the literature presents Video Spatial Knockout as a cross-cutting methodological motif. In Video-LLM analysis, it is a controlled ablation of intra-frame attention links. In generation and editing, it is the selective authorization or suppression of occupancy, motion, and object identity through 3D blocking or mask tubes. In spatial reasoning, it becomes a robustness lens on whether spatial relations remain recoverable when evidence is partially withheld. The shared thread is not a single benchmark, but a common experimental logic: remove or constrain spatial information in a targeted way, then measure what the system still understands, preserves, or produces.