DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoE

Abstract: In the literature, prior research on Security-oriented Video Understanding (SVU) has predominantly focused on detecting and localize the threats (e.g., shootings, robberies) in videos, while largely lacking the effective capability to generate and evaluate the threat causes. Motivated by these gaps, this paper introduces a new chat paradigm SVU task, i.e., In-depth Security-oriented Video Understanding (DeepSVU), which aims to not only identify and locate the threats but also attribute and evaluate the causes threatening segments. Furthermore, this paper reveals two key challenges in the proposed task: 1) how to effectively model the coarse-to-fine physical-world information (e.g., human behavior, object interactions and background context) to boost the DeepSVU task; and 2) how to adaptively trade off these factors. To tackle these challenges, this paper proposes a new Unified Physical-world Regularized MoE (UPRM) approach. Specifically, UPRM incorporates two key components: the Unified Physical-world Enhanced MoE (UPE) Block and the Physical-world Trade-off Regularizer (PTR), to address the above two challenges, respectively. Extensive experiments conduct on our DeepSVU instructions datasets (i.e., UCF-C instructions and CUVA instructions) demonstrate that UPRM outperforms several advanced Video-LLMs as well as non-VLM approaches. Such information.These justify the importance of the coarse-to-fine physical-world information in the DeepSVU task and demonstrate the effectiveness of our UPRM in capturing such information.

• The paper introduces DeepSVU, a unified system that identifies threats, localizes their timestamps, and explains their likely causes using four physical-world experts for pose, objects, background, and coarse video context.
• UPRM improves CUVA performance over Hawkeye by reducing FNR from 13.24 to 10.46, raising F2 from 84.35 to 88.41, and increasing average localization mAP from 81.85 to 86.24.
• A physical-world trade-off regularizer balances expert contributions, while ablations show that removing PTR raises CUVA FNR to 18.07 and removing physical-world pre-tuning costs 10.7 FNR points and 10.29 mAP.

Task definition and motivation

The paper introduces In-depth Security-oriented Video Understanding (DeepSVU), a chat-paradigm extension of Security-oriented Video Understanding (SVU) that goes beyond threat detection and temporal localization to include causal attribution of threatening segments. Formally, given a video $V$ of $T$ segments, the model must output a set $\{(s_i, e_i, c_i)\}$ where $s_i, e_i$ are threat timestamps and $c_i$ is a natural-language causal explanation. The authors position this against prior LLM-based SVU systems — Holmes (Zhang et al., 2024), Ex-VAD, and Hawkeye — which they argue identify and locate threats but do not generate or evaluate threat causes, and do not model fine-grained physical-world semantics.

The task is instantiated on two instruction datasets: UCF-C instructions (built from UCF-Crime, evaluating identifying and locating only, since UCF-Crime lacks cause annotations) and CUVA instructions (built from CUVA, which provides root-cause captions and thus supports all three sub-tasks). Instructions are templated for identification ("determine whether threats exist"), localization ("locate exact timestamps"), and attribution ("explain root causes").

Two challenges

The paper identifies two obstacles. First, existing Video-LLMs model coarse-grained video representations while neglecting fine-grained physical-world information (human pose, object interactions, background context), whose heterogeneity across encoder architectures strains a fixed-capacity transformer. Second, the physical-world information in the constructed datasets is imbalanced: coarse-grained descriptions appear in 100% of samples, human pose in 41% (UCF-C) and 46% (CUVA), with object relations and visual background lower still. A vanilla MoE router trained on this distribution over-weights the dominant experts, which the authors show empirically leads to degraded threat detection.

The UPRM approach

UPRM builds on Video-LLaVA (7B, Vicuna backbone with LanguageBind/CLIP ViT-L14 encoders) and comprises two components.

Unified Physical-world Enhanced MoE (UPE) block

Four experts process complementary signals:

• Human-Pose Expert (HPE): HigherHRNet extracts 17-joint pose vectors per frame; a graph-based Human-Pose aware Attention Mechanism aggregates joint-node features, followed by cross-attention that injects pose-aware tokens into the video representation through residual MSA/FFN layers.
• Object-Relation Expert (ORE): RelTR constructs per-frame scene graphs; a Graph Transformer Network for Masked Object Interactions (GTN-MOI) applies bounding-box masking plus graph transformer layers to strengthen region-specific representations.
• Visual-Background Expert (VBE): SAM segments scene elements; lightweight FFNs refine the resulting background tokens.
• Coarse-grained Video Expert (CVE): standard LanguageBind frame features passed through an FFN.

HigherHRNet, RelTR, and SAM remain frozen throughout training; only the fusion components and LoRA adapters on Video-LLaVA are tuned.

Physical-world Trade-off Regularizer (PTR)

PTR addresses expert imbalance via two mechanisms. A Trade-off Aware Expert Router computes fine-grained expert weights $\mathbf{R}(h)_i$ and couples them to the coarse-grained expert through a gate $\mathbf{G}$ (the maximum fine-grained weight), so coarse and fine contributions are jointly regulated. A Gated Physical-world Trade-off Loss penalizes large aggregated logits:

$$\mathcal{L}_z = \frac{1}{T}\sum_{i=1}^T \left(\log\left(\sum_j e^{x_j^{(i)}} + e^{x_g}\right)\right)^2$$

which suppresses any single expert from dominating. The total objective is $$\mathcal{L} = \mathcal{L}_{ce} + \alpha \mathcal{L}_z$$ with $\alpha = 0.05$.

Training strategy

Stage 1 pre-tunes Video-LLaVA on RefCOCO, HumanML3D, and RSI-CB subsets (25,000 videos, 160,000 images) with manually designed physical-world instructions. Stage 2 performs DeepSVU instruction tuning on the two new datasets using AdamW (lr 2e-5, warm-up ratio 0.03), LoRA (rank 16, scaling 32, dropout 0.05), batch size 8, one epoch, on A100-40GB hardware.

Evaluation protocol

Identification uses False-Negative Rates (FNRs) and F2 score, deliberately recall-weighted since missed threats are costlier than false alarms. Localization uses mAP@tIoU at thresholds {0.1, 0.2, 0.3} for UCF-C and {0.1, 0.3, 0.5} for CUVA. Attribution is scored with ROUGE, BLEU, Sentence-BERT similarity, GPT-based scoring (relevance and sentiment consistency, 1–10), and three-annotator human ratings (1–5). Baselines include seven Video-LLMs (VideoChat, Valley, PandaGPT, mPLUG-Owl, Chat-UniVi, Omni-SILA, Holmes, Hawkeye) reimplemented under identical settings, plus non-LLM detectors (VadCLIP, BiConvLSTM, X3D, CLIP-TSA).

Main results

On CUVA instructions, UPRM achieves an FNR of 10.46 and F2 of 88.41, versus Hawkeye's 13.24/84.35 — reductions of 2.78 FNR and gains of 4.06 F2 and 4.39 average mAP@tIoU (86.24 vs. 81.85). Against the best non-LLM baseline (X3D), the margins are larger: 33.5 FNR reduction, 29.67 F2 improvement, and 19.74 mAP gain. On UCF-C instructions, UPRM reaches 37.28 FNR / 49.33 F2 / 26.25 average mAP, exceeding Holmes by 5.09 FNR, 3.83 F2, and 2.82 mAP, and CLIP-TSA by 27.77 FNR, 17.83 F2, and 11.68 mAP. For attribution, UPRM surpasses Hawkeye by +0.04 SB, +0.03 ROUGE, +6.51 BLEU, +0.19 GPT, and +0.15 human score. These are consistent, if moderate, improvements over the strongest LLM baselines; the dramatic gaps are relative to non-LLM methods, which is expected given their lack of generative capability.

Ablations

Removing any single fine-grained expert degrades performance, with HPE removal most damaging (FNR rises from 10.46 to 15.16 on CUVA). Removing the entire UPE block raises FNR by 9.03 on CUVA and 16.09 on UCF-C; removing PTR raises FNR to 18.07 on CUVA and 44.84 on UCF-C. Expert-weight analysis shows that without PTR the router concentrates weight on the coarse-grained and pose experts, whereas PTR balances contributions — direct evidence for the imbalance hypothesis motivating the regularizer. Removing stage-1 pre-tuning costs 10.7 FNR and 10.29 mAP on CUVA, confirming that physical-world pre-tuning is a substantial contributor rather than incidental. UPRM also converges fastest among compared models (final loss 0.19 on CUVA, 0.23 on UCF-C).

Robustness across scenes

Per-category FNR analysis over 13 UCF-C scenes and 11 CUVA scenes shows UPRM leading in most categories (e.g., FNR 20.99 on Vandalism and 31.64 on Shooting in UCF-C; 2.02 on Water Incident and 3.4 on Burning in CUVA). Notably, the paper concedes that UPRM does not lead in every category: on UCF-C it trails the best results in Arrest, Accident, Explosion, Shoplifting, and Stealing. This heterogeneity suggests the fixed four-expert design does not uniformly transfer across threat types.

Limitations and open questions

Several limitations are stated or evident. First, inference speed: UPRM lags behind other Video-LLMs in per-video inference time, an accuracy–latency trade-off the authors explicitly flag as relevant to real-time deployment. Second, the causal-attribution evaluation depends on CUVA's cause annotations; UCF-Crime provides no ground-truth causes, so attribution quality cannot be verified on UCF-C. Third, the physical-world pipeline relies on frozen third-party extractors (HigherHRNet, RelTR, SAM), meaning upstream errors in pose estimation, scene-graph construction, or segmentation propagate uncorrected into the experts. Fourth, the trade-off loss coefficient $T$0 is tuned empirically without sensitivity analysis reported. Finally, the paper leaves open whether additional real-world information — events, geo-location, semantic mappings — would further improve predictive accuracy, and why certain categories (e.g., Shoplifting, Arrest) resist the current expert design.

Conclusion

DeepSVU reframes security video analysis as a joint identify–locate–attribute problem and demonstrates that explicit coarse-to-fine physical-world modeling, combined with a gated routing regularizer that counteracts data imbalance among experts, yields measurable gains over both Video-LLM and non-LLM baselines on two newly constructed instruction datasets. The ablations attribute the gains specifically to the UPE block, the PTR regularizer, and physical-world pre-tuning, while the conceded weaknesses in inference latency, uneven per-category robustness, and dependence on external annotators delineate the boundaries within which these results hold.