Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search

Abstract: Text-based person anomaly search retrieves specific behavioral events from surveillance archives using natural-language queries. Although recent pose-aware methods align geometric structures well, they face a fundamental Pose-Semantic Gap: semantically different actions can share similar skeletal geometries. While Multimodal LLMs (MLLMs) can reduce this ambiguity, using them for large-scale retrieval is computationally prohibitive. We propose the Structure-Semantic Decoupled Cascade (SSDC) framework, which decouples retrieval into two stages: (1) Structure-Aware Coarse Retrieval, where a lightweight model quickly filters candidates by skeletal similarity ; and (2) Detective Squad Interaction, a multi-agent semantic verification module. The squad consists of a Detective for fast binary filtering, an Analyst for evidence extraction, and a Writer for semantic synthesis. Finally, we re-rank candidates by fusing the synthesized captions with structural priors. Experiments on the PAB benchmark show that SSDC achieves state-of-the-art performance by balancing efficiency and semantic reasoning.

• The paper introduces a cascade framework (SSDC) that decouples structural filtering from semantic reasoning to overcome the pose-semantic gap in text-based person anomaly search.
• It leverages a lightweight pose-aware model for coarse retrieval and a multi-agent system for fine-grained semantic verification, achieving notable improvements (87.01% Rank-1, 92.74% mAP).
• The framework efficiently reduces computational overhead by activating deep semantic analysis only for high-confidence candidates, ensuring robust and scalable anomaly search.

Bridging the Pose-Semantic Gap in Text-Based Person Anomaly Search

Motivation and Challenge: Pose-Semantic Gap

Text-based person anomaly search, which retrieves anomalous behavioral events from surveillance data via natural language queries, is fundamentally limited by the "Pose-Semantic Gap". Conventional pose-driven models robustly match geometric structures, but fail to distinguish semantically divergent actions that manifest similar skeletal configurations (e.g., falling versus doing push-ups), resulting in high false positives. Multimodal LLMs (MLLMs) are promising for semantic differentiation but pose prohibitive computational costs for large-scale retrieval. Hence, a solution must balance high-throughput structural filtering with deep semantic reasoning.

Figure 1: The SSDC framework bridges the Pose-Semantic Gap by combining structural filtering with a collaborative agent workflow to generate semantically accurate captions.

SSDC Framework: Structure-Semantic Decoupled Cascade

The proposed Structure-Semantic Decoupled Cascade (SSDC) framework systematically bridges the pose-semantic gap. The architecture decomposes retrieval into two stages:

1. Structure-Aware Coarse Retrieval: Utilizes a lightweight pose-aware model to rapidly recall candidate images based on structural similarity, employing pose-aware cross-attention to fuse skeletal and appearance features. Hard negative mining ensures the retriever learns to decouple appearance and action semantics.
2. Detective Squad Interaction: Invokes a multi-agent verification pipeline—Detective (binary filter), Analyst (evidence extraction), Writer (caption synthesis). The squad engages in multi-round reasoning, extracting fine-grained visual evidence and synthesizing high-fidelity semantic anchors for re-ranking.

   Figure 2: Coarse-to-fine pipeline architecture of the SSDC framework, with structural filtering, semantic verification, and adaptive fusion mechanisms.

Detective Squad: Collaborative Semantic Verification

The Detective Squad is instantiated via supervised fine-tuning with structural hard negatives mined from Stage I. Each agent is role-tuned via specific instruction templates and LoRA adaptation. The reasoning protocol follows:

• The Detective discards clear negatives.
• The Analyst extracts attributes from a domain-relevant checklist.
• The Writer synthesizes a refined caption, $T_{new}$.

Semantic scoring is performed by computing the cosine similarity between the user query and $T_{new}$ embeddings from a frozen text encoder (BERT/CLIP). Adaptive fusion integrates structural priors ($S_{str}$) and semantic verdicts ($S_{sem}$), gating computationally intensive semantic analysis by a high-confidence threshold.

Figure 3: Stage-wise pipeline of the Detective Squad, including coarse retrieval, multi-agent semantic verification, and final score fusion.

Experimental Results and Analysis

Quantitative Performance

SSDC demonstrates superior performance across the Pedestrian Anomaly Behavior (PAB) benchmark, encompassing over 1M image-text pairs and diverse anomaly/normal action types. The complete SSDC pipeline achieves 87.01% Rank-1 and 92.74% mAP, outperforming both general vision-language pretraining models and specialized pose-aware retrieval baselines by 2-8% absolute in Rank-1 under various environmental conditions. Integrating the Detective Squad into simpler backbones consistently elevates retrieval accuracy, confirming the efficacy of collaborative verification for ambiguous cases.

Robustness and Generalization

SSDC maintains performance under multi-weather perturbations and strongly generalizes to the UCF-Crime-derived UCC test set (59.45% R@1, 45.25% mAP), exceeding baselines by 5-7%. This validates its resilience against environmental degradation and OOD dataset shift due to explicit semantic reasoning and fine-grained evidence integration.

Ablations: Agent Roles and Model Selection

Ablation studies show the Analyst's evidence extraction is the critical bottleneck, with the full squad outperforming individual agents. LoRA fine-tuning is indispensable for domain alignment. Among foundation models, Qwen3-VL-8B demonstrates optimal balance for both visual chain-of-thought and instruction following.

Efficiency

Cascade filtering reduces computational burden by activating semantic verification only for high-confidence candidates (99.9% reduction in gallery size). The Detective agent acts as a gatekeeper in dynamic inference, focusing semantic scrutiny on ambiguous samples.

Parameter Sensitivity

Performance peaks when the semantic score weight is 0.6 ($\lambda=0.4$) and the structural confidence threshold is high ($\xi=0.95$), indicating reliance on semantic reasoning for final disambiguation.

Figure 4: Parameter sensitivity analysis confirms optimal threshold and weighting for activation/fusion strategies.

Figure 5: Rank-1 and mAP evolve across multi-agent interaction rounds, with most gains accruing in the initial reasoning cycle and stabilizing quickly.

Implications and Future Directions

SSDC's cascade decoupling and multi-agent verification paradigm advance text-based anomaly retrieval by resolving fine-grained behavioral ambiguities without prohibitive overhead. The explicit semantic anchoring via collaborative agents offers interpretability, robust generalization, and resilience to environmental variance. Practically, the framework is deployable in real-time surveillance analytics at scale.

Theoretically, this approach substantiates the utility of multi-agent prompting, hierarchical reasoning, and explicit semantic fusion in multimodal retrieval, suggesting future work in more granular persona tuning, adaptive activation policies, and extension to multi-object and temporal anomaly scenarios. Further research into domain-agnostic instruction tuning and cascading reasoning architectures may yield broader vision-language retrieval generalization.

Conclusion

The SSDC framework for text-based person anomaly search systematically bridges the pose-semantic gap through cascade decoupling and collaborative semantic reasoning. Extensive quantitative, ablation, and robustness analyses establish its dominance over baseline architectures, demonstrating that multi-agent hierarchical verification and dynamic inference are core to high-fidelity, efficient anomaly search (2604.23282).