AnomalyLMM: Text-Based Anomaly Search

• The paper introduces AnomalyLMM, the first framework that integrates LMM generative reasoning with a discriminative retrieval backbone for text-based person anomaly search.
• It employs a multi-stage retrieval process that uses masked cloze prompting and semantic re-ranking to align subtle textual anomaly descriptions with brief visual cues.
• Experimental results on the PAB benchmark demonstrate measurable improvements in R@1 and mAP, highlighting the effectiveness of bridging generative knowledge with discriminative retrieval.

AnomalyLMM is a framework for text-based person anomaly search that uses large multimodal models (LMMs) to retrieve individuals exhibiting abnormal behaviors from natural-language descriptions such as falling, being hit, collision, or staggering while clutching their chest (Ju et al., 4 Sep 2025). It is presented as the first framework to harness LMMs for this task and is designed to address two stated difficulties: fine-grained cross-modal alignment between textual anomalies and visual behaviors, and anomaly recognition under sparse real-world samples. Its central design is a coarse-to-fine, training-free retrieval pipeline that bridges generative knowledge in LMMs with discriminative retrieval, rather than using an LMM as a direct end-to-end matcher (Ju et al., 4 Sep 2025).

1. Task formulation and problem characteristics

AnomalyLMM targets a retrieval problem in which the input is a natural-language description of an abnormal pedestrian behavior and the output is a ranked list of images or frames whose top entries should show the person matching that anomalous behavior (Ju et al., 4 Sep 2025). The paper distinguishes this setting from conventional text-based person search, where queries typically emphasize stable appearance attributes such as clothing, gender, or pose. In the anomaly-search setting, the decisive signal is a behavioral event that may be subtle, brief, or only partially visible.

The framework is motivated by two specific challenges. First, anomaly descriptions require fine-grained cross-modal alignment. Verbs such as “falling,” “being hit,” or “staggering” must be matched to small and often transient visual cues rather than to static identity cues. Second, real anomalies are sparse, diverse, and long-tailed, which creates insufficient training data and weak supervision for fine-tuning (Ju et al., 4 Sep 2025).

The paper argues that LMMs are relevant because they are strong at scene understanding and language reasoning, but that they are not directly suited to discriminative ranking. It identifies a domain gap between generative knowledge and discriminative retrieval, together with the absence of efficient adaptation strategies for deployment, as the main obstacles to direct use of LMMs for anomaly retrieval (Ju et al., 4 Sep 2025). AnomalyLMM is therefore formulated as a mechanism for wrapping LMM reasoning around a retrieval backbone.

2. Coarse-to-fine retrieval architecture

The full pipeline is organized into five stages (Ju et al., 4 Sep 2025). First, an off-the-shelf text-to-image retrieval model produces an initial short list of candidates. In the main experiments this coarse retriever is X$^2$VLM, with CMP also evaluated as an alternative. This stage provides efficient candidate generation and reduces the search space for the more expensive reasoning stages.

Second, the query is rewritten into a masked cloze form. An LLM replaces verbs with <VERB> and color-related words with <COLOR>. The stated purpose is to disentangle action semantics from appearance attributes so that downstream reasoning concentrates on anomaly-relevant evidence. A query such as “A man in a red shirt is stumbling and falling” is transformed into a cloze-style template like “A man in a <COLOR> shirt is <VERB> and <VERB>” (Ju et al., 4 Sep 2025).

Third, for each candidate image, the masked query is paired with the image and sent to an LMM for cloze completion. The model fills in the masked tokens using visual evidence. An important design rule is that the model must output UNKNOWN if it cannot confidently infer the missing token from the image, which is intended to reduce hallucination and discourage completions based only on language priors (Ju et al., 4 Sep 2025).

Fourth, the completed text is compared to the original query using an LLM-based semantic comparison step. This comparison is not literal. The paper explicitly allows semantically close verbs and colors to count as matches. Examples given include “balancing” versus “sliding,” and “gray” versus “dark” (Ju et al., 4 Sep 2025). The ranking signal therefore depends on how many placeholders are semantically matched.

Fifth, the initial retrieval score and the re-ranking score are fused. The paper defines the second-stage re-ranking score for candidate $n$ as

$S^{2}_n = \beta^n,$

and the final score as

$$S_n = \alpha_1 S^{1}_n + \alpha_2 S^{2}_n, \quad n \in [0, N-1],$$

where $S^{1}_n$ is the initial retrieval score, $S^{2}_n$ is the re-ranking-derived score, and $\alpha_1,\alpha_2$ balance the two sources (Ju et al., 4 Sep 2025). The paper emphasizes that a strong initial ranking remains important and that the second-stage semantic score acts as a corrective signal.

3. Training-free adaptation cookbook

AnomalyLMM is described as a training-free adaptation cookbook with three named components: masked cross-modal prompting, behavioral saliency prediction, and knowledge-aware re-ranking (Ju et al., 4 Sep 2025).

Masked cross-modal prompting refers to the cloze generation and completion design. By forcing the model to recover masked anomaly-related words from image evidence, the framework attempts to make the LMM attend to the visual evidence relevant to anomalous behavior. This provides a retrieval-oriented use of generative multimodal reasoning rather than direct retrieval from a joint embedding space.

Behavioral saliency prediction is explicitly not introduced as a separate learned saliency module. The paper describes it as an emergent effect of cloze completion: the model is guided to focus on the image regions that support the masked action or attribute tokens (Ju et al., 4 Sep 2025). This is a useful clarification, because a common misunderstanding would be to treat saliency as a separately supervised detector.

Knowledge-aware re-ranking uses LLM semantic comparison to exploit world knowledge about paraphrase and near-equivalence in behavior descriptions. The paper presents this as especially valuable because anomaly descriptions vary widely in wording. The framework therefore does not require exact lexical agreement between the original query and the generated completion (Ju et al., 4 Sep 2025).

A plausible implication is that AnomalyLMM recasts anomaly retrieval as a structured reasoning problem over a short candidate set, rather than as a single monolithic retrieval computation. That interpretation is consistent with the paper’s repeated emphasis on bridging generative and discriminative capabilities.

4. Dataset, implementation, and reported performance

The evaluation is conducted on PAB (Pedestrian Anomaly Behavior), which the paper describes as the first large-scale benchmark for text-based pedestrian anomaly search, while the abstract describes it as the only publicly available benchmark for text-based person anomaly search (Ju et al., 4 Sep 2025). The dataset covers diverse scenarios including running, performing, playing soccer, and ice-skating, together with anomalous counterparts such as lying, being hit, and falling.

The reported dataset composition is as follows:

Split / method	R@1	mAP
X$^2$VLM	83.77%	90.43%
X$^2$VLM + AnomalyLMM	84.73%	90.89%
CMP	69.51%	81.06%
CMP + AnomalyLMM	73.15%	83.64%

The PAB training set contains 1,013,605 synthesized image-text pairs generated by a diffusion-based pipeline, and the test set contains 1,978 real-world image-text pairs with manually verified anomalies (Ju et al., 4 Sep 2025). Metrics are mAP and Recall@K, with Recall@1 emphasized because top-ranked correctness matters most for this fine-grained task.

Most of AnomalyLMM is training-free. The only trained component is the text-to-image retrieval model, which is fine-tuned on PAB with batch size 22, AdamW, weight decay $1\times 10^{-2}$, initial learning rate $n$0, warmup 5000 iterations, and learning-rate decay at $n$1 and $n$2 iterations (Ju et al., 4 Sep 2025).

Quantitatively, X$n$3VLM + AnomalyLMM reaches 84.73% R@1 and 90.89% mAP, compared with 83.77% R@1 and 90.43% mAP for plain X$n$4VLM, corresponding to +0.96% R@1 and +0.46% mAP (Ju et al., 4 Sep 2025). For CMP, the gains are larger: from 69.51% R@1 and 81.06% mAP to 73.15% R@1 and 83.64% mAP, i.e., +3.64% R@1 and +2.58% mAP (Ju et al., 4 Sep 2025). The paper therefore presents AnomalyLMM as beneficial both with a strong retriever and with a weaker coarse stage.

5. Ablations, model choices, and interpretability

The ablation study separates the contribution of the major components. With an X$n$5VLM baseline, cloze completion only yields 84.12 R@1 and 90.60 mAP; re-ranking only yields 84.22 R@1 and 90.65 mAP; and the full method yields 84.73 R@1 and 90.89 mAP, compared with 83.77 R@1 and 90.43 mAP for the baseline (Ju et al., 4 Sep 2025). The paper also states that its own similarity-based ablation using MiniLM embeddings is weaker than LLM semantic comparison.

For score fusion, the preferred weighting strategy is exponential decay with $n$6, outperforming linear decay (Ju et al., 4 Sep 2025). The study of the candidate list length tests $n$7 and reports the best performance with 3 candidates, with the interpretation that too many candidates introduce noise while too few can miss the true match (Ju et al., 4 Sep 2025).

The model-choice experiments report that QWQ-Max works best for cloze generation, QVQ-Max works best for cloze completion, and QWQ-Max or Qwen3-8B performs strongly for re-ranking (Ju et al., 4 Sep 2025). For the fusion coefficients, increasing $n$8 generally improves performance until saturation, whereas $n$9 has a narrow useful range and too much weight hurts performance. This is consistent with the claim that the first-stage retrieval score should dominate and re-ranking should act as correction rather than replacement (Ju et al., 4 Sep 2025).

Qualitative results are used to support interpretability. The paper reports that AnomalyLMM often produces human-interpretable alignments between textual anomalies and visual behaviors. A query verb such as “waving” may be completed as “swinging,” and the framework can detect partial cues such as an impending fall (Ju et al., 4 Sep 2025). The interpretability claim rests on the visibility of the cloze outputs, which expose what behaviors the model inferred and why a candidate was ranked highly.

The stated failure cases are equally specific. Errors arise when the anomaly is very subtle, when the query is ambiguous, or when the visual evidence is incomplete. One example described in the paper is a fall-like pose that resembles a leap, producing confusion (Ju et al., 4 Sep 2025). The authors therefore suggest that stronger temporal/context reasoning would help.

6. Position within LMM- and LLM-based anomaly research

AnomalyLMM belongs to a broader body of work that uses large models in anomaly detection, but it occupies a distinct position within that literature. In AD-LLM, LLMs are benchmarked as zero-shot detectors, data generators and describers, and model selectors for NLP anomaly detection (Yang et al., 2024). In LLM-DAS, the LLM is repositioned from a “data processor” to an “algorithmist” that synthesizes detector-specific hard anomalies for tabular anomaly detection (Ye et al., 4 Oct 2025). In AnomalyR1, a multimodal LLM is trained as an end-to-end industrial anomaly detector with GRPO and ROAM (Chao et al., 16 Apr 2025).

Against that background, AnomalyLMM is notable for using LMMs neither as a direct anomaly scorer nor as a retrained end-to-end detector. Instead, it inserts LMM reasoning into a training-free, coarse-to-fine retrieval pipeline whose final decision is still anchored by a discriminative retriever and a fusion rule (Ju et al., 4 Sep 2025). This suggests a specific design pattern within anomaly research: rather than replacing specialized retrieval systems, large multimodal models can be used to refine them through structured prompting, semantic completion, and reasoning-aware re-ranking.

The paper’s broader implication is therefore narrower than a claim of universal anomaly understanding. It shows that, for text-based person anomaly search, LMMs can contribute measurable gains when their generative world knowledge is constrained by a retrieval backbone, a masked-query protocol, and a semantically informed second-stage ranking mechanism (Ju et al., 4 Sep 2025).