---
title: 'AnomalyLMM: Text-Based Anomaly Search'
url: https://www.emergentmind.com/topics/anomalylmm
type: topic
---

# AnomalyLMM: Text-Based Anomaly Search

AnomalyLMM is a framework for **text-based person anomaly search** that uses **large multimodal models (LMMs)** to retrieve individuals exhibiting abnormal behaviors from natural-language descriptions such as *falling*, *being hit*, *collision*, or *staggering while clutching their chest* [2509.04376]. It is presented as the **first framework** to harness LMMs for this task and is designed to address two stated difficulties: **fine-grained cross-modal alignment between textual anomalies and visual behaviors**, and **anomaly recognition under sparse real-world samples**. Its central design is a **coarse-to-fine, training-free retrieval pipeline** that bridges **generative knowledge** in LMMs with **discriminative retrieval**, rather than using an LMM as a direct end-to-end matcher [2509.04376].

## 1. Task formulation and problem characteristics

AnomalyLMM targets a retrieval problem in which the input is a natural-language description of an abnormal pedestrian behavior and the output is a ranked list of images or frames whose top entries should show the person matching that anomalous behavior [2509.04376]. The paper distinguishes this setting from conventional text-based person search, where queries typically emphasize stable appearance attributes such as clothing, gender, or pose. In the anomaly-search setting, the decisive signal is a **behavioral event** that may be **subtle, brief, or only partially visible**.

The framework is motivated by two specific challenges. First, anomaly descriptions require **fine-grained cross-modal alignment**. Verbs such as “falling,” “being hit,” or “staggering” must be matched to small and often transient visual cues rather than to static identity cues. Second, real anomalies are **sparse, diverse, and long-tailed**, which creates insufficient training data and weak supervision for fine-tuning [2509.04376].

The paper argues that LMMs are relevant because they are strong at scene understanding and language reasoning, but that they are not directly suited to discriminative ranking. It identifies a **domain gap between generative knowledge and discriminative retrieval**, together with the **absence of efficient adaptation strategies for deployment**, as the main obstacles to direct use of LMMs for anomaly retrieval [2509.04376]. AnomalyLMM is therefore formulated as a mechanism for wrapping LMM reasoning around a retrieval backbone.

## 2. Coarse-to-fine retrieval architecture

The full pipeline is organized into five stages [2509.04376]. First, an off-the-shelf text-to-image retrieval model produces an initial short list of candidates. In the main experiments this coarse retriever is **X$^2$VLM**, with **CMP** also evaluated as an alternative. This stage provides efficient candidate generation and reduces the search space for the more expensive reasoning stages.

Second, the query is rewritten into a **masked cloze** form. An LLM replaces **verbs** with `<VERB>` and **color-related words** with `<COLOR>`. The stated purpose is to disentangle **action semantics** from **appearance attributes** so that downstream reasoning concentrates on anomaly-relevant evidence. A query such as “A man in a red shirt is stumbling and falling” is transformed into a cloze-style template like “A man in a `<COLOR>` shirt is `<VERB>` and `<VERB>`” [2509.04376].

Third, for each candidate image, the masked query is paired with the image and sent to an LMM for **cloze completion**. The model fills in the masked tokens using visual evidence. An important design rule is that the model must output **UNKNOWN** if it cannot confidently infer the missing token from the image, which is intended to reduce hallucination and discourage completions based only on language priors [2509.04376].

Fourth, the completed text is compared to the original query using an LLM-based **semantic comparison** step. This comparison is not literal. The paper explicitly allows semantically close verbs and colors to count as matches. Examples given include “balancing” versus “sliding,” and “gray” versus “dark” [2509.04376]. The ranking signal therefore depends on how many placeholders are semantically matched.

Fifth, the initial retrieval score and the re-ranking score are fused. The paper defines the second-stage re-ranking score for candidate \(n\) as
\[
S^{2}_n = \beta^n,
\]
and the final score as
\[
S_n = \alpha_1 S^{1}_n + \alpha_2 S^{2}_n, \quad n \in [0, N-1],
\]
where \(S^{1}_n\) is the initial retrieval score, \(S^{2}_n\) is the re-ranking-derived score, and \(\alpha_1,\alpha_2\) balance the two sources [2509.04376]. The paper emphasizes that a strong initial ranking remains important and that the second-stage semantic score acts as a corrective signal.

## 3. Training-free adaptation cookbook

AnomalyLMM is described as a **training-free adaptation cookbook** with three named components: **masked cross-modal prompting**, **behavioral saliency prediction**, and **knowledge-aware re-ranking** [2509.04376].

**Masked cross-modal prompting** refers to the cloze generation and completion design. By forcing the model to recover masked anomaly-related words from image evidence, the framework attempts to make the LMM attend to the visual evidence relevant to anomalous behavior. This provides a retrieval-oriented use of generative multimodal reasoning rather than direct retrieval from a joint embedding space.

**Behavioral saliency prediction** is explicitly not introduced as a separate learned saliency module. The paper describes it as an emergent effect of cloze completion: the model is guided to focus on the image regions that support the masked action or attribute tokens [2509.04376]. This is a useful clarification, because a common misunderstanding would be to treat saliency as a separately supervised detector.

**Knowledge-aware re-ranking** uses LLM semantic comparison to exploit world knowledge about paraphrase and near-equivalence in behavior descriptions. The paper presents this as especially valuable because anomaly descriptions vary widely in wording. The framework therefore does not require exact lexical agreement between the original query and the generated completion [2509.04376].

A plausible implication is that AnomalyLMM recasts anomaly retrieval as a structured reasoning problem over a short candidate set, rather than as a single monolithic retrieval computation. That interpretation is consistent with the paper’s repeated emphasis on bridging **generative** and **discriminative** capabilities.

## 4. Dataset, implementation, and reported performance

The evaluation is conducted on **PAB (Pedestrian Anomaly Behavior)**, which the paper describes as the **first large-scale benchmark for text-based pedestrian anomaly search**, while the abstract describes it as the **only publicly available benchmark for text-based person anomaly search** [2509.04376]. The dataset covers diverse scenarios including running, performing, playing soccer, and ice-skating, together with anomalous counterparts such as lying, being hit, and falling.

The reported dataset composition is as follows:

| Split / method | R@1 | mAP |
|---|---:|---:|
| X$^2$VLM | 83.77% | 90.43% |
| X$^2$VLM + AnomalyLMM | 84.73% | 90.89% |
| CMP | 69.51% | 81.06% |
| CMP + AnomalyLMM | 73.15% | 83.64% |

The PAB training set contains **1,013,605 synthesized image-text pairs** generated by a diffusion-based pipeline, and the test set contains **1,978 real-world image-text pairs** with manually verified anomalies [2509.04376]. Metrics are **mAP** and **Recall@K**, with **Recall@1** emphasized because top-ranked correctness matters most for this fine-grained task.

Most of AnomalyLMM is **training-free**. The only trained component is the text-to-image retrieval model, which is fine-tuned on PAB with **batch size 22**, **AdamW**, **weight decay \(1\times 10^{-2}\)**, **initial learning rate \(5\times 10^{-5}\)**, **warmup 5000 iterations**, and learning-rate decay at **\(2\times 10^4\)** and **\(3.5\times 10^4\)** iterations [2509.04376].

Quantitatively, **X$^2$VLM + AnomalyLMM** reaches **84.73% R@1** and **90.89% mAP**, compared with **83.77% R@1** and **90.43% mAP** for plain X$^2$VLM, corresponding to **+0.96% R@1** and **+0.46% mAP** [2509.04376]. For **CMP**, the gains are larger: from **69.51% R@1** and **81.06% mAP** to **73.15% R@1** and **83.64% mAP**, i.e., **+3.64% R@1** and **+2.58% mAP** [2509.04376]. The paper therefore presents AnomalyLMM as beneficial both with a strong retriever and with a weaker coarse stage.

## 5. Ablations, model choices, and interpretability

The ablation study separates the contribution of the major components. With an X$^2$VLM baseline, **cloze completion only** yields **84.12 R@1** and **90.60 mAP**; **re-ranking only** yields **84.22 R@1** and **90.65 mAP**; and the **full method** yields **84.73 R@1** and **90.89 mAP**, compared with **83.77 R@1** and **90.43 mAP** for the baseline [2509.04376]. The paper also states that its own similarity-based ablation using MiniLM embeddings is weaker than LLM semantic comparison.

For score fusion, the preferred weighting strategy is **exponential decay** with \(\beta = 0.5\), outperforming linear decay [2509.04376]. The study of the candidate list length tests \(N \in \{2,3,4,5\}\) and reports the best performance with **3 candidates**, with the interpretation that too many candidates introduce noise while too few can miss the true match [2509.04376].

The model-choice experiments report that **QWQ-Max** works best for cloze generation, **QVQ-Max** works best for cloze completion, and **QWQ-Max** or **Qwen3-8B** performs strongly for re-ranking [2509.04376]. For the fusion coefficients, increasing \(\alpha_1\) generally improves performance until saturation, whereas \(\alpha_2\) has a narrow useful range and too much weight hurts performance. This is consistent with the claim that the first-stage retrieval score should dominate and re-ranking should act as correction rather than replacement [2509.04376].

Qualitative results are used to support interpretability. The paper reports that AnomalyLMM often produces **human-interpretable alignments** between textual anomalies and visual behaviors. A query verb such as “waving” may be completed as “swinging,” and the framework can detect partial cues such as an impending fall [2509.04376]. The interpretability claim rests on the visibility of the cloze outputs, which expose what behaviors the model inferred and why a candidate was ranked highly.

The stated failure cases are equally specific. Errors arise when the anomaly is **very subtle**, when the query is **ambiguous**, or when the **visual evidence is incomplete**. One example described in the paper is a fall-like pose that resembles a leap, producing confusion [2509.04376]. The authors therefore suggest that stronger **temporal/context reasoning** would help.

## 6. Position within LMM- and LLM-based anomaly research

AnomalyLMM belongs to a broader body of work that uses large models in anomaly detection, but it occupies a distinct position within that literature. In **AD-LLM**, large language models are benchmarked as **zero-shot detectors**, **data generators and describers**, and **model selectors** for NLP anomaly detection [2412.11142]. In **LLM-DAS**, the language model is repositioned from a “data processor” to an “algorithmist” that synthesizes detector-specific hard anomalies for tabular anomaly detection [2510.03904]. In **AnomalyR1**, a multimodal large language model is trained as an end-to-end industrial anomaly detector with GRPO and ROAM [2504.11914].

Against that background, AnomalyLMM is notable for using LMMs neither as a direct anomaly scorer nor as a retrained end-to-end detector. Instead, it inserts LMM reasoning into a **training-free, coarse-to-fine retrieval pipeline** whose final decision is still anchored by a discriminative retriever and a fusion rule [2509.04376]. This suggests a specific design pattern within anomaly research: rather than replacing specialized retrieval systems, large multimodal models can be used to refine them through structured prompting, semantic completion, and reasoning-aware re-ranking.

The paper’s broader implication is therefore narrower than a claim of universal anomaly understanding. It shows that, for **text-based person anomaly search**, LMMs can contribute measurable gains when their **generative world knowledge** is constrained by a retrieval backbone, a masked-query protocol, and a semantically informed second-stage ranking mechanism [2509.04376].

Source: https://www.emergentmind.com/topics/anomalylmm