AnomalyMoE: Hierarchical Anomaly Detection

• AnomalyMoE is a unified visual anomaly detection framework that decomposes anomalies into local, component-level, and global logical types.
• It leverages a frozen DINOv2 encoder and sparse expert routing to efficiently reconstruct features at multiple semantic levels for diverse applications.
• Auxiliary mechanisms like Expert Selection Balancing and Expert Information Repulsion promote diversity and enhance overall anomaly detection performance.

Searching arXiv for the cited AnomalyMoE paper and closely related MoE anomaly-detection work. AnomalyMoE is a unified visual anomaly detection framework that formulates anomaly detection as a hierarchically decomposed sparse Mixture-of-Experts problem. It is designed as a language-free generalist model that addresses a broad spectrum of anomaly types by separating them into three semantic hierarchies—local structural anomalies, component-level semantic anomalies, and global logical anomalies—and assigning each hierarchy to a dedicated family of experts. In its reported formulation, a frozen DINOv2 ViT-B/14 encoder produces patch and global representations, a lightweight router performs top-$K$ sparse expert selection from the global embedding, and the selected experts reconstruct features at their assigned semantic level. Two auxiliary mechanisms, Expert Selection Balancing and Expert Information Repulsion, regularize routing and discourage redundancy among experts. The resulting system is evaluated on eight datasets spanning industrial imaging, 3D point clouds, medical imaging, video surveillance, and logical anomaly detection, where it is reported to establish new state-of-the-art average performance while remaining substantially faster at inference than language-prior-based alternatives (Gu et al., 8 Aug 2025).

1. Conceptual scope and problem formulation

AnomalyMoE is motivated by the observation that existing anomaly detection methods are often highly specialized for particular anomaly types, such as textural defects or logical errors, and therefore exhibit limited performance outside their designated contexts. The framework addresses this specialization problem by using a single MoE architecture that explicitly decomposes anomaly detection into three semantic hierarchies: local structural anomalies, component-level semantic anomalies, and global logical anomalies (Gu et al., 8 Aug 2025).

This decomposition is the central organizing principle of the method. Local structural anomalies correspond to fine-grained deviations visible at the patch level. Component-level semantic anomalies concern errors in part identity or component semantics. Global logical anomalies refer to inconsistencies in overall arrangement or scene-level structure. AnomalyMoE assigns each level to a distinct expert family, with the aim of enabling one model to concurrently understand and detect heterogeneous anomaly phenomena rather than requiring separate specialized systems (Gu et al., 8 Aug 2025).

The method is explicitly described as language-free. In context, this means that it does not depend on LLM or VLM priors. The paper contrasts this design with component/global anomaly detection methods based on language priors, naming UniVAD and LogSAD among the baselines, and reports substantially lower inference latency than those systems (Gu et al., 8 Aug 2025). A plausible implication is that the model’s generality is intended to arise from structured expert specialization rather than cross-modal language supervision.

2. Core architecture and semantic expert hierarchy

The architecture begins with a frozen DINOv2 ViT-B/14 encoder, which processes an input image, point cloud, or slice into patch embeddings $F_p \in \mathbb{R}^{N\times D}$ and a global $[CLS]$ embedding $E_{[cls]} \in \mathbb{R}^D$. A lightweight router implemented as a single FFN computes logits for all experts from the global embedding, and top-$K$ sparse routing selects the $K=3$ experts with the highest logits. A Softmax over those selected logits yields the gate weights $G_j$, while all non-selected experts receive zero weight (Gu et al., 8 Aug 2025).

The expert pool is organized into three semantic groups, each containing six experts. Patch-level experts address local structural anomalies; component-level experts address component semantics; global-level experts address logical anomalies. In the reported configuration, $N_p=N_c=N_g=6$, giving a total of 18 experts, with sparse selection controlled by the router (Gu et al., 8 Aug 2025).

The main training signal is a weighted combination of the selected experts’ reconstruction losses. The selected experts reconstruct features at their assigned semantic level, and those losses are combined according to the router’s gate weights. In parallel, Expert Selection Balancing (ESB) regularizes routing usage and Expert Information Repulsion (EIR) promotes diversity among expert outputs (Gu et al., 8 Aug 2025).

This architecture places AnomalyMoE within a broader family of anomaly-detection methods that use MoE factorization to address heterogeneity, but its distinctive feature is the semantic hierarchy. Other recent MoE-based anomaly systems divide normality or anomaly structure in different ways: Graph-MoE uses one expert per GNN layer for multivariate time-series anomaly detection (Huang et al., 2024), Adapted-MoE partitions same-category normal samples into subclass-specific experts with test-time adaptation (Lei et al., 2024), and UniMMAD uses a sparsely-gated cross MoE to decompress unified features across modalities and classes (Zhao et al., 30 Sep 2025). AnomalyMoE differs by explicitly aligning expert families with structural, semantic, and logical levels of visual abnormality (Gu et al., 8 Aug 2025).

3. Expert families and reconstruction objectives

The patch-level experts are designed for local structural anomalies. Each patch-level expert is a small Transformer decoder block that uses Linear Attention instead of Softmax Attention. Its input is $F_{target}$ corrupted by Gaussian noise and dropout with $p=0.2$, and its reconstruction head outputs $F_p \in \mathbb{R}^{N\times D}$0. The associated loss is the mean cosine distance: $F_p \in \mathbb{R}^{N\times D}$1 At inference, pointwise cosine distances produce an anomaly map $F_p \in \mathbb{R}^{N\times D}$2 (Gu et al., 8 Aug 2025).

The component-level experts are intended for component-level semantic anomalies. The method first precomputes a K-means component knowledge base for normal samples and assigns each patch to one of $F_p \in \mathbb{R}^{N\times D}$3 clusters to form masks. The target features are then masked-average-pooled into component embeddings $F_p \in \mathbb{R}^{N\times D}$4. Each of the $F_p \in \mathbb{R}^{N\times D}$5 experts is an MLP autoencoder with a bottleneck that reconstructs $F_p \in \mathbb{R}^{N\times D}$6. The loss is again based on mean cosine distance: $F_p \in \mathbb{R}^{N\times D}$7 Inference scores are computed as $F_p \in \mathbb{R}^{N\times D}$8 (Gu et al., 8 Aug 2025).

The global-level experts address global logical anomalies. Each is a convolutional autoencoder with an encoder $F_p \in \mathbb{R}^{N\times D}$9 and decoder $[CLS]$0 that reconstructs $[CLS]$1 from the raw input $[CLS]$2. The loss is a squared $[CLS]$3 reconstruction error: $[CLS]$4 At inference, per-location squared error yields the score map $[CLS]$5 (Gu et al., 8 Aug 2025).

Taken together, these three expert types instantiate a structured multi-resolution notion of abnormality. This suggests that AnomalyMoE treats anomaly detection not as a single reconstruction problem, but as a gated collection of reconstruction subproblems defined at different semantic scales.

4. Routing, sparsity, and auxiliary regularization

The routing mechanism takes the global embedding $[CLS]$6 and a weight matrix $[CLS]$7. It computes logits

$[CLS]$8

then selects the top-$[CLS]$9 indices and defines gate weights

$E_{[cls]} \in \mathbb{R}^D$0

In the reported setup, $E_{[cls]} \in \mathbb{R}^D$1 (Gu et al., 8 Aug 2025).

The router is not trained through an explicit routing supervision objective. Instead, its gradients arise indirectly from three sources: the weighted expert reconstruction losses, the ESB loss, and the EIR loss. This coupling makes routing an emergent consequence of reconstruction quality and regularized expert allocation (Gu et al., 8 Aug 2025).

EIR is introduced to prevent multiple experts from learning redundant features and to enforce functional specialization. The method approximates mutual information $E_{[cls]} \in \mathbb{R}^D$2 between the outputs $E_{[cls]} \in \mathbb{R}^D$3 and $E_{[cls]} \in \mathbb{R}^D$4 of two experts using the CLUB upper bound: $E_{[cls]} \in \mathbb{R}^D$5 The total repulsion penalty sums this quantity over all $E_{[cls]} \in \mathbb{R}^D$6 within each expert group: $E_{[cls]} \in \mathbb{R}^D$7 Its role is explicitly to promote expert diversity (Gu et al., 8 Aug 2025).

ESB is introduced to avoid router collapse, where only a few experts are repeatedly selected. For a batch of size $E_{[cls]} \in \mathbb{R}^D$8, the average gate probability of expert $E_{[cls]} \in \mathbb{R}^D$9 is

$K$0

and $K$1 is the number of samples in the batch assigned to expert $K$2. The ESB loss is

$K$3

In practice, $K$4 (Gu et al., 8 Aug 2025).

The joint use of sparse top-$K$5 routing, balancing, and repulsion situates AnomalyMoE near the standard engineering pattern of sparse MoE systems, but with anomaly-detection-specific semantics. Related MoE anomaly frameworks also include explicit balancing or diversity constraints, though with different formulations: UniMMAD uses a load-balancing regularizer based on the coefficient of variation of gating probabilities (Zhao et al., 30 Sep 2025), and MoECLIP applies both a balance loss and an ETF loss to prevent expert collapse in zero-shot anomaly detection (Park et al., 3 Mar 2026). AnomalyMoE’s contribution is to bind these regularizers to semantic-level experts rather than purely architectural expert partitions (Gu et al., 8 Aug 2025).

5. Optimization, inference, and anomaly scoring

The total training objective combines the gated reconstruction loss with the two auxiliary regularizers: $K$6 where $K$7 is $K$8, $K$9, or $K=3$0 depending on the selected expert type (Gu et al., 8 Aug 2025).

The reported optimization settings are AdamW with learning rate $K=3$1, weight decay $K=3$2, batch size 16 on 4$K=3$3A6000 GPUs, and 50 000 iterations. The auxiliary loss weights are $K=3$4 and $K=3$5. The paper states that no curriculum is used and that training proceeds with a simple constant learning rate (Gu et al., 8 Aug 2025).

At inference, anomaly evidence is derived from reconstruction discrepancies at the relevant semantic level. Patch experts produce pointwise cosine-distance maps, component experts produce component-level cosine-distance scores, and global experts produce per-location squared-error maps (Gu et al., 8 Aug 2025). The paper’s summary emphasizes image-level AUC and pixel-level AUC as the reported evaluation metrics, indicating that these level-specific discrepancies are aggregated into both localization and image-level anomaly measures (Gu et al., 8 Aug 2025).

A notable property of the system is inference efficiency. The reported inference latency is 58.5 ms/image, compared with 658.5 ms for UniVAD and 1932.6 ms for LogSAD (Gu et al., 8 Aug 2025). Because the model is language-free and uses sparse top-$K=3$6 expert activation, this suggests that its computational profile benefits from both the absence of language-model overhead and the selective use of expert subnetworks.

6. Empirical performance, ablations, and reported limitations

AnomalyMoE is evaluated on eight datasets: MVTec AD, VisA, MVTec 3D-AD, MVTec-LOCO, BrainMRI, LiverCT, RESC, and UCSD Ped2. These datasets span industrial images, industrial 3D point clouds, logical anomalies, medical imaging, and video surveillance. The reported baselines include patch-level reconstruction methods such as UniAD, ReContrast, UniNet, and Dinomaly, as well as component/global methods using language priors, notably UniVAD and LogSAD (Gu et al., 8 Aug 2025).

The paper reports that AnomalyMoE outperforms all baselines on average. Representative image-level results include 99.5% on MVTec AD versus Dinomaly’s 98.8%, 98.1% on VisA versus 95.7%, 87.5% on MVTec-LOCO versus 86.2%, 92.1% on BrainMRI versus 90.9%, and 97.1% on Ped2, tied with the best baseline (Gu et al., 8 Aug 2025). These comparisons are presented as evidence that a single hierarchical MoE can compete across anomaly regimes that are often addressed by separate specialized models.

The ablation studies isolate the contributions of semantic hierarchy and auxiliary routing regularization. A semantic-level ablation compares patch only, patch+component, patch+global, and the full three-level model; the full model is reported as consistently best, with MVTec AD improving from patch only $K=3$7 to the full model $K=3$8 in image-level and pixel-level AUC (Gu et al., 8 Aug 2025). Auxiliary module ablations show that removing ESB produces a performance drop of approximately 0.3–1% on most datasets, while removing EIR causes a drop of approximately 0.1–1%, especially on logical tasks (Gu et al., 8 Aug 2025).

Sensitivity analysis further reports that expert count $K=3$9 is best, top-$G_j$0 is best, $G_j$1 is optimal, and $G_j$2 is optimal (Gu et al., 8 Aug 2025). These findings indicate that both the degree of sparsity and the strength of the auxiliary losses materially affect performance.

The paper also identifies limitations. It states that the method requires precomputed component clusters via K-means for the component experts, is sensitive to auxiliary-loss weights, and uses a fixed $G_j$3 that may not suit all tasks. It proposes several extensions, including incorporating temporal experts for video sequences beyond frame-wise anomalies, learning component masks end-to-end rather than through K-means, extending to multi-modal inputs such as RGB+D or audio, and exploring more advanced sparse-routing schemes or learnable $G_j$4 (Gu et al., 8 Aug 2025).

7. Position within the MoE anomaly-detection literature

AnomalyMoE belongs to an increasingly diverse line of anomaly-detection methods that use expert decomposition to capture heterogeneity, but its problem framing is distinct. In multivariate time-series anomaly detection, Graph-MoE attaches one expert per GNN layer and uses a memory-augmented router to integrate hierarchical graph information before conditional density estimation; it reports gains on SWaT, WADI, PSM, MSL, and SMD, and emphasizes plug-and-play integration with GNN-based detectors (Huang et al., 2024). In image anomaly detection, Adapted-MoE partitions normal data into multiple subclass regions and applies test-time adaptation to align unseen test features with expert-specific distributions, improving I-AUROC and P-AUROC on the Texture AD benchmark (Lei et al., 2024). In unified multi-modal and multi-class anomaly detection, UniMMAD uses a sparsely-gated cross MoE in a decoder to decompress general features into modality-specific and class-specific forms (Zhao et al., 30 Sep 2025). In zero-shot anomaly detection, MoECLIP uses patch-specialized LoRA experts with per-patch routing inside a frozen CLIP vision encoder (Park et al., 3 Mar 2026).

Within this broader literature, AnomalyMoE’s distinguishing property is the explicit semantic-hierarchy prior. Rather than dividing experts by data subclass, graph depth, modality, class, or patch subspace, it divides them by anomaly semantics: local structure, component meaning, and global logic (Gu et al., 8 Aug 2025). This suggests a more ontology-driven decomposition of anomaly space.

The naming of “AnomalyMoE” is itself somewhat ambiguous in the wider literature. The 2021 “Rare and Different” method combines an autoregressive flow with an ensemble of Deep SVDD models and is described in one summary as AnomalyMoE, but it does not use a learned gating network or semantic expert hierarchy; its fusion consists of fixed operators such as AND, OR, PROD, and AVG applied to rareness and differentness scores (Caron et al., 2021). In contrast, the 2025 AnomalyMoE paper defines a specific sparse-routed visual MoE architecture centered on hierarchical semantic experts (Gu et al., 8 Aug 2025). For precision, AnomalyMoE in current usage most specifically denotes the latter language-free generalist visual anomaly detector.

In that sense, AnomalyMoE represents a convergence point between generalist anomaly detection and sparse conditional computation. It preserves the modular advantages of MoE systems while grounding expert specialization in a structured taxonomy of anomaly types. A plausible implication is that future extensions may generalize this semantic partitioning to temporal, multimodal, or open-world settings, especially since the paper already identifies temporal experts, end-to-end component discovery, and richer sparse-routing schemes as natural next steps (Gu et al., 8 Aug 2025).