AVAR-Net: Multimodal Anomaly Recognition

• AVAR-Net is a lightweight multimodal anomaly recognition framework designed to fuse audio and visual cues, enhancing detection under occlusion and low illumination.
• It employs efficient feature extraction using a MobileViT backbone for video and Wav2Vec2 for audio, integrated via an early cross-modal fusion strategy.
• Benchmark results on the VAAR and XD-Violence datasets demonstrate high accuracy and average precision, setting a new baseline for robust anomaly detection.

AVAR-Net is a lightweight, multimodal deep learning framework for audio-visual anomaly recognition, proposed to address unreliability in unimodal anomaly detection systems operating under occlusion, low illumination, and adverse conditions. It introduces an efficient cross-modal architecture and a medium-scale, real-world synchronized audio-visual dataset, advancing benchmarked evaluation in multimodal anomaly recognition. AVAR-Net achieves high recognition accuracy and average precision on established datasets and sets a new baseline for both methodological design and empirical performance in the field (Ali et al., 15 Oct 2025).

1. Motivation and Context

Audio-visual anomaly recognition is crucial in applications demanding robustness, such as surveillance, transportation, healthcare, and public safety, where reliance on a single modality (typically visual) limits the detection of abnormal events during challenging conditions (e.g., occlusion and poor lighting). Prior methods fall short due to a lack of quality multimodal datasets and limited architectural strategies for fusing heterogeneous inputs. AVAR-Net addresses both limitations by designing a computationally efficient audio-visual anomaly detector and by constructing the Visual-Audio Anomaly Recognition (VAAR) dataset, a novel benchmark comprising 3,000 synchronized video-audio clips spanning ten diverse anomaly classes.

2. Network Architecture and Components

AVAR-Net is structured into four principal modules, each optimized for speed and representation expressivity:

1. Video Feature Extractor:
   • Backbone: Pre-trained MobileViT model for visual frames.
   • Processing Pipeline: Framewise convolutional encoding followed by an unfolding–Transformer–folding sequence enables efficient local and global context integration.
   • Skip Connections: Restore resolution and fuse local-global representations.
   • Mathematical Details:
     • Global relations: $$X_G(p) = \text{Transformer}(X_{\text{Unfold}}(p)),\ 1 \leq p \leq P$$.
2. Audio Feature Extractor:
   • Backbone: Wav2Vec2 model processing raw waveform inputs.
   • Feature Construction: Sequential 1D convolutions, group normalization, and GELU activations yield temporal-spectral representations.
   • Dimensionality: Initial features (512-d), projected to 768-d, enriched with positional embeddings; audio encoder consists of 12 Transformer layers with self-attention and two-layer FFN components.
   • Representative Equations:
     • Convolutional layer: $$f^{(i)} = \text{GELU}(W^{(i)} * f^{(i-1)} + b^{(i)})$$
     • Self-attention: $$H' = \text{LayerNorm}(H + \text{softmax}(QK^T/\sqrt{d_k}) V)$$
3. Fusion Strategy:
   • Early Fusion: Features from both modalities are concatenated (“early fusion”), forming joint sequences on which subsequent temporal modeling is performed:
     • $F = \operatorname{Concat}(V, A)$
4. Sequential Pattern Learning Network:
   • Model: Multi-Stage Temporal Convolutional Network (MTCN) with dilated convolutions.
   • Temporal Modeling: Dilations grow exponentially ($d = 2^l$ at layer $l$) for long-range dependencies, with hierarchical receptive field $r(B, L) = B \cdot 2^L$.
   • Residual Connections:
     • $S_t^{(j, l)} = S'_t^{(j, l)} + V S_t^{(j, l-1)} + e$
   • Attention: After temporal convolutions, a 1D-CNN plus self-attention module refines the anomaly-relevant information.
   • Output: SoftMax classifier computes anomaly class probabilities per time step:
     • $Y'_t = \text{softmax}(U Z_t + c)$

3. Audio and Visual Feature Extraction Techniques

MobileViT for Video: Combines convolutional neural networks’ efficiency with Transformers’ global relational modeling. It processes input tensors ($X \in \mathbb{R}^{H \times W \times C}$), applying convolution for fine-grained spatial features, followed by patch-wise global aggregation via Transformer blocks, facilitating detection in spatially complex or degraded episodes.

Wav2Vec2 for Audio: Applies convolutional and self-attention layers directly to raw waveforms for robust feature adaptation. Temporal feature extraction yields representations effective for non-stationary, real-world noise scenarios—such as sirens, explosions, or crowd disturbances—rendering fine-grained temporal resolutions critical for anomaly detection.

4. Fusion Mechanism: Early Cross-modal Integration

Early fusion ensures joint consideration of audio and visual cues at the feature level prior to any temporal modeling, making the network resilient when one input stream is degraded or partially unavailable. This strategy creates a composite representation for subsequent sequence learning, enabling cross-modal anomaly patterns to be exploited. A plausible implication is improved recall under conditions such as occlusion, where audio input compensates for missing visual evidence.

5. Temporal Pattern Learning: Dilated Convolutional Networks

The Multi-Stage Temporal Convolutional Network (MTCN) employs exponentially increasing dilation rates, allowing vast temporal receptive fields without deepening the model parameters, which makes AVAR-Net lightweight and effective for long-duration anomalies. Residual connections and CNN–self-attention modules further enable transmission of salient time-dependent features, strengthening the detection of anomalies spanning variable temporal extents.

6. Benchmark Dataset: VAAR

The VAAR dataset is curated to include 3,000 audio-visual clips split equally among ten anomaly categories (abuse, baby cry, crash, brawling, explosion, intruder, pain, police siren, vandalism, and normal). Approximately 80% of samples derive from authentic surveillance, with the remaining clips sourced from movies and web content, supporting both structured and unconstrained real-world testing. Each clip is temporally segmented (lengths 5s–2min) and labeled by multiple independent annotators, establishing high inter-label agreement and facilitating robust evaluation for binary and multiclass anomaly recognition.

7. Experimental Results and Implications

AVAR-Net, evaluated with Accuracy, Precision, Recall, F1-score, and Average Precision, exhibits the following performance:

• VAAR: 89.29% accuracy (multiclass anomaly detection).
• XD-Violence: 88.56% Average Precision, a 2.8% improvement against prior state-of-the-art.
• Results are supported by ablation studies comparing AVAR-Net with recurrent (GRU, LSTM) and Transformer-based models.

These outcomes demonstrate that AVAR-Net delivers robust, generalizable anomaly recognition under diverse modalities and variable sequence conditions. The lightweight architecture (MobileViT backbone, early fusion, MTCN) confirms suitability for deployment in real-time surveillance, healthcare monitoring, and public safety infrastructure.

8. Applications and Prospective Research

AVAR-Net’s design and benchmarking facilitate deployment in real-world surveillance, transportation hubs, and health monitoring, particularly where device constraints and environmental adversities challenge unimodal detectors. The VAAR dataset invites future research in adaptive fusion, multi-view learning, and on-device edge deployment for large-scale anomaly detection. Extension opportunities include integration of expanded temporal attention mechanisms and testing across additional modalities or larger evaluation suites.

In sum, AVAR-Net advances multimodal anomaly detection through efficient cross-modal representation learning, robust sequential modeling, and by establishing a benchmark dataset tailored for real-world multimodal anomaly tasks, marking a clear step forward as objectively substantiated by empirical results (Ali et al., 15 Oct 2025).