Transformer Based Machine Fault Detection From Audio Input

Abstract: In recent years, Sound AI is being increasingly used to predict machine failures. By attaching a microphone to the machine of interest, one can get real time data on machine behavior from the field. Traditionally, Convolutional Neural Net (CNN) architectures have been used to analyze spectrogram images generated from the sounds captured and predict if the machine is functioning as expected. CNN architectures seem to work well empirically even though they have biases like locality and parameter-sharing which may not be completely relevant for spectrogram analysis. With the successful application of transformer-based models in the field of image processing starting with Vision Transformer (ViT) in 2020, there has been significant interest in leveraging these in the field of Sound AI. Since transformer-based architectures have significantly lower inductive biases, they are expected to perform better than CNNs at spectrogram analysis given enough data. This paper demonstrates the effectiveness of transformer-driven architectures in analyzing Sound data and compares the embeddings they generate with CNNs on the specific task of machine fault detection.

• The paper demonstrates transformer-based models achieving superior AUCs in both supervised and unsupervised fault detection compared to CNNs.
• The study employs log-Mel spectrogram conversion with pre-trained and fine-tuned variants, integrating a Local Outlier Factor for anomaly detection.
• The research indicates that transformer embeddings, despite lacking inherent spatial biases, enable robust feature extraction for predictive maintenance.

Transformer-Based Machine Fault Detection from Audio: Comparative Efficacy and Embedding Analysis

Introduction

The analysis of industrial machine sounds for fault detection has become an important facet of predictive maintenance in the Industry 4.0 landscape. This study systematically compares CNN-based models and transformer-based architectures—specifically, the Audio Spectrogram Transformer (AST)—in their ability to detect faults from real-world machine audio. Leveraging the MIMII dataset, which encompasses both normal and anomalous sounds from diverse machine types, the work addresses both supervised and unsupervised anomaly detection, critically examining the discriminative power and inductive biases of different model classes.

Methodological Framework

The experimental protocol is anchored on three models: a baseline autoencoder, a ResNeXt101_32x8d CNN, and the AST transformer. Audio processing involves conversion to log-Mel spectrograms, simulating human auditory sensitivity and facilitating compatibility with image-based neural architectures. For all models, both pre-trained and fine-tuned variants are assessed to isolate the effect of domain adaptation and representation learning.

• In unsupervised scenarios, embeddings from either CNN or AST are generated and subsequently input to a Local Outlier Factor (LOF) anomaly detector, circumventing the scarcity of labeled anomalous samples in industrial datasets.
• For supervised approaches, a classification head is appended to both CNN and AST, and layered fine-tuning with progressive learning rates is used to maintain parameter efficiency.
• Evaluation across models is performed using AUC to address imbalance in the datasets, especially given the relative paucity of faulty data.

Key Results and Quantitative Findings

The supervised AST model consistently achieves superior AUCs, e.g., 0.9999 and 1.0000 on "slide" and "valve" tasks, subtly outperforming the CNN counterpart which nonetheless approaches near-optimal performance (e.g., 0.9990 on "fan"), with both models dramatically surpassing the autoencoder baseline. In the unsupervised context, the AST's embeddings generally yield higher AUCs than those from the CNN (e.g., 0.8502 vs. 0.8201 for "pump"), except in the case of "fan," where the CNN marginally prevails. This divergence is rationalized in the paper by the lower visual discriminability between normal and anomalous "fan" sounds in their spectrogram representations, which appears to favor the feature locality biases in CNNs under specific acoustic contexts.

The study demonstrates that even with minimal anomalous training data, AST-based architectures rapidly converge to high-performance fault detection with minimal epochs, attributed to their capacity to exploit pretraining and self-attention's global receptive field.

Inductive Biases and Embedding Discriminability

Transformers, by design, lack the spatial inductive biases of CNNs—specifically, parameter-sharing and locality—as discussed extensively in the work. In spectrogram analysis, these biases can be detrimental: frequency-time patterns do not obey the translation invariance that benefits conventional image CNNs. AST's multi-head self-attention mechanism effectively models long-range dependencies and captures global context from raw spectrograms, while CNNs’ early layers have a strictly local view, as evidenced in the analysis of Grad-CAMs and attention maps.

The t-SNE visualization of learned embeddings confirms the superior class separability achieved by ASTs, especially for machine types with sharp acoustic transitions. Attention analysis reveals that even in early layers, certain heads in AST exhibit global context awareness—an attribute unattainable by CNNs within their architectural constraints.

Practical and Theoretical Implications

Practically, transformer-based audio analysis pipelines offer compelling advantages for predictive maintenance, enabling contactless, cheap, and non-intrusive fault monitoring using ubiquitous microphones. Their rapid adaptation and superior anomaly detection capabilities are especially significant given the challenges in acquiring large, balanced labeled datasets from industrial environments.

Theoretically, the work substantiates the hypothesis that lower architectural inductive bias and global receptive field facilitate improved feature extraction and anomaly detection in temporally and spectrally complex domains such as raw machine audio. The study also advances understanding of parameter-efficient training regimens for large models, making such architectures accessible without specialized compute resources.

Furthermore, empirical evidence from this paper solidifies the two-stage hybrid approach—deep model embeddings coupled with traditional anomaly scoring—as state-of-the-art for sound anomaly detection in resource-constrained and label-scarce industrial contexts.

Future Prospects

• Efficient Embedding Fusion: The results hint at the potential for diverse model ensembles which leverage heterogeneity in CNN and transformer embeddings, hypothesizing that diversity in feature extraction pipeline yields greater anomaly discriminability.
• Domain Adaptation and Synthetic Data: Models remain susceptible to domain shifts; active research in domain adaptation and the use of generative models to synthesize rare anomalous events is necessary for robust real-world deployment.
• Expansion Beyond Industry: The architectures, analyses, and methodologies from this study have direct implications for other acoustic anomaly domains, including health diagnostics (e.g., respiratory sound detection), environmental monitoring, and broader IoT sensory applications.
• Foundation Models for Audio: Removing residual biases inherited from image-pretrained transformers by training large-scale models on audio data directly represents a key research trajectory for truly universal acoustic foundation models.

Conclusion

This study provides a rigorous, comparative evaluation of transformer and CNN models for the detection of machine faults from audio, demonstrating the clear advantages of transformer-based architectures in both supervised and unsupervised scenarios. The removal of image-inspired inductive biases, combined with attention-based modeling of global dependencies, results in superior feature embeddings for anomaly detection, as supported by strong quantitative evidence and detailed embedding analyses. The path forward includes exploration of heterogeneous ensemble methods, more robust domain adaptation, and end-to-end audio-specific foundation models. This work establishes a robust reference point for subsequent research and field deployment of intelligent predictive maintenance systems using Sound AI.

Reference: "Transformer Based Machine Fault Detection From Audio Input" (2604.12733)