DAPT: Domain Adaptive Pre-Training

• Domain Adaptive Pre-Training (DAPT) is an approach that extends pre-training on in-domain data to learn fine-grained, attribute-aware representations for specialized tasks.
• It employs a Vision Transformer backbone with a student-teacher setup using the UFO loss for frame-level reconstruction and global matching.
• The method enhances performance through pseudo-label generation via agglomerative clustering and supervised fine-tuning, setting a new state of the art in industrial sound classification.

Domain-Adaptive Pre-Training (DAPT) is an intermediate representation learning paradigm that continues the pre-training of a large neural backbone—whether for audio, text, or images—on unlabeled, in-domain data chosen to bridge distributional and feature gaps between general-purpose corpora and specialized downstream settings. In the context of anomalous sound detection (ASD), DAPT centers the learning of "attribute-aware" representations for industrial machine sounds, enabling high-fidelity distinguishing of machine attributes in the face of domain shift, incomplete label coverage, and intra-class variability (Fang et al., 16 Sep 2025).

1. Addressing Domain Shift in Audio Representation Learning

Classical pre-training (e.g., using AudioSet, comprising ~2M clips spanning speech, music, and environmental audio) imparts broad, generic feature knowledge but fails to capture the fine-grained subtleties necessary for robust ASD in industrial environments. DAPT remedies this shortfall by further pre-training the model on all available machine-sound datasets from the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge (years 2020–2025), thus exposing the backbone to a spectrum of machine types and operating conditions. This shift ensures learned representations encapsulate "attribute-aware" features and prevents intra-machine attribute collapse, a phenomenon wherein models indistinguishably cluster similar machine sounds regardless of underlying operational or environmental variation (Fang et al., 16 Sep 2025).

2. Neural Architecture and Pre-Training Protocol

The DAPT approach leverages a Vision Transformer (ViT) backbone with patch embedding and transformer blocks. The procedure adopts an EAT-inspired student-teacher setup, where both student and teacher share the ViT architecture. Training proceeds as follows:

• Student input: spectrogram patches with random masking, appended with a global learnable CLS token for utterance-level summarization.
• Teacher input: unmasked spectrograms with exponentially moving average (EMA) parameter updates.

Initialization is conducted with the ViT backbone pre-trained on AudioSet using the standard EAT procedure, then resumed (“domain-adaptive pre-training”) on DCASE machine data.

The pre-training objective is the Utterance-Frame Objective (UFO) loss, which integrates frame-level reconstruction and utterance-level global matching:

$$X_o \in \mathbb{R}^{P \times D} \quad \text{(student output)}, \qquad Y_o \in \mathbb{R}^{P \times D} \quad \text{(teacher output)} \ c \in \mathbb{R}^D \quad \text{(student CLS token)}, \qquad y \in \mathbb{R}^D \quad \text{(teacher global mean)}$$

$$\mathcal{L}_f = \|X_o - Y_o\|_2^2, \qquad \mathcal{L}_u = \|c - y\|_2^2, \qquad \mathcal{L}_{UFO} = \mathcal{L}_f + \mathcal{L}_u$$

Parameters $\theta_s$ (student) minimize $\mathcal{L}_{UFO}$, while $\theta_t$ (teacher) update via EMA: $$\theta_t \leftarrow \alpha \theta_t + (1-\alpha) \theta_s$$.

3. Pseudo-Attribute Label Assignment via Agglomerative Clustering

Industrial datasets frequently lack exhaustive attribute labels, necessitating pseudo-label assignment for full machine coverage. DAPT embeddings for each clip, $E_{DA} = (1/P) \sum_{i=1}^P z_i$, serve as the basis for agglomerative hierarchical clustering (Ward linkage) to generate attribute clusters:

$ESS(C) = \sum_{x\in C} (x - \mu_C)^\top(x - \mu_C)$

At each merging step, the pair of clusters yielding the smallest increase in total $ESS$ is chosen. The final cluster assignments become pseudo-attribute labels, augmenting downstream supervised tasks with synthetic classes.

4. Supervised Model Adaptation for Attribute Classification

Post-DAPT, the encoder $F_{DA}$ is fine-tuned in a supervised regime for machine attribute classification (MAC):

• Input: $10$s log-Mel spectrograms (128 bins, $25$ms window, $10$ms hop).
• Classifier: ArcFace-style head $C_{attr}$, trained on $|A_g| + |A_p|$ classes (real and pseudo attributes).
• Loss: cross-entropy,

$$\mathcal{L}_{ASD} = CE\left(C_{attr}(F_{DA}(X)),l_{attr}\right)$$

with $l_{attr}$ drawn from ground-truth or pseudo-attributes.

• Augmentations: Mixup and SpecAugment regularize training.

Hyperparameters: $20$ epochs, batch size $32$, AdamW optimizer with cosine-annealing learning rate (max LR=$5 \times 10^{-5}$, $120$ warmup steps).

5. Quantitative Impact and Ablation Analysis

Empirical evaluation demonstrates substantial distances between DAPT variants and baselines:

Embedding Quality: t-SNE analysis reveals DAPT yields tightly clustered intra-attribute features and distinct inter-attribute separation, unlike standard fine-tuning.

Pseudo-label Ablation (Table 1):

Method	Harmonic mean (AUC, pAUC)
No pseudo labels	59.22
Pseudo from general PT	58.69
Pseudo from finetuned	59.85
Pseudo from DAPT	60.67
Pseudo from DAPT+data	61.09

Model Adaptation Ablation (Table 2):

Setting	Harmonic mean (AUC, pAUC)
Vanilla fine-tuning	59.22
+pseudo only	61.09
+DAPT-based fine-tuning only	61.28
+DAPT-based fine-tuning+pseudo	62.33

Challenge Performance: DAPT achieves a new state of the art on DCASE2025 Eval, harmonic-mean(AUC,pAUC) = 62.60% with only 87M parameters.

6. Contributions and Practical Significance

DAPT's workflow for ASD demonstrates:

• Effective domain alignment from general audio corpora to industrial machine sounds by self-supervised ViT-based UFO loss.
• Robust pseudo-attribute generation via hierarchical clustering on domain-adapted features.
• Enhanced downstream classification via supervised adaptation with ArcFace heads and sophisticated input augmentations.

The pipeline meaningfully surpasses previous top-ranking systems in both benchmark accuracy and representation clustering, especially under domain shift and label scarcity. The modularity of the DAPT/fine-tuning strategy enables consistent gains without increasing model complexity or parameter count.

7. Future Directions and Extensions

While current DAPT approaches for ASD yield reliable state-of-the-art performance, open research areas include:

• Extending pseudo-label granularity to finer operational states or anomaly subtypes.
• Investigating alternate clustering algorithms or density-based assignment for pseudo-attributes.
• Exploring further augmentation techniques or self-training with automatically mined pseudo-labels.
• Domain adaptation to settings with even fewer labels or more heterogeneous machine types.

The DAPT paradigm is generalizable to domains with limited labeled data and pronounced distributional shifts, as evidenced across audio, vision, and textual modalities. Its principled use of self-supervised adaptation, cluster-based pseudo-labeling, and strong downstream classifiers renders it a foundational technique for robust representation learning under domain shift (Fang et al., 16 Sep 2025).