Audio Recurrent Encoder (ARE): Unsupervised Audio Model

• Audio Recurrent Encoder (ARE) is an unsupervised neural sequence model that converts variable-length audio into fixed-length embeddings using recurrent cells like GRU or LSTM.
• It employs an encoder-decoder framework with reconstruction loss to preserve information, yielding superior performance in tasks such as acoustic event classification and audio captioning.
• Variations in preprocessing and architecture—such as convolutional front-ends and bidirectional layers—enhance ARE’s adaptability across environmental, captioning, and animal signal analysis applications.

The Audio Recurrent Encoder (ARE) is a class of unsupervised neural sequence models that extract compact, fixed-length representations from variable-length audio sequences. ARE architectures are grounded in encoder–decoder frameworks where recurrent cells (GRU or LSTM) map framed audio features into vectorial encodings suited for downstream tasks. AREs have demonstrated efficacy in acoustic event classification, audio captioning, and animal signal analysis, often surpassing hand-crafted features in representation quality and classification accuracy (Zhang et al., 2017, Drossos et al., 2017, Kohlsdorf et al., 2020).

1. Fundamental Architecture and Mathematical Formulation

ARE systems universally adopt an encoder–decoder structure engineered for unsupervised learning. The encoder, typically implemented via GRUs or LSTMs, operates on sequences of feature vectors extracted from audio (e.g., MFCCs, log-mel filterbanks, spectrogram frames) to produce a fixed-length embedding. Decoder networks aim to reconstruct the original temporal sequence from this embedding, enforcing an information-preserving bottleneck.

A canonical ARE (Zhang et al., 2017) follows:

• Encoder (variable-length to fixed-length):

$$h_0 = 0 \ h_t = \mathrm{GRUCell}(x_t, h_{t-1}),\quad t=1,...,T \ v = h_T \in \mathbb{R}^n$$

where $x_t$ is a $d$-dimensional feature vector; $v$ is the embedding.

• Decoder (fixed-length to variable-length, reconstruction):

$$\hat{h}_0 = v \ \hat{h}_t = \mathrm{GRUCell}(x_{t-1}, \hat{h}_{t-1}),\quad t=1,...,T \ \hat{x}_t = W_{\mathrm{out}}\hat{h}_t + b_{\mathrm{out}}$$

• Training Objective: Mean squared error (MSE) reconstruction loss:

$$\mathcal{L}_{\mathrm{AE}} = \frac{1}{T} \sum_{t=1}^T \|x_t - \hat{x}_t\|_2^2$$

For spectrogram-based AREs (Kohlsdorf et al., 2020), convolutional and pooling operations precede the recurrent encoder. Input window $x\in\mathbb{R}^{T\times F}$ undergoes 2D convolution and max-pooling, followed by bi-directional LSTM (many-to-many) and eventual compression via many-to-one LSTM:

$e = \text{LSTM}_{T}(h^{{bi}})$

2. Input Representations and Preprocessing

AREs operate on feature representations dependent on audio application:

• Environmental Sound/AEC: 13-dimensional MFCCs per frame (60 ms length/shift, up to 167 frames; ~10 s clips) (Zhang et al., 2017).
• Captioning: 64-dimensional log-mel filterbanks from 2048-sample Hamming windows (≈46 ms, 50% overlap; 1289 frames/30 s) (Drossos et al., 2017).
• Animal Communication: STFT magnitude spectrogram windows, typically $T=128$ frames/0.75 s, $F\approx 220-256$ bins, with z-score normalization applied on each frame (Kohlsdorf et al., 2020).

A plausible implication is that ARE input preprocessing—including frame shifting, spectrogram calculation, and feature normalization—must align with audio characteristics and the intended semantic extraction.

3. Encoder–Decoder Variants: Depth, Width, and Bidirectional Design

Architectural choices in AREs reflect task requirements and data complexity:

• GRU vs. LSTM Cells: GRUs favored for faster convergence/lower parameter count while maintaining long-range dependency modeling (Zhang et al., 2017). LSTMs preferred for bidirectional context in animal signal processing (Kohlsdorf et al., 2020).
• Depth/Width Exploration: AREs are configured as deep (stacked layers; 1–3 GRUs of 512 units each) or wide (single GRU layer of 512/1024/2048 units). Both variants affect representational fidelity and classification F1 (Zhang et al., 2017).
• Bidirectionality: Three-layer bidirectional GRU encoder enables forward and backward temporal context, with residual connections enhancing representational flow (Drossos et al., 2017).
• Convolution-Pooling Front-end: For spectrogram inputs, Conv2D (256 filters) and frequency-domain max-pool provide frequency-shift invariance, followed by recurrent layers encoding temporal order (Kohlsdorf et al., 2020).

The following table organizes major ARE configurations in the literature:

Study	Encoder Type	Depth/Width	Input Feature
(Zhang et al., 2017)	GRU (stacked)	Deep: 1–3×512 / Wide: 1×512–2048	MFCCs
(Drossos et al., 2017)	Bi-GRU (stacked)	3 layers (64, 64, 128)	Log-mel
(Kohlsdorf et al., 2020)	Bi-LSTM/Conv	Conv + Bi-LSTM + LSTM	Spectrogram

4. Training Protocols and Optimization

ARE training is unsupervised, focusing on sequence reconstruction:

• Optimization Algorithms: SGD (initial LR=0.7, decay when loss stagnates) and Adam (lr=0.001); gradient norm clipping commonly applied (Zhang et al., 2017, Kohlsdorf et al., 2020).
• Batching: Mini-batch size varied (64–128 for event/captioning, 50 for animal signals), with zero-padding for unequal lengths; loss computed only over valid frames (Zhang et al., 2017, Drossos et al., 2017, Kohlsdorf et al., 2020).
• Early Stopping: Training halted after a predetermined number of steps with no improvement in validation loss.
• Regularization: Dropout employed (input-dropout 0.5; recurrent 0.25), but no use of weight decay or layer normalization reported (Drossos et al., 2017).
• Feature Standardization: Embeddings standardised (zero mean, unit variance) prior to classifier training (Zhang et al., 2017). Spectrograms z-scored per frame (Kohlsdorf et al., 2020).

5. Downstream Tasks: Classification, Captioning, and Clustering

ARE-generated embeddings serve diverse tasks across audio domains:

• Acoustic Event Classification (AEC): Final encoder hidden state vector $v$ passed to SVM or 1-layer GRU-RNN classifiers. ARE embeddings (e.g., GRU-ED 2048) yielded $F_1$ scores of 85–89% (vs. 50–54% for best hand-crafted features) (Zhang et al., 2017).
• Automated Audio Captioning: Encoder representations aligned via attention; decoder GRUs generate word sequences, optimized via cross-entropy. Performance benchmarks use BLEU, METEOR, ROUGE, and CIDEr-D metrics; ARE achieves BLEU$_1$=0.191, CIDEr-D=0.526 (Drossos et al., 2017).
• Dolphin Signal Analysis: Embedding vectors clustered via k-means (k=100), facilitating signal detection (binary, accuracy=96%) and 4-way classification (accuracy=85%) (Kohlsdorf et al., 2020).

A plausible implication is that AREs, via unsupervised reconstruction, yield representations effective for both supervised and clustering-based downstream analytics.

6. Attention Mechanisms in AREs

Audio captioning AREs augment basic encoder–decoder models with alignment (attention) mechanisms. At each decoder time-step, Bahdanau-style soft attention computes scalar alignment scores over encoder outputs, yielding context vectors for word prediction (Drossos et al., 2017):

$$e_{i,t} = v^\top \tanh(W_h h^3_t + W_s h'_{i-1} + b) \ a_{i,t} = \text{softmax}_t(e_{i,t}) \ c_i = \sum_{t=1}^T a_{i,t} h^3_t$$

Weights are shared across all decoding steps, enabling contextually sensitive audio-to-text mappings.

7. Empirical Results, Ablations, and Comparative Impact

Across tasks, unsupervised AREs consistently outperform hand-crafted features and alternative baselines:

• In environmental sound classification, ARE embeddings increase F1 scores by up to +35pp absolute over ComParE13 hand-crafted features (Zhang et al., 2017).
• Ablations confirm that increased depth and width monotonically improve classifier performance (e.g., SVM F₁: 1/2/3 layers: 58.1/68.4/80.6%; width: 512→1024→2048: 58.1→72.0→85.2%) (Zhang et al., 2017).
• AREs are robust for fine-grained subclasses (e.g., 229 classes, F₁=47.7% vs. 23.1% with handcrafted+GRU) (Zhang et al., 2017).
• In dolphin audio, unsupervised clustering isolates “pure” signal clusters (86% purity post-filtering) (Kohlsdorf et al., 2020).
• For captioning, AREs reliably select event keywords but struggle with sentence structure (BLEU$_4$ lower than BLEU$_1$; METEOR recall remains limited) (Drossos et al., 2017).

The unsupervised, information-preserving compression provided by AREs is a key driver of their superiority for sequence-level audio representation and downstream generalization. AREs leverage recurrent architectures to both absorb long-range dependencies and model variable-length input scenarios, with convolutional and bidirectional augmentations providing additional invariance as necessary.