---
title: ChordFormer Architecture for Audio Chord Recognition
url: https://www.emergentmind.com/topics/chordformer-architecture
type: topic
---

# ChordFormer Architecture for Audio Chord Recognition

ChordFormer is a conformer-based deep learning architecture designed for large-vocabulary audio chord recognition, emphasizing structured chord decomposition, hybrid local-global sequence modeling, and mitigation of class imbalance. The model targets transcription of polyphonic music audio into detailed, musically meaningful chord labels, addressing the challenges posed by the long-tail distribution of chord types and the inherent need to capture both fine spectral structure and extended harmonic context [2502.11840].

## 1. Design Objectives and Core Challenges

ChordFormer was developed to transcribe audio into structural chord labels encompassing root+triad, bass, seventh, ninth, eleventh, and thirteenth extensions. The architecture addresses several key challenges:
- **Long-tail chord distribution**: Many rare chord types and extensions are sparsely represented in datasets, exacerbating class imbalance and limiting recognition performance.
- **Contextual modeling**: Accurate chord recognition depends on capturing both fine-grained local spectral features (e.g., chord partials, voicing) and long-range harmonic dependencies (e.g., progressions, modulations).
- **Structured chord representation**: Using a musically meaningful decomposition enhances interpretability and allows for effective parameter sharing and cross-family generalization.
- **Class imbalance**: Handled explicitly via a re-weighted loss, allowing robust learning even for underrepresented chord types.

## 2. Input Pipeline and Feature Extraction

ChordFormer processes audio sampled at 22,050 Hz. The primary feature input is a Constant-Q Transform (CQT) spectrogram spanning C1–C8, with 36 bins per octave, resulting in 252 frequency bins per frame. The spectrogram is converted to a decibel scale (librosa’s `amplitude_to_db`) and normalized to the per-track maximum. The hop length of 512 samples yields a temporal resolution of approximately 23.2 ms per frame. Data augmentation is performed via pitch-shifting each training sample by –5 to +6 semitones, with both spectrograms and chord labels shifted accordingly [2502.11840].

## 3. Structured Chord Output Representation

Each time frame $t$ is annotated with a 6-dimensional vector 
$$
Z^{(t)} = [ z_1^{(t)}, z_2^{(t)}, z_3^{(t)}, z_4^{(t)}, z_5^{(t)}, z_6^{(t)} ]
$$
where each element encodes a musically interpretable component:
- $z_1$: root+triad (13 roots × 7 triads + no-chord “N”)
- $z_2$: bass pitch (12 chromas + “N”)
- $z_3$: seventh extension ($\in \{\mathrm{N}, 7, \flat7, \flat\flat7\}$)
- $z_4$: ninth extension ($\in \{\mathrm{N}, 9, \sharp9, \flat9\}$)
- $z_5$: eleventh extension ($\in \{\mathrm{N}, 11, \sharp11\}$)
- $z_6$: thirteenth extension ($\in \{\mathrm{N}, 13, \flat13\}$)

This structured, one-hot-encoded representation allows the problem of large-vocabulary chord recognition to be decomposed into six smaller multiclass classification tasks, reflecting music theory hierarchies and enabling parameter sharing across related chord types [2502.11840].

## 4. ChordFormer Model Architecture

### 4.1 Conformer Blocks

The core of ChordFormer lies in its stack of Conformer blocks, which hybridize convolutional and attention-based sequence modeling. The initial CQT is linearly projected from 252 to $D=256$ dimensions. The architecture contains $N=4$ stacked Conformer blocks, each comprising:

- **First half-step Feed-Forward (FFN)**: 
  $$
  \tilde Z_i = Z_i + \frac{1}{2} \mathrm{FFN}(Z_i)
  $$
  utilizing pre-layer normalization, Swish activation, dropout, and residual connections.
- **Multi-Head Self-Attention (MHSA)**: Employs relative sinusoidal positional encoding and pre-norm. Each head $j$ computes queries, keys, and values:
  $$
  Q_j=(ZW_Q)_j, \quad K_j=(ZW_K)_j, \quad V_j=(ZW_V)_j
  $$
  $$
  \mathrm{Attention}(Q,K,V) = \mathrm{softmax}(QK^\top/\sqrt{d_K})V
  $$
  The output is concatenated and projected. The block output updates as
  $$
  Z_i^{(a)} = \tilde Z_i + \mathrm{MHSA}(\tilde Z_i)
  $$
- **Convolutional module**: Involves pre-norm, pointwise convolution (followed by GLU gating), depthwise 1D convolution (kernel size $K=31$), batch normalization, Swish activation, and dropout. The module output is 
  $$
  Z_i^{(c)} = Z_i^{(a)} + \mathrm{Conv}(Z_i^{(a)})
  $$
- **Second half-step FFN and LayerNorm**: 
  $$
  Z_i^{(o)} = \mathrm{LayerNorm}(Z_i^{(c)} + \frac{1}{2} \mathrm{FFN}(Z_i^{(c)}))
  $$

Schematic descriptions of these modules correspond to Figure 3 in [2502.11840].

### 4.2 Global Sequence Configuration

The overall architectural configuration is:
- Input: linear 252→256 projection, followed by $N=4$ Conformer blocks
- Embedding: $D=256$
- Attention: $n_h=4$ heads, $d_K=64$ per head
- FFN inner dim: 1024
- Convolutional kernel: 31, expansion factor 2
- Activation: Swish, with Softmax output
- Dropout: 0.1 after each sublayer
- Residual pre-normalization throughout

### 4.3 Output Projection and CRF Decoding

The final network state (shape $T \times 256$) is linearly mapped into six output vectors $S^{(t,j)} \in \mathbb{R}^{M_j}$ for $j=1\dots 6$. Softmax normalization yields per-component probabilities,
$$
\beta_m^{(t,j)} = \frac{ \exp(S_m^{(t,j)}) }{ \sum_{m'} \exp(S_{m'}^{(t,j)}) }
$$

Decoding is performed not by simple per-frame argmax but via a linear-chain conditional random field (CRF) imposing temporal smoothness. The probability of a chord label sequence $Z$ given input $X$ is modeled as:
$$
P(Z|X) \propto \prod_{t=1}^{T} \phi(Z^{(t)}, X) \prod_{t=2}^{T} \psi(Z^{(t-1)}, Z^{(t)})
$$
with emission potential
$$
\phi(Z^{(t)}, X) = \exp\left( \sum_{j,m} I[m = z_j^{(t)}] \log \beta_m^{(t,j)} \right)
$$
and transition potential
$$
\psi(Z^{(t-1)}, Z^{(t)}) = \exp\left( -\gamma \cdot I[Z^{(t-1)} \ne Z^{(t)}] \right)
$$
where $I[\cdot]$ is the indicator function and $\gamma$ controls transition penalties.

## 5. Class Imbalance Mitigation

ChordFormer introduces a weighted cross-entropy objective over all frames $t$ and chord components $j$:
$$
L = -\sum_{t=1}^T \sum_{j=1}^6 \sum_{m=1}^{M_j} w_m^{(j)} I[m = z_j^{(t)}] \log \beta_m^{(t,j)}
$$
Weights $w_m^{(j)}$ are computed as:
$$
w_m^{(j)} = \min\left\{ \left( \frac{ n_m^{(j)} }{ \max_{m'} n_{m'}^{(j)} } \right)^{-\gamma}, \, w_\mathrm{max} \right\}
$$
where $n_m^{(j)}$ is the count of training samples for label $m$ in component $j$; $\gamma \in [0,1]$ controls the balancing tradeoff, and $w_\mathrm{max}$ caps the largest class weight. Empirical tuning (e.g., $\gamma=0.7, w_\mathrm{max}=20$) amplifies gradient signals for rare chords, improving class-level accuracy while controlling overemphasis [2502.11840].

## 6. Training Protocol and Optimization

ChordFormer is optimized using AdamW with an initial learning rate of $1 \times 10^{-3}$, subject to a plateau scheduler (decay by $0.1\times$ after 5 non-improving epochs) and an early stop when the learning rate drops below $1 \times 10^{-6}$. During training, each epoch for a given song randomly extracts a 1000-frame segment ($\approx$23.2s), with batch size 24 (total $\approx$24,000 frames). Regularization includes dropout (rate 0.1), pre-norm residuals, and batch normalization within convolutional modules. Augmentation is performed as described, with pitch shifts of –5 to +6 semitones [2502.11840].

## 7. Empirical Performance and Module Impact

On the Humphrey–Bello 1,217-song corpus (5-fold cross-validation, 60/20/20 split), ChordFormer attained:
- **Frame-wise accuracy:** 78.77% (vs. CNN+BLSTM 76.76%, +2.01 pp)
- **Class-wise accuracy:** 38.84% (vs. CNN+BLSTM 33.15%, +5.69 pp)
- **MIREX score:** 83.62% (vs. CNN+BLSTM 81.52%)
- Breakdown: Root 84.69%, Maj/Min 84.09%, Triads 77.55%, Sevenths 72.28%

Ablation studies revealed:
- **Transformer-only**: improved global modeling, weaker local spectral detail, triad accuracy $\sim$67.8%
- **CNN-only**: robust to local patterns, lacking long-range context, seventh/extension recall $\sim$67.3%
- **CNN+BLSTM**: incremental improvements over either backbone individually but behind Conformer hybrid
- **ChordFormer-R** (with reweighted loss): optimally addresses rare-class prediction, with class-wise accuracy peaking at 44.71% for specific weight settings; MIREX improves an additional 0.8% relative to baseline

Increased reweighting ($\gamma$, $w_\mathrm{max}$) improves recall for rare classes (e.g., diminished, augmented, extended chords) with only modest trade-off in overall frame accuracy. Confusion matrices demonstrate that hybrid modeling reduces misclassification among chord extensions.

## Summary Table: ChordFormer Distinctives

| Component                      | Feature/Role                       | Empirical Impact             |
| :----------------------------- | :--------------------------------- | :--------------------------- |
| Constant-Q spectrogram         | Input representation               | High spectral resolution     |
| Structured 6-part chord output | Semantic decomposition             | Improved interpretability    |
| 4-layer Conformer stack        | Hybrid local/global context        | SOTA accuracy, balanced recall |
| CRF decoder                    | Temporal coherence                 | Smoothed predictions         |
| Reweighted loss                | Class imbalance mitigation         | Raised rare-class recall     |

Contextually, ChordFormer advances the field of large-vocabulary chord recognition by successfully combining conformer-based sequence modeling, structured chord interpretation, adaptive loss weighting, and temporal CRF smoothing, achieving leading results on benchmark datasets and robust performance across all chord types [2502.11840].

Source: https://www.emergentmind.com/topics/chordformer-architecture