Weighted Chord Symbol Recall (WCSR)

• WCSR is a chord estimation metric that computes per-frame weighted recall based on the pitch-class overlap between predicted and reference chords.
• It addresses shortcomings in traditional ACE metrics by penalizing both omissions and extra notes, thus capturing partial correctness.
• The metric integrates with standard evaluation protocols to provide nuanced diagnostic insights and improved performance analysis.

Weighted Chord-Symbol Recall (WCSR) is a chord estimation metric designed to address limitations inherent in traditional evaluation methods within automatic chord estimation (ACE) systems. Unlike conventional metrics that treat each symbolic chord label as atomic and independent, WCSR explicitly encodes and scores the pitch-class overlap between estimated and reference chords on a per-frame basis. By integrating a set-theoretic comparison of pitch content, WCSR aims to provide a more musically meaningful assessment of estimated chords, supporting nuanced evaluation and deeper diagnostic insight into chord estimation algorithms (Devaney, 2022).

1. Motivation and Problem Statement

Standard ACE evaluation strategies, such as those used in MIREX ACE, treat misclassified chord labels equivalently regardless of their musical proximity. For example, mislabeling an F major chord as G major is penalized identically to mislabeling F major as d minor, even though F and d minor share two of three pitch classes. This atomic treatment ignores the structural relationship between chords, resulting in several deficiencies:

• Over-penalization of close mistakes: Predictions with significant pitch overlap are scored as entirely incorrect.
• Under-penalization of over-estimates: Predictions that include excessive pitch classes are not sufficiently penalized.
• Vulnerability to rigid vocabulary mapping: Forced reductions or coarse mappings are required to align model output vocabularies with evaluation sets.

Traditional metrics thus obscure partial correctness and distort recall/precision, prompting the development of WCSR as a weighting-driven variant that properly respects pitch-class relationships (Devaney, 2022).

2. Mathematical Definition

Let $T$ denote the total number of time frames. For each frame $t \in \{1, \ldots, T\}$:

• $G_t \subseteq \{0,1,\ldots,11\}$: set of ground-truth pitch classes
• $E_t \subseteq \{0,1,\ldots,11\}$: set of estimated pitch classes

Set cardinalities and overlaps are defined as:

• $k_t = |G_t|$ (ground-truth chord size)
• $e_t = |E_t|$ (estimated chord size)
• $c_t = |G_t \cap E_t|$ (number of correctly predicted notes)
• $i_t = |E_t \setminus G_t|$ (number of extra, false-positive notes)

The per-frame weighting function is:

$$w(G_t, E_t) = \frac{c_t - i_t + k_t}{2 k_t} \qquad \text{for}\ k_t > 0$$

Boundary cases:

• If $E_t = G_t$, $t \in \{1, \ldots, T\}$0 (perfect match).
• If $t \in \{1, \ldots, T\}$1 contains no correct notes and $t \in \{1, \ldots, T\}$2, $t \in \{1, \ldots, T\}$3.
• If both $t \in \{1, \ldots, T\}$4 and $t \in \{1, \ldots, T\}$5 are “no-chord,” $t \in \{1, \ldots, T\}$6; if $t \in \{1, \ldots, T\}$7 is “no-chord” but $t \in \{1, \ldots, T\}$8 is not, $t \in \{1, \ldots, T\}$9.

Let $G_t \subseteq \{0,1,\ldots,11\}$0 denote the 0/1 indicator that $G_t \subseteq \{0,1,\ldots,11\}$1 is scored as correct under the chosen chord-matching rule, and $G_t \subseteq \{0,1,\ldots,11\}$2 iff frame $G_t \subseteq \{0,1,\ldots,11\}$3 is counted in the conventional recall denominator ($G_t \subseteq \{0,1,\ldots,11\}$4 not “no-chord”).

The overall WCSR is then defined as:

$G_t \subseteq \{0,1,\ldots,11\}$5

This yields a normalized score in $G_t \subseteq \{0,1,\ldots,11\}$6, assessing the fraction of ground-truth chord pitch content that is correctly estimated after considering inserted notes (Devaney, 2022).

3. Algorithmic Workflow

The WCSR is computed frame-by-frame and can be integrated into standard chord-evaluation pipelines with minimal change. The pseudocode is:

$k_t = |G_t|$8

Here, CountRef(G[t]) returns 1 unless $G_t \subseteq \{0,1,\ldots,11\}$7 is “no-chord.” Practitioners must consistently choose how no-chord frames are counted. The weighting function $G_t \subseteq \{0,1,\ldots,11\}$8 penalizes both note omissions and extra insertions, capturing the specific degree of correctness for each estimation (Devaney, 2022).

4. Integration with Chord Evaluation Protocols

WCSR can be inserted as a drop-in replacement for the numerator in customary frame-level recall metrics. Instead of incrementing a true-positive count for exact chord matches, the weighted sum of $G_t \subseteq \{0,1,\ldots,11\}$9 is accumulated over all evaluated frames, then normalized by the total number of ground-truth chord frames. Precision and $E_t \subseteq \{0,1,\ldots,11\}$0-measure variants are defined analogously by summing over frames with estimates as denominators. There are no free parameters aside from the chosen chord-label matching rule (e.g., triad, root-only, inclusion/exclusion of inversions). No normalization by the maximum possible weight is employed (Devaney, 2022).

5. Illustrative Examples

The utility of WCSR is evident from concrete scenarios. The following table summarizes key examples:

Ground Truth (G)	Estimate (E)	$E_t \subseteq \{0,1,\ldots,11\}$1	$E_t \subseteq \{0,1,\ldots,11\}$2	$E_t \subseteq \{0,1,\ldots,11\}$3	$E_t \subseteq \{0,1,\ldots,11\}$4
$E_t \subseteq \{0,1,\ldots,11\}$5 (F major)	$E_t \subseteq \{0,1,\ldots,11\}$6 (d min)	3	2	1	$E_t \subseteq \{0,1,\ldots,11\}$7
$E_t \subseteq \{0,1,\ldots,11\}$8 (F major)	$E_t \subseteq \{0,1,\ldots,11\}$9 (G maj)	3	0	3	$k_t = |G_t|$0
$k_t = |G_t|$1 (C major)	$k_t = |G_t|$2 (C maj$k_t = |G_t|$3)	3	3	1	$k_t = |G_t|$4

Traditional 0/1 recall would score all non-exact matches as 0, whereas WCSR distinguishes between partial and complete errors. This illustrates the metric’s sensitivity to musically relevant partial correctness and over-completeness in estimations (Devaney, 2022).

6. Experimental Observations and Practical Considerations

No large-scale numerical comparisons are presented, but qualitative examples highlight that WCSR reveals partial correctness and pitch relationships obscured in traditional evaluation. In particular, algorithms that systematically omit certain chord extensions (such as sevenths) but estimate triads accurately experience less severe performance loss in WCSR than in unweighted recall. This suggests that WCSR is effective in exposing systematic error patterns and in guiding system diagnostics.

Best-practice guidelines are:

• Interpretation: WCSR $k_t = |G_t|$5 measures the average fraction of each chord’s pitch content correctly predicted, after penalizing spurious notes. A score of $k_t = |G_t|$6 means, on average, $k_t = |G_t|$7 of each chord’s pitches were estimated correctly.
• Edge case handling: Consistency in inclusion/exclusion of "no-chord" frames is essential.
• No hyperparameters: Only the standard chord-matching policy is required.
• Normalization: Always by the count of ground-truth chord frames.

WCSR thus serves as a nuanced, pitch-aware replacement for traditional chord recall scoring, immediately improving the granularity and musical relevance of ACE system evaluation (Devaney, 2022).