Chromatic Waveform Plots in Audio Analysis

• Chromatic waveform plots are visual representations that encode both amplitude and phase of audio signals into HSV images for precise, lossless reconstruction.
• They employ a log-scaled, two-piece mapping to convert complex spectrogram data into vivid hue, saturation, and value representations, revealing subtle phase shifts.
• This method enhances pattern recognition in audio analysis with applications in music tuning, transient detection, and species call identification.

Chromatic waveform plots are canonical visual representations that encode both magnitude and phase information from the complex spectrogram of an audio signal into a color image. Utilizing a log-scaled complex-to-HSV (hue–saturation–value) color mapping, these plots provide a simultaneous display of amplitude and phase, enabling exact reconstruction of the original sound—including its phase—from the image. This representation leverages advanced human visual pattern recognition to reveal features typically obscured in amplitude-only plots, such as phase shifts and micro-tuning (Wedekind et al., 2019).

1. Mathematical Mapping: Complex Spectrogram to Chromatic Image

Let $X(t,f)$ denote the complex spectrogram coefficient at time-frame $t$ and frequency-bin $f$ computed via the short-time Fourier transform (STFT). The representation encodes:

• Magnitude: $A(t,f) = |X(t,f)|$
• Phase: $\varphi(t,f) = \arg X(t,f)$, with $-\pi < \varphi \leq \pi$

Mapping to HSV color space proceeds as follows:

Hue Encoding

$H(t,f) = \frac{\varphi(t,f)}{2\pi}$, $H \in [0,1]$

• Full wrapping over the $2\pi$ phase cycle, visually indicating phase evolution as color shifts.

Saturation and Value Encoding

Define $L(A) = \ln A$. The log-scaled amplitude mapping, symmetric about $A=1$, is: $$S = \begin{cases} 1 & A \leq 1 \ \frac{1}{1+\ln A} & A > 1 \end{cases} \qquad V = \begin{cases} \frac{1}{1-\ln A} & A \leq 1 \ 1 & A > 1 \end{cases}$$

• Lower amplitudes ($A \leq 1$): Full saturation, value increases as amplitude decreases.
• Higher amplitudes ($A > 1$): Saturation falls, brightness maximized.

RGB Conversion

HSV values are transformed into RGB color via the standard HSV→RGB mapping.

2. Rationale for Color Space and Visual Encoding

The HSV color space is chosen for its perceptual alignment with audio features:

• Hue ($H$) codes phase cycles, wrapping from red→green→blue for positive phase progression and red→blue→green for negative. This imparts visually obvious cycling stripes and unambiguous encoding of phase shifts.
• Saturation ($S$) and Value ($V$) jointly encode amplitude using a split log-dynamic-range assignment. The symmetric mapping ensures that both weak and strong components are rendered distinguishable, and prevents loss of detail or amplitude clipping at the extremes.
• The vertical axis of the image can be assigned either linear frequency or, optionally, log-frequency scaling (e.g., musical half-notes).

3. End-to-End Algorithmic Workflow

The full transformation from waveform to chromatic plot (and its exact inverse) is as follows, with all steps directly traceable to (Wedekind et al., 2019):

Forward Transformation: Waveform → Chromatic Image

1. STFT Computation: For each audio frame (\textit{windowed by $w[n]$}), compute FFT:
   • $$X[:,t] = \text{FFT}_N\{x[\text{start}+k] \cdot w[k]\}$$
2. Magnitude/Phase Extraction: For $f = 0\dots N/2$,
   • $A(t,f) = |X[f,t]|$
   • $\varphi(t,f) = \arg X[f,t]$
3. Optional Log-Frequency Warping: Interpolate $X$ between bins for perceptually uniform frequency spacing.
4. Color Mapping: For each pixel $(m,t)$,
   • $H = (\varphi)/(2\pi) \bmod 1$
   • $S, V$ assigned per the two-piece log mapping.
   • Convert HSV to RGB.
5. Rendering: Paint each $(t, m)$ pixel using the computed $(R, G, B)$.

Inverse Transformation: Chromatic Image → Waveform

1. HSV Extraction: From $RGB[m, t]$, compute $(H, S, V)$.
2. Amplitude/Phase Recovery:
   • $\varphi = H \cdot 2\pi$
   • If $V < 1$: $A = \exp(1 - 1/V)$
   • If $V = 1$: $A = \exp\left((1/S) - 1\right)$
3. Complex STFT Reconstruction: $X_{\text{rec}}[m, t] = A \cdot \exp(i \varphi)$
4. Inverse Log-Frequency Warp: (If used) Interpolate back to linear frequency bins.
5. Waveform Recovery: Apply inverse STFT (overlap-add synthesis) to recover audio.

4. Representative Use-Cases and Pattern Recognition Utility

Chromatic waveform plots have been utilized in various settings, leveraging phase and magnitude encoding for enhanced pattern detection:

• Frequency Modulation: Beats manifest as alternating hue stripes; cycle rate equals Hertz offset from bin center.
• Harmonic Structure: Vertical stacks of constant-hue bands reflect harmonics; minute detunings prompt visible hue drift.
• Transients: Onsets and rapid temporal changes appear as pronounced color discontinuities.
• Phase Shifts: Stereo panning, room reflections, and phase discontinuities are visually represented by abrupt hue reversals.
• Species Identification: Bird-call spectrogram guides permit direct matching via mobile overlay and cross-correlation.
• Instrument Tuning: Real-time display yields visual feedback on "sharp" vs. "flat" pitches through color beats.
• Reversible Music Notation: Compositions encoded as color images (retaining exact audio including phase) can be losslessly reverted.

5. Performance, Technical Constraints, and Limitations

• Dynamic Range and Contrast: The log-magnitude approach enables preservation of detail for both weak and strong signal components, though mid-level contrast may be compressed.
• Color Quantization: 8-bit per channel precision limits the granularity for $A$ and $\varphi$ encoding; higher-resolution (10–12 bit) displays yield improved accuracy.
• Interpolation Artifacts: Rectangular interpolation in log-frequency warping generates thin black lines at zero amplitude due to cancellation; these are algorithmic not related to actual silence. Polar interpolation may circumvent this but can introduce phase ambiguity.
• Real-Time Execution: Implementation in HTML5/JavaScript on mobile hardware yields approximately three times real-time audio processing speed for 2048-point FFTs at 44.1 kHz. Computational requirements scale with FFT and hop size.
• Human Perceptual Factors: Low saturation ($A \ll 1$) yields subtle hue difference, which may be enhanced algorithmically or via visual processing filters.

6. Canonicality, Losslessness, and Research Significance

These plots produce canonical and invertible images of total sound—with both phase and magnitude preserved. The process is strictly lossless: starting from audio, transforming to the chromatic plot, and inverting yields the original waveform precisely. The approach enables sophisticated visual pattern analysis of complex audio phenomena and mitigates limitations inherent in traditional, amplitude-only spectrograms. It establishes a framework whereby compositional, musical, and acoustic analysis can exploit both amplitude and phase, mapped into a single visual domain (Wedekind et al., 2019).