VoxEffects: A Speech-Oriented Audio Effects Dataset and Benchmark

Abstract: Speech audio in the wild is often processed by post-production effects, but existing speech datasets rarely provide precise annotations of effects and parameters, limiting systematic study. We introduce VoxEffects, a speech audio effects dataset that pairs produced speech with exact effect-chain supervision at multiple granularities. VoxEffects supports speech-oriented audio effect identification: given a produced waveform, infer which effects are present and how they are applied. Built from minimally edited clean speech, it provides an extensible rendering pipeline for both offline synthesis and on-the-fly rendering for efficient training and evaluation. The audio effect identification benchmark includes effect presence detection, preset classification, and intensity prediction, with a robustness protocol covering capture-side and platform-side degradations. We provide an AudioMAE-based multi-task baseline and analyses of domain shift, robustness, input duration, and gender fairness.

• The paper introduces VoxEffects, a dataset with precise effect-level supervision enabling identification of applied speech post-production effects.
• It employs a reproducible synthesis pipeline with six canonical effects and expert-curated presets to generate 2520 effect combinations under various degradation configurations.
• The evaluation shows robust Transformer-based accuracy in-domain while exposing challenges in out-of-domain preset classification and effect ambiguity.

VoxEffects: Construction and Evaluation of a Speech-Oriented Audio Effects Dataset and Benchmark

Introduction

This paper introduces "VoxEffects: A Speech-Oriented Audio Effects Dataset and Benchmark" (2604.12389), contextualizing the systematic study of post-production audio effects in speech pipelines. The primary objective is to address the absence of datasets that provide precise effect-level and parameter-level supervision for processed speech, a gap which limits robust development of production-aware audio understanding, engineering assistance, and forensics. VoxEffects supplies exact multi-granularity supervisory information tied to a reproducible chain of effects, supporting the novel task of speech audio effect identification (AEI)—inferring which effects and settings have been applied to a speech waveform. The contribution also includes an extensible synthesis pipeline, a multi-task AEI benchmark, a rigorous robustness evaluation, and a comprehensive analysis with a strong Transformer-based baseline.

Figure 1: Overview of the VoxEffects framework highlighting dataset generation from clean speech through a speech post-processing chain, and benchmark evaluation with multi-granularity effect prediction.

Dataset Design and Synthesis Pipeline

VoxEffects is constructed by processing recordings from carefully selected clean-speech corpora using an ordered chain of six canonical post-production effects: denoising (DN), dynamic range compression (DRC), equalization (EQ), de-essing (DS), reverberation (RVB), and limiting (LIM). Each effect is represented by a discrete, expert-curated preset bank reflecting common speech processing practices, yielding 2520 preset combinations. The effect chain design and preset parameterization are tightly constrained—seeking to balance tractability and practical coverage for speech AEI.

The dataset supports two modes of synthesis: offline generation for fixed dataset creation and on-the-fly rendering for data-efficient training. To model deployment-relevant distortions, a degradation module is included, simulating both capture-side and platform-side artifacts (additive noise, resampling, and lossy codecs), and applied in five experimental configurations. VoxEffects is extensible to additional clean-corpora or effect regimes without interface changes.

Benchmark and Tasks

The AEI benchmark is formulated with multiple granularities:

• Effect presence detection: binary per-effect inference.
• Preset classification: multiclass prediction (2520-way) for effect setting identification.
• Effect count: prediction of the number of active effects.
• Preset intensity regression: scalar and effect-wise strength regression using preset indices.

Each task is evaluated both in-domain (ID) and out-of-domain (OOD), with OOD represented by VCTK—a corpus outside the training distribution but rendered using the same effect pipeline and parameters. Evaluation metrics include per-effect macro-accuracy, exact match ratio (EMR), Top-1/Top-5 preset accuracy, effect count accuracy, and mean absolute error (MAE) for regression tasks.

AudioMAE-Fx: Baseline Architecture

The baseline, AudioMAE-Fx, is a Transformer-based model adapting AudioMAE for speech AEI. Input waveforms are converted to log-mel features and fed to the pretrained backbone. Task-specific lightweight heads (classification/regression) are attached, and simultaneous optimization is performed via a weighted combination of losses for each prediction task. Two-stage robustness-oriented training is employed: (i) fine-tuning on clean rendered data, followed by (ii) curriculum-style augmentation with strong capture and platform degradations.

Experimental Results

The experimental analysis confirms several key findings:

• Robustness to domain shift and distribution shifts is critical. Presence detection achieves up to 95.6% ID accuracy and 86.2% OOD with robustness-augmented training, compared to substantial drops for models trained only on clean data. Top-1 preset classification remains challenging (36.8%/12.2% ID/OOD), highlighting the difficulty of distinguishing perceptually overlapping configurations. EMR is similarly impacted, showing the limits of exact chain inference.
• Robustness-oriented fine-tuning mitigates accuracy loss under all tested degradation configurations, cementing the need for task-aligned augmentation protocols.
• Scalar/vector intensity regression achieves low MAE (down to 0.10/0.17 ID/OOD), with residual errors primarily attributable to effect ambiguity under domain shift and distributional changes.

Effect-wise and Contextual Analyses

Effect-wise performance is heterogeneous:

• Most effects (DRC, RVB, DS, EQ) exhibit only minor ID→OOD accuracy drops in presence detection.
• DN performance degrades significantly OOD, attributable to corpus-dependent spectral and noise-floor signatures exploited by denoising presets.
• Preset and intensity prediction for LIM and DRC degrade further OOD, reflecting high sensitivity to recording-level and peak statistics.

  Figure 2: Per-effect radar plots showing detailed metrics for in-domain and out-of-domain test conditions.

A temporal analysis reveals that accurate effect identification generally necessitates longer waveforms; presence accuracy and EMR increase markedly up to 5s, with diminishing returns beyond ~3s. Degradations lower performance and disrupt monotonic temporal trends, suggesting model decision boundaries are somewhat confounded by artifact cues under high distortion.

Figure 3: OOD presence detection accuracy as a function of input duration, showing strong dependence on context length and robustness to artifact.

Gender Fairness Evaluation

A balanced evaluation on male/female subsets demonstrates tightly matched performance under all settings. Performance drop is primarily artifact-induced, not gender-linked. These findings suggest current model inequities are not strongly gender-sensitive, though further content-controlled audits are proposed.

Figure 4: Gender fairness analysis demonstrating minimal accuracy gap between female and male speech across conditions.

Limitations

The main limitations stem from the assumption of a fixed effect chain, use of a single effect implementation stack, and a finite discrete preset space. These design choices exclude alternative chain orderings, repeated stages, and non-canonical parameterizations common in broad production practice. Additionally, subtle effect cues (especially for mild denoising/limiting) limit observable alignment between label and acoustic artifact, particularly under significant domain shift. More diverse renderers and expansion toward time-varying processing are natural extensions.

Conclusion

VoxEffects provides both infrastructure and a challenging benchmark for the systematic study of AEI on speech, capturing a key facet of realistic audio pipeline variation that is not addressed in prior datasets. The results demonstrate that robust effect presence and configuration recognition is feasible but fundamentally impacted by corpus/domain and artifact shift, underscoring the need for targeted domain balancing, augmentation, and effect-specific modeling strategies.

The implications for AI research are immediate: robust post-production effect identification increases the reliability of downstream automated speech analysis and enables more production-aware content understanding, engineering, and forensics workflows. Future directions include higher-fidelity effect modeling (continuous parameterization, alternative chainings), diverse renderer integration, and transfer learning to in-the-wild media datasets with unconstrained processing, fostering greater alignment with real-world speech signal processing challenges.