SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrails

Abstract: Scripted vs spontaneous speech detection is appealing for interview guardrails, but benchmark performance can be inflated by shortcuts tied to corpus identity, channel conditions, and recording artifacts rather than speaking style itself. We present SEAM, a shortcut-aware framework for real-time scriptedness detection that combines uniform preprocessing, seam-aware sampling, non-speech augmentation, and a compact DistilHuBERT backbone. With 8s windows, the model achieves 0.971 +- 0.004 ROC-AUC on an external interview-domain evaluation set. Removing the shortcut-prevention components improves internal held-out metrics but sharply reduces external performance, indicating shortcut learning. Post-training quantization reduces the model footprint to 41.8MB with little loss in external performance. The results demonstrate that robust real-time scriptedness detection depends not only on the backbone, but on shortcut-aware data design and evaluation. We release code and model checkpoints.

• The paper demonstrates that robust, shortcut-aware sampling and noise augmentation enhance real-world transfer accuracy for distinguishing speech styles.
• SEAM integrates uniform audio preprocessing, seam-aware sampling, and comprehensive provenance tracking to prevent spurious correlations.
• The framework, built on a shallowly adapted DistilHuBERT, achieves high accuracy while remaining compact and real-time deployable.

SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrails

Introduction and Motivation

The distinction between scripted and spontaneous speech plays a critical role in audio-based AI systems, specifically in adversarial and high-integrity contexts such as AI-assisted interviews. Contemporary speech classification systems—powered by large self-supervised models—often inadvertently learn spurious correlations due to dataset-specific artifacts including recording conditions, channel properties, and corpus-specific content. This issue, known as shortcut learning, can severely undermine reliability and generalization to new domains.

The paper introduces SEAM, a real-time, shortcut-aware framework for scriptedness detection tailored for interview guardrail applications (2606.06837). SEAM emphasizes the disentanglement of speaking-style cues from nuisance variables through coordinated enhancements in preprocessing, sampling, augmentation, model architecture, and evaluation. The core claim is that, for robust transfer, shortcut prevention in data pipelines is equally as important as model backbone selection.

Shortcut-Aware Framework: Design and Implementation

SEAM targets three principal shortcut failure modes: (i) channel-based shortcuts (e.g., studio vs. noisy room audio), (ii) provenance-based cues (e.g., windowing across corpus boundaries), and (iii) evaluation mismatches (e.g., high held-out scores that do not transfer). The framework mitigates these via:

1. Uniform Waveform Preprocessing: Standardizes all audio to 16 kHz mono, with DC removal, high-pass filtering, integrated loudness normalization, and peak limiting. This suppresses channel/codec cues while preserving informative prosody.
2. Seam-Aware Sampling: Segments are only sampled within recording boundaries to prevent the model from exploiting artificial transitions as style markers. Mixed or padded windows employ a non-speech noise bank, collected with Silero VAD and specifically sampled to break the “clean audio” heuristic often tied to scripted speech.
3. Comprehensive Provenance Tracking: Manifest-based tracking ensures rigorous separation of sources and prevents leakage between train, eval, and test splits.

The task is posed as windowed binary classification (scripted vs. spontaneous) on 8s audio segments, with outputs aggregated for longer spans. This not only enhances real-time response but also enables analysis of natural speaking-style transitions.

Figure 1: Window-level scriptedness scores for a natural within-speaker transition over 120 s; the model’s outputs track gradual shifts from scripted to spontaneous speech rather than mere audio artifacts.

Model Architecture and Deployment

SEAM employs a shallow adaptation strategy built atop DistilHuBERT, a compact 23M-parameter self-supervised encoder, selected for its optimal quality-efficiency tradeoff. Only the MLP head and the top transformer layer are unfrozen for supervised training, maintaining deployment efficiency and regularization.

Training uses the AdamW optimizer with differential learning rates for the encoder and classification head, cosine learning-rate decay, and data augmentation via noise-injection. Evaluation protocols report window-level and file-level accuracy and ROC-AUC, both on grouped internal splits and an adversarially constructed, proprietary interview-domain test set.

Post-training quantization (down to INT4) is validated, yielding a highly compact model (<42 MB) with negligible accuracy loss, affirming the system's suitability for low-latency, multi-component pipelines.

Empirical Results

The full SEAM system reaches an average internal test accuracy of 0.962 (ROC-AUC: 0.977) and—crucially—an external interview-domain accuracy of 0.952 (ROC-AUC: 0.971). Ablations highlight the centrality of shortcut-prevention: disabling seam-aware sampling and noise augmentation increases internal metrics (ROC-AUC: 0.956 from 0.929) but degrades external performance (ROC-AUC: 0.732 from 0.899), directly evidencing shortcut reliance.

Window length studies support 8s as optimal for latency–quality tradeoff. Shallow adaptation is validated: unfreezing only the top transformer layer outperforms both head-only and deeper-finetuned variants. Quantized (INT4) models retain interview-domain accuracy (0.953) with a 41.8 MB footprint.

Qualitative audits (see Figure 1) demonstrate that SEAM's output scores reflect meaningful, gradual speaking-style transitions, not transient or edit-based artifacts.

Implications and Future Directions

SEAM illustrates that robust deployment of speaking-style classification requires data-centric shortcut mitigation. The findings call for evaluation regimes that include deliberately adversarial, out-of-corpus test sets, as controlled internal splits may substantially overestimate real-world performance.

Practically, the framework enables narrow, low-latency integrity guardrails in high-stake interview settings. Theoretically, this work foregrounds the continued risk of shortcut learning—even in the SSL era—and sets a new bar for evaluation methodology in style-classification research.

Limitations include residual entanglement of scriptedness with genre or corpus attributes, English-first implementation, and imperfect cross-lingual generalization. Zero-shot transfer remains partial, limiting direct application in multilingual contexts.

Future research should expand to broader linguistic and acoustic domains, advance calibration across unseen conditions, and deepen interpretability of the specific cues driving model predictions.

Conclusion

SEAM demonstrates that shortcut-aware data and evaluation pipelines, in conjunction with lightweight self-supervised architectures, are necessary for real-time, highly transferable scripted-vs-spontaneous speech classification. The approach prioritizes realistic, adversarial robustness over inflated benchmark performance and provides a blueprint for future integrity-focused audio AI systems.