Quantify factors governing corrective-estimator effectiveness

Quantify how ASR model performance, the randomness of differences between ASR and human transcriptions, and the number of human transcriptions required interact to determine the effectiveness of corrective estimators for large-scale corpus keyword counting.

Background

The paper applies Prediction-Powered Inference and related corrective-estimator frameworks to estimate keyword frequencies from large police-radio audio corpora when ASR transcripts are systematically unreliable. These estimators combine numerous inexpensive machine-generated transcripts with a smaller set of human transcriptions to correct statistical bias and produce confidence intervals.

The effectiveness of this strategy may depend jointly on the ASR model's quality, the variability or randomness of ASR–human transcription differences, and the amount of human-labeled data available. The paper identifies the need to characterize these interactions so that researchers can determine how many human transcriptions are needed and when corrective estimators will be practically reliable across datasets.

References

For any given dataset, there are likely relationships between model performance, randomness of ASR-human differences, and the number of transcriptions needed for corrective estimators to be effective. Future work to quantify how these features interact could greatly improve the practicality of conducting large scale corpus keyword counting.

— Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data  (2610.26614 - Molloy et al., 19 Oct 2026) in Section Discussion