---
title: Open-Vocabulary Keyword Spotting
url: https://www.emergentmind.com/topics/open-vocabulary-keyword-spotting-kws
type: topic
---

# Open-Vocabulary Keyword Spotting

Open-vocabulary keyword spotting (KWS) is the task of detecting user-specified keywords or short phrases in speech when the keyword set is not fixed in advance. In this setting, the system must accept an arbitrary keyword specified at runtime, ideally from text rather than requiring a recorded audio enrollment example, and must discriminate that keyword from continuous background speech, often under streaming and on-device constraints. Across recent work, OV-KWS has been formulated as text-enrollment audio-text matching, audio-enrollment query-by-example, multimodal flexible enrollment, CTC-based streaming search, and keyword-conditioned detection, with the common objective of supporting arbitrary keywords without retraining a fixed-class classifier [2312.09760][2206.15400][2406.07923].

## 1. Problem formulation and scope

OV-KWS is distinct from closed-vocabulary KWS because the target keyword is not restricted to a predefined inventory seen during training. Several papers describe the task as detecting arbitrary user-defined keywords or phrases from continuous speech, with enrollment provided as text, speech, or both, and without retraining the model for each new keyword [2312.09760][2512.15124][2603.18023]. Text-based enrollment is repeatedly motivated as a practical alternative to spoken enrollment because spoken enrollment is sensitive to speaker variation, recording-session mismatch, and background noise, whereas text enrollment can be performed by typing or displaying the phrase [2206.15400][2404.03914].

The task has been framed at multiple operational levels. Some systems perform utterance-level match or non-match decisions on bounded segments; others produce frame-level or window-level confidence scores over continuous speech streams so that a trigger can fire when the keyword appears anywhere in the stream [2512.15124][2406.07923]. Several papers explicitly connect OV-KWS to personalized assistants, contextual biasing for ASR, and low-latency edge deployment, where a new command or named entity must be introduced without retraining a full speech recognizer [2309.09552][2606.11279].

A further extension is personalized OV-KWS, in which detection is conditioned not only on keyword identity but also on speaker identity. PCOV-KWS formulates this as joint KWS and speaker verification, so that a positive sample must contain both the target keyword and the target speaker [2603.18023]. This suggests that OV-KWS is increasingly treated not merely as lexical retrieval, but as a configurable speech interface primitive that can incorporate enrollment modality, speaker specificity, and downstream ASR biasing.

## 2. Architectural families

Recent OV-KWS systems fall into several recurring architectural families. They differ primarily in how they represent the enrolled keyword, how they align keyword and speech, and whether detection is performed by explicit search, similarity scoring, or keyword-conditioned classification.

| Family | Representative papers | Distinguishing mechanism |
|---|---|---|
| CTC or ASR-derived search | [2002.10851], [2406.07923], [2312.09760] | Phone or token posteriors searched for keyword paths |
| Cross-modal correspondence and shared embeddings | [2206.15400], [2404.03914], [2505.16735], [2601.14012] | Audio and text mapped into a common or aligned space |
| Keyword-conditioned or hypernetwork detectors | [1912.07575], [2309.08561], [2508.04857] | Keyword text generates detector parameters or normalization parameters |
| Multimodal flexible-enrollment systems | [2412.20805], [2512.15124], [2606.10365] | Audio, text, phoneme, or fused enrollment used within one framework |
| OV-KWS for contextual biasing | [2309.09552], [2606.11279] | Detected keywords are passed to an ASR decoder as prompts or hotwords |

In CTC-centered systems, the acoustic model emits frame-level token or phone posteriors, and detection is performed by keyword search over these posteriors. Small-footprint quantized LSTM systems and CTC-aligned audio-text embedding systems are representative: both use CTC structure to support arbitrary keyword pronunciations or on-the-fly text alignment without training a dedicated detector for each keyword [2002.10851][2406.07923]. U2-KWS extends this line by making the first pass a keyword-aware streaming CTC detector and the second pass a keyword-conditioned verifier within one unified two-pass model [2312.09760].

Cross-modal correspondence detectors encode speech and text separately, then use cross-attention or similarity scoring to decide whether they match. CMCD introduced an attention-based correspondence detector with monotonic matching and de-noising losses, while later work argued that directly learned shared spaces suffer from heterogeneous modality representations and audio-text mismatch [2206.15400][2404.03914]. ADML, MATE, and related utterance-level or phoneme-level metric-learning systems keep the text-enrollment setting but refine the embedding objective through adversarial alignment, deep metric learning, or nested sub-embeddings [2505.16735][2601.14012].

A separate line conditions the detector on the keyword more directly. In the predicted-filter model of Bluche and colleagues, the keyword encoder reads a phone sequence and predicts the top keyword-detection filter of a compact neural detector [1912.07575]. AdaKWS instead uses a text encoder to produce Adaptive Instance Normalization parameters that modulate a Whisper-based audio classifier, turning KWS into keyword-conditioned audio classification rather than pure shared-space matching [2309.08561]. HyperSpotter uses a hyper-network that takes the desired keyword as a character string and generates matched-filter convolution weights for a Perceiver-based detector [2508.04857].

Multimodal flexible-enrollment systems combine text, enrollment audio, phoneme sequences, or mixed embeddings. PLCL jointly optimizes audio-text and audio-audio matching at the phoneme level, while Synaspot caches audio, text, and mixed enrollment embeddings and performs streaming similarity-based decoding with only the audio encoder online [2412.20805][2512.15124]. KFC-KWS similarly combines enrollment audio, phoneme, and text features, but emphasizes CTC-guided keyframe selection for confusable-word disambiguation [2606.10365].

## 3. Alignment, supervision, and scoring

A central technical issue in OV-KWS is how to align heterogeneous audio and keyword representations. Cross-attention is a pervasive mechanism. Multiple papers use the formulation
\[
\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,
\]
with text representations often used as queries and audio representations as keys and values, so that temporal correlation or compatibility between speech and text is explicitly modeled [2312.09760][2404.03914][2206.15400].

Training objectives vary with the alignment granularity. Binary classification over audio-text pairs is common in CMCD and the TTS-transfer system, both of which classify whether an audio segment and a text keyword correspond [2206.15400][2404.03914]. CTC-based detectors use CTC loss on keyword transcripts or sampled keyword subsequences, sometimes combined with a second-stage attention loss, as in
\[
L=\lambda L_{ctc}+(1-\lambda)L_{att},
\]
for U2-KWS [2312.09760]. Metric-learning systems add utterance-level or phoneme-level deep metric learning losses, keyword classification losses, or monotonic matching losses to structure the shared space more explicitly [2505.16735][2601.14012].

Phoneme-level supervision has become especially prominent in work targeting false alarms from confusable words. PLCL uses forced alignment to average query audio within phoneme spans, then applies InfoNCE-style contrastive learning to aligned audio-text and audio-audio phoneme pairs. It also introduces a context-agnostic phoneme memory bank and a third-category discriminator for hard confusable negatives [2412.20805]. KFC-KWS uses CTC posterior peaks to select phoneme-salient keyframes and fuses them with full-utterance representations through cross-attention [2606.10365]. ADML also emphasizes phoneme-level audio-text alignment, but adds Modality Adversarial Learning to reduce the domain gap between audio and text embeddings [2505.16735].

Several papers target specific mismatch mechanisms. The TTS-transfer framework argues that a pre-trained Tacotron 2 text encoder produces audio-aware text representations that reduce audio-text mismatch, with the Bi-LSTM block output E3 giving the best results across datasets [2404.03914]. Synaspot explicitly reduces speaker-specific voiceprint information in audio enrollment through gradient reversal and an adversarial speaker classifier, so that enrollment audio represents keyword identity rather than speaker identity [2512.15124]. MATE addresses a different issue: fixed-dimensional utterance embeddings. It organizes a single 256-dimensional embedding into nested prefixes and aligns them to PCA-compressed teacher targets, so that lower-dimensional prefixes concentrate salient keyword cues while higher dimensions add detail [2601.14012].

This variety of objectives reflects a stable theme: accurate OV-KWS depends less on generic ASR competence than on explicit supervision for keyword discrimination, cross-modal agreement, and hard-negative separation.

## 4. Streaming and small-footprint implementations

Streaming operation and low memory use are recurring design constraints. Early small-footprint systems used quantized unidirectional LSTM acoustic models trained with CTC and a trie-based decoder over keyword pronunciations. The resulting model weighed less than 500KB and supported arbitrary user-defined keywords without keyword-specific training data [2002.10851]. A related predicted-filter architecture reported a quantized keyword detector under 250KB, where the top convolutional filter is generated from the keyword phone sequence [1912.07575].

Later work preserved this deployment emphasis while replacing explicit phone search with richer alignment or verification. CTCAT performs streaming OV-KWS with text-based enrollment by updating the best forced-aligned keyword path frame by frame, aggregating frame-level acoustic embeddings into token-, word-, or phrase-level representations, and combining embedding similarity with CTC score. Its decoding complexity is \(O(U)\), where \(U\) is the length of the target keyword at inference time, and the main setup uses only 155K model parameters [2406.07923].

U2-KWS adopts a unified two-pass design. The first-pass CTC branch is always on and scans audio chunk by chunk for candidate keyword regions; the second-pass attention decoder is activated only when the first stage signals a likely keyword. The architecture uses a shared Conformer encoder trained with dynamic chunks, a keyword encoder, bias modules in both branches, and spike-based clipping of candidate segments. The paper reports 4.75M total parameters, but only 0.04M extra always-on parameters relative to the baseline because the keyword encoder is only used at initialization and the decoder is only triggered on candidates [2312.09760].

Synaspot pushes runtime simplification further by making online inference depend only on the audio encoder. Audio, text, and mixed enrollment embeddings are computed once and cached, while detection uses causal framewise similarity, smoothing, and score fusion over the cached representations [2512.15124]. Massive OV-KWS transfers this deployment logic to contextual biasing for ASR at much larger glossary scales by compressing Whisper encoder embeddings along layer, hidden-dimension, and frame-rate axes. The reported system stores features with a memory footprint up to 128 times smaller than a comparable baseline, is about 6× faster, and can store about 894,784 terms on a 48 GB L40 GPU [2606.11279].

These results indicate that “open vocabulary” in current KWS work is not restricted to lexical flexibility. It is increasingly tied to practical streaming decoders, cached enrollment features, conditional second-pass verification, and glossary-scale memory compression.

## 5. Evaluation regimes, benchmarks, and recurrent failure modes

OV-KWS evaluation is unusually sensitive to negative-example design. LibriPhrase and its easy and hard splits, \(LP_E\) and \(LP_H\), are widely used because the hard split is built from Levenshtein-based confusable negatives [2206.15400][2404.03914][2406.07923]. Other recurrent datasets include Google Speech Commands, Qualcomm Keyword Speech, AISHELL-1, WSJ, VoxPopuli, Multilingual LibriSpeech, WenetPhrase, and internal corpora for vehicles, technical talks, or clinical speech [2312.09760][2505.16735][2606.11279].

Metrics also vary with application. Detection papers on LibriPhrase and related datasets usually report EER and AUC; some also report F1, FRR@FAR5%, or ROC curves [2206.15400][2404.03914][2508.04857]. Streaming wake-word style systems report wake-up rate at a fixed false alarm rate, such as 0.5 false alarms per hour in U2-KWS [2312.09760]. Contextual-biasing systems evaluate downstream ASR with mixed error rate and entity recall in addition to KWS F1 [2309.09552][2606.11279].

A recurrent failure mode is confusable-word or partial-overlap error. TTS-transfer work highlights closely related pronunciations such as “madame” versus “modem” [2404.03914]. PLCL, KFC-KWS, and ADML all target phonetically confusable negatives explicitly through phoneme-level alignment, memory banks, or keyframe selection [2412.20805][2606.10365][2505.16735]. The most explicit treatment is prefix bias. “No Word Left Behind” argues that many strong OV-KWS systems overweight the beginning of the enrolled phrase, so phrases such as “turn the volume up” and “turn the volume down” are poorly disambiguated. It introduces the Partial Overlap Benchmark with POB-Spark and POB-LibriPhrase to expose this weakness [2602.08930].

The proposed Equal-weighting Position Scoring replaces position-specific final-layer weights with a shared linear map and average pooling,
\[
z_i=w^\top X_i,\qquad z=\tfrac{1}{m}\sum_{i=1}^m z_i+b,
\]
so that all positions contribute equally [2602.08930]. Using EPS alone reduces EER on POB-Spark from 64.4% to 29.3% and improves POB-LP accuracy from 87.6% to 96.8%, while maintaining performance on LibriPhrase and Google Speech Commands [2602.08930]. This suggests that benchmark choice can conceal serious failure modes when negatives diverge only late in the phrase.

## 6. Reported advances and current directions

Representative results show steady gains along several orthogonal axes. In streaming two-pass detection, U2-KWS reports a significant relative wake-up rate improvement of 41% compared to traditional customized KWS systems at a fixed false alarm rate of 0.5 times per hour [2312.09760]. In text-enrollment audio-text matching, Tacotron-2 transfer learning improves over CMCD by 8.22% in AUC and 12.56% in EER on LibriPhrase Hard, where pronunciation confusability is most severe [2404.03914]. In streaming joint alignment, phrase-level CTCAT reaches LP\(_E\) EER 6.06 and LP\(_H\) EER 29.63 with 155K parameters [2406.07923].

On LibriPhrase hard negatives, more recent phoneme-aware and multimodal systems report markedly lower error rates. PLCL achieves \(LP_H\) AUC 96.59% and EER 8.47%, with state-of-the-art performance attributed to phoneme-level contrastive alignment and hard-negative modeling [2412.20805]. KFC-KWS, with augmentation, reports \(LP_H\) AUC 97.65% and EER 7.75%, as well as balanced AUC 98.73%, by combining CTC-guided keyframe selection with multimodal fusion [2606.10365]. MATE reports LP\(_H\) EER 20.06 and AUC 88.70 without inference overhead, improving utterance-level matching through nested sub-embeddings [2601.14012]. ADML + SF2 reports LP\(_H\) EER 20.09 and AUC 88.71, showing that modality-adversarial training also materially improves text-enrollment OV-KWS [2505.16735]. These results are not directly interchangeable because architectures, supervision, and baselines differ, but they consistently indicate that hard-negative handling and alignment design dominate performance on realistic confusable cases.

Multilingual and out-of-domain generalization remain active directions. AdaKWS reports strong multilingual performance, including 94.6 overall F1 on VoxPopuli with AdaKWS-Small and 71.6 overall F1 on low-resource FLEURS languages with AdaKWS-Base [2309.08561]. KeySEM, although pretrained only on English, improves few-shot KWS on Japanese, Esperanto, Polish, and Portuguese and supports sequential keyword addition without revisiting older positive examples [2106.02443]. HyperSpotter-c reaches AUC 98.90, F1 94.88, and EER 4.918 on VoxPopuli with 5.5M parameters, while the smallest HyperSpotter-c model has 4.2M parameters and is presented as matching or outperforming several larger models [2508.04857].

Another direction is integration with ASR. CB-Whisper inserts OV-KWS between Whisper’s encoder and decoder, performs keyword spotting on encoder hidden states with a lightweight CNN, and uses detected entities as prompts for contextual biasing [2309.09552]. Massive OV-KWS extends the same application to glossaries of 16,062 terms and beyond, showing that compressed acoustic OV-KWS can remain practical even when contextual biasing would otherwise be memory-bound [2606.11279].

The present literature also identifies unresolved trade-offs. POB training improves partial-overlap robustness but can degrade performance on Google Speech Commands, especially because that benchmark contains only one-word commands [2602.08930]. U2-KWS notes that full-decoder mode improves accuracy but adds latency and compute [2312.09760]. Synaspot’s scoring remains heuristic and similarity-based rather than a learned streaming decoder [2512.15124]. Massive OV-KWS still scales linearly with the number of keywords, even after large compression gains [2606.11279]. A plausible implication is that future OV-KWS work will increasingly focus on benchmark design, confusable-word curricula, glossary-size scalability, and architectures that remain robust across both short commands and long partially overlapping phrases.

Source: https://www.emergentmind.com/topics/open-vocabulary-keyword-spotting-kws