Open-Vocabulary Keyword Spotting

Updated 8 July 2026

Open-vocabulary keyword spotting is the task of detecting user-specified keywords in continuous speech using text or audio enrollment without retraining fixed classifiers.
It leverages diverse architectures like CTC-based search, cross-modal embeddings, and keyword-conditioned detectors to align audio and text representations effectively.
Recent advances focus on streaming efficiency, low-footprint implementation, and phoneme-level alignment to address confusable negatives and scalability challenges.

Open-vocabulary keyword spotting (KWS) is the task of detecting user-specified keywords or short phrases in speech when the keyword set is not fixed in advance. In this setting, the system must accept an arbitrary keyword specified at runtime, ideally from text rather than requiring a recorded audio enrollment example, and must discriminate that keyword from continuous background speech, often under streaming and on-device constraints. Across recent work, OV-KWS has been formulated as text-enrollment audio-text matching, audio-enrollment query-by-example, multimodal flexible enrollment, CTC-based streaming search, and keyword-conditioned detection, with the common objective of supporting arbitrary keywords without retraining a fixed-class classifier (Zhang et al., 2023, Shin et al., 2022, Jin et al., 2024).

1. Problem formulation and scope

OV-KWS is distinct from closed-vocabulary KWS because the target keyword is not restricted to a predefined inventory seen during training. Several papers describe the task as detecting arbitrary user-defined keywords or phrases from continuous speech, with enrollment provided as text, speech, or both, and without retraining the model for each new keyword (Zhang et al., 2023, Li et al., 17 Dec 2025, Pan et al., 5 Mar 2026). Text-based enrollment is repeatedly motivated as a practical alternative to spoken enrollment because spoken enrollment is sensitive to speaker variation, recording-session mismatch, and background noise, whereas text enrollment can be performed by typing or displaying the phrase (Shin et al., 2022, V et al., 2024).

The task has been framed at multiple operational levels. Some systems perform utterance-level match or non-match decisions on bounded segments; others produce frame-level or window-level confidence scores over continuous speech streams so that a trigger can fire when the keyword appears anywhere in the stream (Li et al., 17 Dec 2025, Jin et al., 2024). Several papers explicitly connect OV-KWS to personalized assistants, contextual biasing for ASR, and low-latency edge deployment, where a new command or named entity must be introduced without retraining a full speech recognizer (Li et al., 2023, Barreiros et al., 9 Jun 2026).

A further extension is personalized OV-KWS, in which detection is conditioned not only on keyword identity but also on speaker identity. PCOV-KWS formulates this as joint KWS and speaker verification, so that a positive sample must contain both the target keyword and the target speaker (Pan et al., 5 Mar 2026). This suggests that OV-KWS is increasingly treated not merely as lexical retrieval, but as a configurable speech interface primitive that can incorporate enrollment modality, speaker specificity, and downstream ASR biasing.

2. Architectural families

Recent OV-KWS systems fall into several recurring architectural families. They differ primarily in how they represent the enrolled keyword, how they align keyword and speech, and whether detection is performed by explicit search, similarity scoring, or keyword-conditioned classification.

Family	Representative papers	Distinguishing mechanism
CTC or ASR-derived search	(Bluche et al., 2020, Jin et al., 2024, Zhang et al., 2023)	Phone or token posteriors searched for keyword paths
Cross-modal correspondence and shared embeddings	(Shin et al., 2022, V et al., 2024, Jung et al., 22 May 2025, Jung et al., 20 Jan 2026)	Audio and text mapped into a common or aligned space
Keyword-conditioned or hypernetwork detectors	(Bluche et al., 2019, Navon et al., 2023, Segal-Feldman et al., 6 Aug 2025)	Keyword text generates detector parameters or normalization parameters
Multimodal flexible-enrollment systems	(Kewei et al., 2024, Li et al., 17 Dec 2025, Li et al., 9 Jun 2026)	Audio, text, phoneme, or fused enrollment used within one framework
OV-KWS for contextual biasing	(Li et al., 2023, Barreiros et al., 9 Jun 2026)	Detected keywords are passed to an ASR decoder as prompts or hotwords

In CTC-centered systems, the acoustic model emits frame-level token or phone posteriors, and detection is performed by keyword search over these posteriors. Small-footprint quantized LSTM systems and CTC-aligned audio-text embedding systems are representative: both use CTC structure to support arbitrary keyword pronunciations or on-the-fly text alignment without training a dedicated detector for each keyword (Bluche et al., 2020, Jin et al., 2024). U2-KWS extends this line by making the first pass a keyword-aware streaming CTC detector and the second pass a keyword-conditioned verifier within one unified two-pass model (Zhang et al., 2023).

Cross-modal correspondence detectors encode speech and text separately, then use cross-attention or similarity scoring to decide whether they match. CMCD introduced an attention-based correspondence detector with monotonic matching and de-noising losses, while later work argued that directly learned shared spaces suffer from heterogeneous modality representations and audio-text mismatch (Shin et al., 2022, V et al., 2024). ADML, MATE, and related utterance-level or phoneme-level metric-learning systems keep the text-enrollment setting but refine the embedding objective through adversarial alignment, deep metric learning, or nested sub-embeddings (Jung et al., 22 May 2025, Jung et al., 20 Jan 2026).

A separate line conditions the detector on the keyword more directly. In the predicted-filter model of Bluche and colleagues, the keyword encoder reads a phone sequence and predicts the top keyword-detection filter of a compact neural detector (Bluche et al., 2019). AdaKWS instead uses a text encoder to produce Adaptive Instance Normalization parameters that modulate a Whisper-based audio classifier, turning KWS into keyword-conditioned audio classification rather than pure shared-space matching (Navon et al., 2023). HyperSpotter uses a hyper-network that takes the desired keyword as a character string and generates matched-filter convolution weights for a Perceiver-based detector (Segal-Feldman et al., 6 Aug 2025).

Multimodal flexible-enrollment systems combine text, enrollment audio, phoneme sequences, or mixed embeddings. PLCL jointly optimizes audio-text and audio-audio matching at the phoneme level, while Synaspot caches audio, text, and mixed enrollment embeddings and performs streaming similarity-based decoding with only the audio encoder online (Kewei et al., 2024, Li et al., 17 Dec 2025). KFC-KWS similarly combines enrollment audio, phoneme, and text features, but emphasizes CTC-guided keyframe selection for confusable-word disambiguation (Li et al., 9 Jun 2026).

3. Alignment, supervision, and scoring

A central technical issue in OV-KWS is how to align heterogeneous audio and keyword representations. Cross-attention is a pervasive mechanism. Multiple papers use the formulation

$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$

with text representations often used as queries and audio representations as keys and values, so that temporal correlation or compatibility between speech and text is explicitly modeled (Zhang et al., 2023, V et al., 2024, Shin et al., 2022).

Training objectives vary with the alignment granularity. Binary classification over audio-text pairs is common in CMCD and the TTS-transfer system, both of which classify whether an audio segment and a text keyword correspond (Shin et al., 2022, V et al., 2024). CTC-based detectors use CTC loss on keyword transcripts or sampled keyword subsequences, sometimes combined with a second-stage attention loss, as in

$L=\lambda L_{ctc}+(1-\lambda)L_{att},$

for U2-KWS (Zhang et al., 2023). Metric-learning systems add utterance-level or phoneme-level deep metric learning losses, keyword classification losses, or monotonic matching losses to structure the shared space more explicitly (Jung et al., 22 May 2025, Jung et al., 20 Jan 2026).

Phoneme-level supervision has become especially prominent in work targeting false alarms from confusable words. PLCL uses forced alignment to average query audio within phoneme spans, then applies InfoNCE-style contrastive learning to aligned audio-text and audio-audio phoneme pairs. It also introduces a context-agnostic phoneme memory bank and a third-category discriminator for hard confusable negatives (Kewei et al., 2024). KFC-KWS uses CTC posterior peaks to select phoneme-salient keyframes and fuses them with full-utterance representations through cross-attention (Li et al., 9 Jun 2026). ADML also emphasizes phoneme-level audio-text alignment, but adds Modality Adversarial Learning to reduce the domain gap between audio and text embeddings (Jung et al., 22 May 2025).

Several papers target specific mismatch mechanisms. The TTS-transfer framework argues that a pre-trained Tacotron 2 text encoder produces audio-aware text representations that reduce audio-text mismatch, with the Bi-LSTM block output E3 giving the best results across datasets (V et al., 2024). Synaspot explicitly reduces speaker-specific voiceprint information in audio enrollment through gradient reversal and an adversarial speaker classifier, so that enrollment audio represents keyword identity rather than speaker identity (Li et al., 17 Dec 2025). MATE addresses a different issue: fixed-dimensional utterance embeddings. It organizes a single 256-dimensional embedding into nested prefixes and aligns them to PCA-compressed teacher targets, so that lower-dimensional prefixes concentrate salient keyword cues while higher dimensions add detail (Jung et al., 20 Jan 2026).

This variety of objectives reflects a stable theme: accurate OV-KWS depends less on generic ASR competence than on explicit supervision for keyword discrimination, cross-modal agreement, and hard-negative separation.

4. Streaming and small-footprint implementations

Streaming operation and low memory use are recurring design constraints. Early small-footprint systems used quantized unidirectional LSTM acoustic models trained with CTC and a trie-based decoder over keyword pronunciations. The resulting model weighed less than 500KB and supported arbitrary user-defined keywords without keyword-specific training data (Bluche et al., 2020). A related predicted-filter architecture reported a quantized keyword detector under 250KB, where the top convolutional filter is generated from the keyword phone sequence (Bluche et al., 2019).

Later work preserved this deployment emphasis while replacing explicit phone search with richer alignment or verification. CTCAT performs streaming OV-KWS with text-based enrollment by updating the best forced-aligned keyword path frame by frame, aggregating frame-level acoustic embeddings into token-, word-, or phrase-level representations, and combining embedding similarity with CTC score. Its decoding complexity is $O(U)$ , where $U$ is the length of the target keyword at inference time, and the main setup uses only 155K model parameters (Jin et al., 2024).

U2-KWS adopts a unified two-pass design. The first-pass CTC branch is always on and scans audio chunk by chunk for candidate keyword regions; the second-pass attention decoder is activated only when the first stage signals a likely keyword. The architecture uses a shared Conformer encoder trained with dynamic chunks, a keyword encoder, bias modules in both branches, and spike-based clipping of candidate segments. The paper reports 4.75M total parameters, but only 0.04M extra always-on parameters relative to the baseline because the keyword encoder is only used at initialization and the decoder is only triggered on candidates (Zhang et al., 2023).

Synaspot pushes runtime simplification further by making online inference depend only on the audio encoder. Audio, text, and mixed enrollment embeddings are computed once and cached, while detection uses causal framewise similarity, smoothing, and score fusion over the cached representations (Li et al., 17 Dec 2025). Massive OV-KWS transfers this deployment logic to contextual biasing for ASR at much larger glossary scales by compressing Whisper encoder embeddings along layer, hidden-dimension, and frame-rate axes. The reported system stores features with a memory footprint up to 128 times smaller than a comparable baseline, is about 6× faster, and can store about 894,784 terms on a 48 GB L40 GPU (Barreiros et al., 9 Jun 2026).

These results indicate that “open vocabulary” in current KWS work is not restricted to lexical flexibility. It is increasingly tied to practical streaming decoders, cached enrollment features, conditional second-pass verification, and glossary-scale memory compression.

5. Evaluation regimes, benchmarks, and recurrent failure modes

OV-KWS evaluation is unusually sensitive to negative-example design. LibriPhrase and its easy and hard splits, $LP_E$ and $LP_H$ , are widely used because the hard split is built from Levenshtein-based confusable negatives (Shin et al., 2022, V et al., 2024, Jin et al., 2024). Other recurrent datasets include Google Speech Commands, Qualcomm Keyword Speech, AISHELL-1, WSJ, VoxPopuli, Multilingual LibriSpeech, WenetPhrase, and internal corpora for vehicles, technical talks, or clinical speech (Zhang et al., 2023, Jung et al., 22 May 2025, Barreiros et al., 9 Jun 2026).

Metrics also vary with application. Detection papers on LibriPhrase and related datasets usually report EER and AUC; some also report F1, FRR@FAR5%, or ROC curves (Shin et al., 2022, V et al., 2024, Segal-Feldman et al., 6 Aug 2025). Streaming wake-word style systems report wake-up rate at a fixed false alarm rate, such as 0.5 false alarms per hour in U2-KWS (Zhang et al., 2023). Contextual-biasing systems evaluate downstream ASR with mixed error rate and entity recall in addition to KWS F1 (Li et al., 2023, Barreiros et al., 9 Jun 2026).

A recurrent failure mode is confusable-word or partial-overlap error. TTS-transfer work highlights closely related pronunciations such as “madame” versus “modem” (V et al., 2024). PLCL, KFC-KWS, and ADML all target phonetically confusable negatives explicitly through phoneme-level alignment, memory banks, or keyframe selection (Kewei et al., 2024, Li et al., 9 Jun 2026, Jung et al., 22 May 2025). The most explicit treatment is prefix bias. “No Word Left Behind” argues that many strong OV-KWS systems overweight the beginning of the enrolled phrase, so phrases such as “turn the volume up” and “turn the volume down” are poorly disambiguated. It introduces the Partial Overlap Benchmark with POB-Spark and POB-LibriPhrase to expose this weakness (Liu et al., 9 Feb 2026).

The proposed Equal-weighting Position Scoring replaces position-specific final-layer weights with a shared linear map and average pooling,

$z_i=w^\top X_i,\qquad z=\tfrac{1}{m}\sum_{i=1}^m z_i+b,$

so that all positions contribute equally (Liu et al., 9 Feb 2026). Using EPS alone reduces EER on POB-Spark from 64.4% to 29.3% and improves POB-LP accuracy from 87.6% to 96.8%, while maintaining performance on LibriPhrase and Google Speech Commands (Liu et al., 9 Feb 2026). This suggests that benchmark choice can conceal serious failure modes when negatives diverge only late in the phrase.

6. Reported advances and current directions

Representative results show steady gains along several orthogonal axes. In streaming two-pass detection, U2-KWS reports a significant relative wake-up rate improvement of 41% compared to traditional customized KWS systems at a fixed false alarm rate of 0.5 times per hour (Zhang et al., 2023). In text-enrollment audio-text matching, Tacotron-2 transfer learning improves over CMCD by 8.22% in AUC and 12.56% in EER on LibriPhrase Hard, where pronunciation confusability is most severe (V et al., 2024). In streaming joint alignment, phrase-level CTCAT reaches LP $_E$ EER 6.06 and LP $_H$ EER 29.63 with 155K parameters (Jin et al., 2024).

On LibriPhrase hard negatives, more recent phoneme-aware and multimodal systems report markedly lower error rates. PLCL achieves $LP_H$ AUC 96.59% and EER 8.47%, with state-of-the-art performance attributed to phoneme-level contrastive alignment and hard-negative modeling (Kewei et al., 2024). KFC-KWS, with augmentation, reports $L=\lambda L_{ctc}+(1-\lambda)L_{att},$ 0 AUC 97.65% and EER 7.75%, as well as balanced AUC 98.73%, by combining CTC-guided keyframe selection with multimodal fusion (Li et al., 9 Jun 2026). MATE reports LP $L=\lambda L_{ctc}+(1-\lambda)L_{att},$ 1 EER 20.06 and AUC 88.70 without inference overhead, improving utterance-level matching through nested sub-embeddings (Jung et al., 20 Jan 2026). ADML + SF2 reports LP $L=\lambda L_{ctc}+(1-\lambda)L_{att},$ 2 EER 20.09 and AUC 88.71, showing that modality-adversarial training also materially improves text-enrollment OV-KWS (Jung et al., 22 May 2025). These results are not directly interchangeable because architectures, supervision, and baselines differ, but they consistently indicate that hard-negative handling and alignment design dominate performance on realistic confusable cases.

Multilingual and out-of-domain generalization remain active directions. AdaKWS reports strong multilingual performance, including 94.6 overall F1 on VoxPopuli with AdaKWS-Small and 71.6 overall F1 on low-resource FLEURS languages with AdaKWS-Base (Navon et al., 2023). KeySEM, although pretrained only on English, improves few-shot KWS on Japanese, Esperanto, Polish, and Portuguese and supports sequential keyword addition without revisiting older positive examples (Awasthi et al., 2021). HyperSpotter-c reaches AUC 98.90, F1 94.88, and EER 4.918 on VoxPopuli with 5.5M parameters, while the smallest HyperSpotter-c model has 4.2M parameters and is presented as matching or outperforming several larger models (Segal-Feldman et al., 6 Aug 2025).

Another direction is integration with ASR. CB-Whisper inserts OV-KWS between Whisper’s encoder and decoder, performs keyword spotting on encoder hidden states with a lightweight CNN, and uses detected entities as prompts for contextual biasing (Li et al., 2023). Massive OV-KWS extends the same application to glossaries of 16,062 terms and beyond, showing that compressed acoustic OV-KWS can remain practical even when contextual biasing would otherwise be memory-bound (Barreiros et al., 9 Jun 2026).

The present literature also identifies unresolved trade-offs. POB training improves partial-overlap robustness but can degrade performance on Google Speech Commands, especially because that benchmark contains only one-word commands (Liu et al., 9 Feb 2026). U2-KWS notes that full-decoder mode improves accuracy but adds latency and compute (Zhang et al., 2023). Synaspot’s scoring remains heuristic and similarity-based rather than a learned streaming decoder (Li et al., 17 Dec 2025). Massive OV-KWS still scales linearly with the number of keywords, even after large compression gains (Barreiros et al., 9 Jun 2026). A plausible implication is that future OV-KWS work will increasingly focus on benchmark design, confusable-word curricula, glossary-size scalability, and architectures that remain robust across both short commands and long partially overlapping phrases.