Open-Vocabulary Keyword Spotting

• Open-vocabulary keyword spotting is a technique that detects arbitrary, user-specified keywords in continuous speech using cross-modal alignment between audio and text.
• It leverages dual-encoder architectures and metric learning losses to generalize to unseen words and ensure robust discrimination under noisy, real-time conditions.
• Innovations include phoneme-level alignment, matched filtering, and quantized models that enable fast, low-latency detection suited for embedded and multilingual applications.

Open-vocabulary keyword spotting (OV-KWS) refers to systems that detect arbitrary, user- or application-defined keywords (words or short phrases) within continuous speech, without limiting the vocabulary to those seen during system training. These systems enable truly flexible, personalized, and scalable voice interfaces, as new keywords can be enrolled using text or audio without retraining the base model. OV-KWS has emerged as a distinct research area at the intersection of speech recognition, deep metric learning, and representation learning for cross-modal audio-text alignment.

1. Fundamental Principles and Problem Scope

Open-vocabulary keyword spotting systems accept as input a user- or application-specified target keyword $K$—represented as text, spoken sample, or phoneme sequence—and must detect instances of $K$ within streaming or archival audio $W = (w_1,\ldots,w_T)$ (Liu et al., 9 Feb 2026, Li et al., 2023). Unlike closed-set KWS, which limits detection to a fixed set of enrolled keywords and often requires training data per keyword, OV-KWS generalizes to unseen or rare words and supports real-time, on-the-fly enrollment.

Core aspects of the OV-KWS problem:

• Unseen keyword generalization: Models must recognize words or multi-word phrases never seen in training (Navon et al., 2023, Zhu et al., 2023).
• Enrollment flexibility: Systems accept enrollment as text, audio, phonemic, or multimodal input (Kewei et al., 2024, Li et al., 17 Dec 2025).
• Robust discrimination: Detections must be robust to acoustic confusability, partial overlaps (e.g., “turn the volume up” vs. “turn the volume down”), and noise (Liu et al., 9 Feb 2026, Kewei et al., 2024).
• Resource and latency constraints: Deployment targets include embedded and streaming settings, requiring small parameter footprint, low latency, and minimal compute (Bluche et al., 2019, Li et al., 17 Dec 2025, Segal-Feldman et al., 6 Aug 2025, Bluche et al., 2020).
• Multilinguality: Cross-lingual transfer and zero-shot support in new languages is a key requirement for universal access (Zhu et al., 2023, Mazumder et al., 2021).

OV-KWS methods are distinguished by their ability to generalize beyond the original training vocabulary, accommodate personalized and domain-specific terms, and scale to large or even massive keyword glossaries with reasonable storage and compute (Barreiros et al., 9 Jun 2026).

2. Model Architectures and Cross-Modal Alignment Strategies

Modern OV-KWS frameworks are dominated by deep neural dual-encoder architectures, often with modality-specific backbones for audio and text (Navon et al., 2023, Jung et al., 22 May 2025, Kewei et al., 2024, Jung et al., 20 Jan 2026). Key technical approaches include:

• Joint Embedding Spaces: Both speech and textual representations are projected into a shared vector space in which similarity reflects semantic and phonetic correspondence. Similarity is typically measured via cosine similarity, dot product, or more advanced metric learning objectives (Navon et al., 2023, V et al., 2024, Jung et al., 22 May 2025, Jung et al., 20 Jan 2026).
• Phoneme-level Alignment: Forced alignment or attention-based modules are used to extract phoneme-synchronous feature representations, promoting fine-grained mapping between keyword and query (Kewei et al., 2024, Jung et al., 22 May 2025, Shin et al., 2022).
• Matched Filters and Hypernetworks: Systems generate per-keyword matched filter weights (from text or phoneme input) that parameterize a convolutional or attention-based detector, tightly coupling keyword properties to the detection process (Bluche et al., 2019, Segal-Feldman et al., 6 Aug 2025).
• Keyword-conditioned Adaptation: Conditioning mechanisms, such as adaptive instance normalization (AdaIN) (Navon et al., 2023) or cross-attention (Li et al., 17 Dec 2025, Shin et al., 2022), allow the detector to dynamically focus on the properties of a query keyword.
• Compact and Quantized Models: Small-footprint LSTM/Conv/DFSMN networks with quantization, integer-only inference, or parameter-efficient adaptations to reduce memory and computation while maintaining OV-KWS capability (Bluche et al., 2019, Bluche et al., 2020, Li et al., 17 Dec 2025).
• Large-scale Retrieval and Compression: For massive glossaries (e.g., $K > 10^4$), embedding compression (layer selection, projection, temporal downsampling) and efficient similarity search methods become essential (Barreiros et al., 9 Jun 2026).

3. Key Algorithmic and Training Innovations

The design and optimization of OV-KWS models require several specialized techniques:

• Metric Learning Losses: Contrastive, triplet, InfoNCE, and deep metric learning (DML) objectives such as relational proxy loss (RPL) and asymmetric-proxy (AsyP) are widely used to structure embedding spaces for sharp inter-class separation and intra-class compactness (Jung et al., 22 May 2025, Jung et al., 20 Jan 2026). Overlap-robust variants are also applied for hard disambiguation (Kewei et al., 2024).
• Prefix Bias Mitigation: Position-biased scoring, where models over-weight prefix phonemes, is a key challenge. Equal-weighted position scoring (EPS) removes position-dependence, preventing “prefix bias” false positives (e.g., for commands differing only at the end) (Liu et al., 9 Feb 2026).
• Hard Negative Mining: Negative sampling strategies expose models to highly similar phonetic or orthographic distractors during training, increasing robustness to confusable queries (Navon et al., 2023, Kewei et al., 2024, Jung et al., 20 Jan 2026). Memory banks of phoneme prototypes and explicit synthesis of hard negatives further boost discriminability (Kewei et al., 2024).
• Modality Gap Reduction: Cross-modal adversarial learning, such as modality adversarial learning (MAL), encourages embedding models to be invariant across audio and text, improving generalization and reducing “modality gap” (Jung et al., 22 May 2025).
• Transfer Learning and TTS-guided Text Encoders: Leveraging intermediate representations from pretrained TTS models (e.g., Tacotron 2) injects audio-aware phonetic knowledge into text encoders, strongly aligning cross-modal embeddings (V et al., 2024).
• Multiscale and Matryoshka Embeddings: Architectures such as MATE encode nested, multi-granular embeddings using PCA-guided prefix alignment, enabling the model to capture both salient and detailed cues within a single vector representation (Jung et al., 20 Jan 2026).
• Streaming and Online Alignment: Fast, streaming-capable models use CTC-aligned methods, dynamic programming, and low-overhead aligners for frame-wise or phrase-level matching with $O(U)$ per-frame cost (Jin et al., 2024, Bluche et al., 2020, Zhang et al., 2023).

4. Evaluation Protocols, Datasets, and Performance Benchmarks

Comprehensive open-vocabulary KWS evaluation leverages datasets and protocols designed to test the full range of system capabilities:

• LibriPhrase: Extracted from LibriSpeech, with “easy” (distant) and “hard” (minimal edit) negative splits; supports phrase-level and multi-word detection (Shin et al., 2022, Liu et al., 9 Feb 2026, Kewei et al., 2024).
• POB (Partial Overlap Benchmark): Specifically constructed to test prefix-overlapped negatives, including POB-LibPhrase (POB-LP) and POB-Spark, with explicit control over partial overlap structure (Liu et al., 9 Feb 2026).
• MSWC, FLEURS, VoxPopuli: Large vocabularies, code-switch, low-resource, multilingual and unseen-language evaluation (Navon et al., 2023, Mazumder et al., 2021, Zhu et al., 2023).
• Streaming Protocols: Sliding windows, streaming segmentation, and real-time evaluation for online/embedded deployment (Bluche et al., 2020, Li et al., 17 Dec 2025, Jin et al., 2024).

Performance is quantified using Equal Error Rate (EER), Area Under the Curve (AUC), F1, entity recall, and memory/runtime cost. Results illustrate rapid gains: SLiCK-EPS reduces EER on POB-Spark from 64.41% to 29.28%, AdaKWS achieves F1 of 94.6 on VoxPopuli multilingual (with only 109M parameters), and LHF-comp achieves 128$\times$ memory reduction versus Whisper-based KWS at negligible loss (Liu et al., 9 Feb 2026, Navon et al., 2023, Barreiros et al., 9 Jun 2026).

5. Open Challenges and Current Limitations

Despite progress, OV-KWS faces several persistent challenges:

• Prefix and Confusability Bias: Architectures and training data must balance long-phrase discrimination with precision on short, single-word queries. Overemphasis on prefix overlap can degrade detection on short commands, as found with POB augmentation (Liu et al., 9 Feb 2026).
• Scalability: Handling massive glossaries introduces bottlenecks in entity scoring, storage, and inference. Solutions leveraging embedding compression and layer selection can address storage and runtime, but distractor management and reranking remain active topics (Barreiros et al., 9 Jun 2026).
• Cross-modal Manifold Bridging: Modality gap continues to limit performance, especially in acoustically or phonetically challenging cases and low-resource languages (Jung et al., 22 May 2025, V et al., 2024).
• Personalization and Customization: Achieving both user-specific and open-vocab performance in a unified model, with minimal adaptation cost, is a central goal (Pan et al., 5 Mar 2026, Bluche et al., 2019).
• Streaming and Low-latency: Maintaining high accuracy under strict streaming and real-time constraints, particularly with small-footprint implementations, is necessary for deployment on edge devices (Li et al., 17 Dec 2025, Bluche et al., 2020).

6. Outlook and Future Research Directions

Recent work suggests promising directions for continued advancement of OV-KWS:

• Data Curriculum and Augmentation: Sophisticated data composition strategies, including curriculum learning, up/down-sampling, and synthetic hard negative generation, may enable models to balance performance across different phrase lengths and overlap distributions (Liu et al., 9 Feb 2026).
• Dynamic Position-weight Regularization: Beyond EPS, learned or phoneme-aware positional weighting and attention schemes could further suppress prefix bias while preserving fine discriminability (Liu et al., 9 Feb 2026).
• Modality-adaptive Encoder Architectures: Jointly fine-tuned audio and text encoders, TTS transfer, and modality-adversarial objectives are likely to further close the audio-text gap (Jung et al., 22 May 2025, V et al., 2024).
• Extremely Lightweight, Streaming, Multilingual Systems: DFSMN-based encoders, streaming CTC-aligned detectors, and quantized or model-compressed variants support on-device deployment with sub-1M parameter footprints (Li et al., 17 Dec 2025, Bluche et al., 2020).
• Matryoshka-Style, Multi-scale Representations: Embeddings with nested subspace alignment (e.g., MATE) provide scalable, loss-agnostic performance enhancements at no extra inference cost (Jung et al., 20 Jan 2026).
• Universal Phoneme-based Models: IPA-symbol alignment confers strong cross-lingual generalization, enabling robust zero-shot KWS and forced alignment in any language (Zhu et al., 2023).
• Massive-scale Candidate Scanning: Sparse layer selection, aggressive quantization, and hierarchical compression will likely see increasing adoption to support massive open-vocabulary search (Barreiros et al., 9 Jun 2026).

By continually addressing the challenges of prefix bias, modality heterogeneity, and resource constraints, and by leveraging advancements in cross-modal representation, streaming, and compression, OV-KWS is positioned as a critical enabling technology for future voice-driven interfaces in diverse and dynamic application contexts.