Device-Directed Speech Detection
- Device-Directed Speech Detection is a binary classification task that differentiates intended device speech from background chatter using acoustic, ASR uncertainty, and lexical cues.
- Early architectures fused multiple modalities and achieved up to a 44% relative improvement over single-stream approaches by combining acoustic embeddings with decoder features.
- Recent methods integrate multimodal fusion, streaming inference, and LLM-based strategies to enhance robustness and enable efficient on-device deployment.
Device-Directed Speech Detection (DDSD) is the binary inference problem of deciding whether a spoken utterance is addressed to a voice assistant or device, as opposed to side conversation, background speech, media audio, or other non-target speech. In the literature summarized here, DDSD is motivated by two recurring product conditions: rejection of false wake-ups or unintended activations, and support for wake-word-free or triggerless follow-up interaction after an initial command. The task has evolved from utterance-level classifiers that fuse acoustics, ASR decoder uncertainty, and lexical hypotheses into broader multimodal, streaming, and large-foundation-model formulations, while retaining the same operational objective: accept device-directed speech and suppress non-device-directed speech under realistic deployment constraints (Mallidi et al., 2018, Krishna et al., 2023, Palaskar et al., 2024, Kim et al., 9 Apr 2026).
1. Canonical task definition and relation to adjacent problems
In its canonical form, DDSD is a binary classification task with labels such as device-directed versus nondevice-directed, or equivalently intended versus unintended speech. The practical setting is usually a voice assistant that has already been activated, either by a wakeword, touch, or a preceding conversational turn, after which the system must determine whether subsequent speech should be routed to downstream ASR, NLU, or response generation modules. This framing is explicit in early follow-up-query work and remains central in later studies of multimodal and LLM-based systems (Mallidi et al., 2018, Gillespie et al., 2020, Ognjen et al., 2024).
A persistent distinction in the literature is between DDSD and wake-word spotting. Wake-word systems detect a specific lexical trigger; DDSD infers user intent in the absence of such a fixed lexical anchor. For touch-based invocation, this distinction becomes especially stark, because the model must determine whether arbitrary payload speech is directed to the device without keyword evidence. Several papers therefore describe DDSD as a form of false-trigger mitigation that is harder than conventional keyword detection because the utterance content is unconstrained and the relevant cues span acoustics, ASR confidence, lexical structure, and conversational context (Garg et al., 2022, Rudovic et al., 2021).
The literature also identifies nearby but non-identical tasks. A mobile-array front/back speech detector can be interpreted as a spatial proxy for likely device-directed speech, but it does not model addressee intent, wake words, or lexical semantics; the “front” class is only an approximation to device-directedness (Pertilä et al., 2021). An accessibility-oriented detector of intentional nonverbal mouth sounds similarly solves a DDSD-like gating problem—distinguishing deliberate user input from speech and background audio—but its target events are closed-set nonverbal commands rather than open-vocabulary directed speech (Lea et al., 2022). These related tasks are best understood as precursors or specialized variants rather than substitutes for canonical DDSD.
A further conceptual reframing appears in work on Sequential Device-Addressed Routing (SDAR), which argues that utterance-local classification is insufficient in multi-speaker, pre-ASR edge settings. In that formulation, DDSD becomes a sequential, cost-sensitive routing problem over current evidence , bounded interaction history , and deployment thresholds, with actions . This suggests that some ambiguity in device addressability is fundamentally temporal rather than utterance-local (Kim et al., 9 Apr 2026).
2. Early architectures: acoustics, ASR hypotheses, and decoder uncertainty
The 2018 baseline formulation established a three-source architecture that remains historically central. It used short-term acoustics, ASR decoder features, and the ASR 1-best hypothesis as complementary signals. The acoustic branch consumed 64-dimensional log filter-bank energies (LFBEs) computed with a 25 ms window and 10 ms shift, trained an LSTM with repeated utterance labels at frame level, and used the pre-softmax output of the last frame as an utterance representation called the acoustic embedding . The lexical branch encoded the 1-best character sequence with pre-trained GloVe embeddings, trained an LSTM over character positions, and used the last character output as the char embedding . Decoder-side features were engineered from the ASR trellis and confusion network, including trellis entropy, Viterbi costs, ASR confidence, and average number of arcs per node. The fused utterance vector was written as
The paper reports this as a 22-dimensional utterance vector passed to a feed-forward DNN classifier (Mallidi et al., 2018).
That system also established a recurring empirical pattern: decoder uncertainty features are individually strong, lexical hypotheses alone are weaker, and multimodal combination is substantially better than any single stream. On real recordings of natural far-field interactions, the reported EERs were 9.3% for decoder features, 10.9% for the acoustic embedding, and 20.1% for the char embedding; pairwise combinations improved further, and the full system reached 5.2% EER, described as a 44% relative improvement over the best single feature set (Mallidi et al., 2018).
Subsequent lexical-semantic work broadened the textual side of DDSD. A 2020 study treated device-directedness classification as a mixed lexical and lightweight acoustic problem using current-turn ASR text, optional previous-turn text, and token-level ASR confidence. Word embeddings were 300-dimensional fastText wiki news embeddings, the lightweight acoustic feature was the token-level ASR confidence, and the best system was an LSTM + Attention model. On a follow-up dataset of roughly 240K utterances, the best configuration using current text, previous text, and token confidences achieved 9.1% EER, compared with 10.6% for the state-of-the-art acoustic-only baseline it referenced. The paper reports this as a 14% relative EER reduction. Its ablation study found removal of token confidences especially damaging, with EER rising to 14.8%; removal of current text raised EER to 13.7%, whereas removal of previous-turn text raised it to 9.5% (Gillespie et al., 2020).
Two points are notable in these early architectures. First, ASR uncertainty is not merely an auxiliary confidence score; it is repeatedly treated as a structured cue to mismatch between speech and a device-directed distribution. Second, lexical information becomes more valuable when previous-turn context is available, particularly in wake-word-free follow-up scenarios where short commands such as “play,” “stop,” or “next” may appear in both directed and non-directed conditions (Mallidi et al., 2018, Gillespie et al., 2020).
3. Multimodal fusion, prosody, and missing-modality robustness
Later DDSD work generalized the classic acoustic-text-decoder triad into broader multimodal systems with explicit robustness mechanisms. One line of work added non-verbal/prosodic cues to otherwise verbal DDSD pipelines. In that formulation, the verbal side comprised an acoustic false trigger mitigator on 40-dimensional filter-bank features, a text DDSD model built as a DNN head on top of a fine-tuned BERT-like model, and an ASR/lattice-based model using a modified LatticeRNN. The prosodic side extracted pitch, voicing, jitter, shimmer, and VAD scores at 100 Hz, with VAD implemented as a CNN-based speech/silence classifier followed by a 2-state HMM. The prosody model was a single-layer GRU with 128 hidden units, trained with Adam, learning rate 0.001, 50 epochs, batch size 150, weighted binary cross-entropy, and about 50K parameters (Krishna et al., 2023).
That work compared late fusion of scalar scores with intermediate fusion of penultimate-layer embeddings. The scores were denoted
and the embeddings
with dimensions 256, 128, 16, and 128 respectively. Its main finding was that intermediate fusion outperforms late fusion, and that prosody provides complementary information when fused nonlinearly at the embedding level. On a dataset of over 19,000 audio recordings, about 530 hours, and roughly 245k segments, verbal-only intermediate fusion reached 7.1% FA@10% FR and 9.2% EER; adding prosody improved this to 6.5% FA@10% FR and 8.9% EER. With modality dropout, the prosody-augmented intermediate-fusion model reached 5.7% FA@10% FR and 8.9% EER on the full test set, and under a missing-modality test condition with 30% of modality inputs randomly dropped, FA@10% FR improved from 11.46% to 10.61% when modality dropout was added (Krishna et al., 2023).
A related but distinct robustness problem arises when text and audio are present but misaligned because ASR transcripts are erroneous. The M0V formulation addresses this by treating DDSD as a multi-view learning problem with four views: audio-only evidence, text-only evidence, multimodal fused evidence, and an explicit text-audio alignment view learned with an InfoNCE-style contrastive loss. The model uses Wav2vec2 as the audio encoder and GPT2 as the text encoder, with separate FFN heads for the audio view 1, text view 2, and multimodal view 3, plus an alignment branch 4. At inference, a policy module either arbitrates among views using thresholds on the alignment score or fuses the four view scores with an SVM. On real in-vehicle assistant data from NOMI, the multi-modal baseline reached 95.27% accuracy and 6.59% EER on the normal test set, while the paper reports 96.41% accuracy for the full framework. On an ASR error test set with 55.60% character error rate (CER), the paper reports 95.71% accuracy and states that this surpasses human judgment performance on that test set (Wang et al., 2024).
These multimodal studies jointly suggest that DDSD performance depends not only on having multiple information streams, but also on how modality reliability is represented. Prosody captures “how” an utterance is produced; alignment branches estimate whether lexical and acoustic evidence should be jointly trusted; modality dropout trains fusion models not to fail when some streams disappear (Krishna et al., 2023, Wang et al., 2024).
4. Streaming and on-device DDSD
A major branch of DDSD research targets streaming inference and strict on-device resource budgets. One purely acoustic line developed a streaming ResLSTM architecture with a CNN front-end, residual connections, stacked unidirectional LSTMs, and a causal mean aggregation layer. The CNN contained 13 convolutional layers total, followed by 3 unidirectional LSTM layers with 64 units each and 2 fully connected layers of size 64. Its key streaming mechanism computed a running utterance representation
5
with the recursive form
6
The model used 256-dimensional log energy of STFT features, 30 ms frames, and utterances truncated to 300 frames. On 4,000 hours of training audio and 6 million utterances, the paper reports that ResLSTM achieved a 41.1% relative EER reduction over its baseline LSTM-S, and that causal mean aggregation matched attention at end-of-utterance evaluation while performing better for earlier decisions (Tong et al., 2020).
A closely related deployment problem is unified streaming DDSD across voice-triggered (VT) and touch-based (TB) invocation. An on-device acoustic false-trigger-mitigation framework addressed both with a common streaming architecture and a new s-TCN decision layer derived from temporal convolutional networks. Audio was processed in causal overlapping blocks of size 7 frames with shift 8, corresponding to about 1.92 s of context per block and a new decision about every 1 second after the first block. The model used 40-D Mel filterbank features, augmented to 280-D with neighboring frames, and 6 streaming attention layers. At the selected operating points, VT FAR values were 3.2% for the non-streaming A2A baseline, 4.2% for s-LSTM, 6.4% for s-TCN, and 8.6% for s-AVE; on the harder TB task, FAR values were 24.3% for A2A, 25.1% for s-TCN, and 28.7% for both s-LSTM and s-AVE. The paper further reports 33.3% runtime peak-memory reduction for s-TCN relative to the non-streaming counterpart, compared with 7.3% for s-LSTM (Rudovic et al., 2021).
Playback interference constitutes another streaming failure mode. Instead of explicit echo cancellation followed by detection, an implicit acoustic echo cancellation (iAEC) framework conditions the detector directly on a playback reference signal. The mixture model is written as
9
or under a linear approximation
0
The most effective variant, iAEC-M, uses shared encoders for mixture and reference features, derives a sigmoid mask
1
and gates the mixture representation via
2
On a real Alexa follow-up-mode dataset with 538k utterances, the baseline TCN had non-playback FRR 0.189 and playback FRR 0.348, whereas iAEC-M at the D2 fusion point achieved non-playback FRR 0.181 and playback FRR 0.150, which the paper describes as a 56% reduction in false-reject rate for the DDD task during device playback conditions (Cornell et al., 2021).
The strongest deployment-oriented reframing appears in SAS, which treats pre-ASR device-addressed speech detection as SDAR. SAS is a three-stage cascade: acoustic geometry or beamforming, utterance-level classification with a lightweight 1D convolutional classifier over 64-dim log-mel filterbank features using 25 ms frames and 10 ms hop, and a small causal Transformer over an 8-second history window of previous Stage-2 outputs, VAD state, and inter-utterance time deltas. Stage 3 produces a multiplicative prior
3
On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieved F1=0.86 with precision=0.89 and recall=0.83; with optional camera-based fusion, audio+video reached F1=0.95, precision=0.97, and recall=0.93. Removing Stage 3 reduced audio+video F1 from 0.95 to 0.57 ± 0.03, described as the largest observed ablation effect. The full system runs on ARM Cortex-A class hardware with <150 ms latency and <20 MB footprint (Kim et al., 9 Apr 2026).
5. Weak supervision, distillation, and compact deployment models
DDSD research has repeatedly confronted the problem that labeled intended-versus-unintended data are expensive or ambiguous. One solution is weak supervision regularized by distillation. In a touch-based false-trigger-mitigation setting, weak labels were generated deterministically using SNR and ASR-text intent rules: samples with 4 and background-speech intent were labeled unintended; samples with 5 and not-background-speech intent were labeled intended; all others were discarded. The paper reports 8% misclassification for the low-SNR rule and 2% misclassification for the high-SNR rule on the covered subset, motivating regularization. An acoustics-only AFTM model trained on 40-D mel filterbanks at 100 fps and a stronger ASR-based LatticeRNN teacher were combined via
6
with 7. The distilled acoustic model, AFTM-D, improved from 18.4% EER to 6.2% EER, which the paper describes as a 66% gain in accuracy over the base AFTM, while fusion of LRNN and AFTM-D achieved 4.0% EER versus 4.8% for LRNN alone (Garg et al., 2022).
A more recent knowledge-distillation line adopts a large pretrained ASR acoustic encoder as teacher and a small DDSD student. In Adaptive Knowledge Distillation for Device-Directed Speech Detection, the teacher is a frozen 12-layer Conformer ASR encoder with a task-specific DDSD head, while the student is a compact Transformer or Conformer of about 5M parameters. The distillation objective combines supervised cross-entropy with embedding distillation,
8
pseudo-labeling,
9
and attention regularization,
0
The paper reports +26% relative improvement in EER for keyword-based invocations and +19% for keyword-free follow-up invocations, and states 22% average EER reduction across invocation types (Chi et al., 4 Aug 2025).
A different compact-deployment strategy appears in SELMA, a speech-enabled LLM that handles DDSD jointly with voice trigger detection and ASR. DDSD is formulated generatively: after a task token such as <|DD|>, the model reads the probability of the next token being yes versus no,
1
SELMA is built on Qwen-Audio-Chat, combining a Whisper-large-v2-style audio encoder of about 650M parameters with a decoder-only Qwen backbone of about 7.4B parameters, but trains only 5.5M parameters, about 0.84% of the full model, using LoRA. A key architectural contribution is mean pooling of the audio sequence to a global summary 2 and concatenation with the original sequence. On a DDSD dataset of about 237k queries, the main configuration SELMA 1 achieved 7.78% EER. The paper reports this as a 22% relative EER improvement over a strong baseline at 10.00% EER, and notes that the gating variant SELMA 4 reached 7.63% EER on DDSD, though it was not the preferred overall multitask configuration (Wagner et al., 31 Jan 2025).
These studies converge on a recurring theme: small deployable DDSD models can be improved either by transferring structure from stronger teachers or by exploiting multitask parameter-efficient adaptation, without abandoning the binary directedness objective (Garg et al., 2022, Chi et al., 4 Aug 2025, Wagner et al., 31 Jan 2025).
6. LLM and foundation-model formulations
A major recent shift is the use of LLMs not merely as text encoders but as multimodal decision engines for DDSD. One GPT2-based approach treats DDSD as autoregressive generation conditioned on acoustic prefixes, ASR 1-best hypotheses, and decoder signals. Whisper provides the acoustic sequence, mean-pooled and projected by a mapping network 3; decoder features are projected by 4; GPT2 then receives the acoustic prefix, decoder-signal prefix, the 1-best hypothesis tokens, and a prompt. The system is trained with the autoregressive cross-entropy objective
5
On combined evaluation data, the reported EERs were 13.00% for Whisper plus a classification head, 11.40% for the text-only GPT2 baseline, 8.57% for the audio-only proposed model, 6.81% for text+audio, 6.45% for text+audio+decoder signals, and 6.34% when decoder signals were combined with text, audio, and an additional 500k text-only corpus. The paper reports relative EER improvements of 38.9% over text-only and 61% over audio-only for multimodal configurations, and notes further gains of up to 18% from larger LLMs and LoRA (Wagner et al., 2024).
A closely related decoder-only LLM framework emphasizes data and resource efficiency. It combines ASR 1-best text, decoder features such as graph cost, acoustic cost, word posterior confidence, and number of alternatives per word, and audio representations from either Whisper-medium or a specialized UAD backbone. The audio encoder output is mean-pooled and mapped to an LLM prefix: 6 while decoder signals are mapped to 7. The system uses Falcon 7B or RedPajama 7B with LoRA and prefix tuning. On a balanced dataset of 80,630 training examples and 37,354 evaluation examples, the best multimodal configuration, Multi 4, achieved 8.23% EER with Falcon + UAD, compared with 9.31% for the best audio-only unimodal baseline using the same backbone. The paper also reports that 10k examples with the multimodal UAD configuration can outperform the 80k audio-only baseline, and argues that specialized 256-dimensional UAD representations are more effective than 1024-dimensional Whisper representations in low-data regimes (Wagner et al., 2023).
FLoRA extends parameter-efficient adaptation from unimodal text LLMs to multimodal DDSD. It projects audio embeddings 8, video embeddings 9, and text embeddings 0 into the text-embedding space, concatenates them, and inserts two adapter layers per encoder/decoder layer. The adapted representation is written as
1
To handle missing modalities, adapter dropout skips modality-specific adapters when that modality is absent. On B2B-1, text-only full fine-tuning achieved EER 8.8 and FA@10 7.0, while FLoRA (A+V+T) achieved EER 6.8 and FA@10 2.6, which the paper reports as a 22% relative EER reduction over the text-only approach and near-parity with full multimodal fine-tuning at only about 5% trainable parameters. On B2B-3, FLoRA reached EER 8.4 and FA@10 6.9, compared with 14.0 and 20.2 for text-only full fine-tuning (Palaskar et al., 2024).
A separate follow-up-conversation study uses Vicuna-7B-v1.3 and models DDSD from the first query plus the follow-up ASR text, optionally with an 8-best hypothesis list and ASR costs. It explores prompt-only inference, LoRA-finetuned prompting, a linear classification head on the LLM, and a LoRA-based classifier head. The paper reports that jointly modeling previous-query context and ASR uncertainty yields 20–40% reduction in false alarms at 10% fixed false rejects compared with modeling follow-ups alone. For the main fixed-FRR metric, the best reported ClassHead result is EER = 7.9% and FAR @ 10% FRR = 4.8%, while the best ClassLoRAHead result is EER = 8.5% and FAR @ 10% FRR = 6.1% (Ognjen et al., 2024).
Taken together, these LLM-based studies do not eliminate classic DDSD ingredients; rather, they recast them. Acoustic encoders still provide prosodic and interactional cues, ASR hypotheses still encode lexical structure, and decoder signals still represent uncertainty. What changes is the fusion mechanism: instead of hand-designed binary classifiers over fixed embeddings, the decision is delegated to multimodal LLMs adapted by LoRA, prefix tuning, or modality-specific adapters (Wagner et al., 2024, Wagner et al., 2023, Palaskar et al., 2024, Ognjen et al., 2024).
7. Evaluation practice, recurring findings, and limitations
The dominant DDSD evaluation metric is Equal Error Rate (EER), defined at the operating point on the Detection Error Tradeoff (DET) curve where false positive and false negative rates are equal. Many papers also report operating-point metrics such as FA@10% FR, FA@4%FR, FAR at 5% FRR, or FAR at 10% FRR, reflecting deployment needs where false accepts and false rejects are not equally costly (Mallidi et al., 2018, Krishna et al., 2023, Garg et al., 2022, Ognjen et al., 2024). More deployment-centered work reports class-specific F1, precision, and recall for the device-addressed class, especially in pre-ASR routing scenarios (Kim et al., 9 Apr 2026).
Several findings recur across otherwise different architectures.
Complementarity of modalities: acoustics, lexical hypotheses, decoder uncertainty, and prosody repeatedly provide non-redundant evidence. Early LSTM fusion, intermediate embedding fusion, GPT2-based multimodal prefixing, and FLoRA-style adapters all report improvements over unimodal baselines (Mallidi et al., 2018, Krishna et al., 2023, Wagner et al., 2024, Palaskar et al., 2024).
Importance of uncertainty: decoder features, ASR n-best lists, and alignment views are consistently useful because 1-best transcripts can be ambiguous or incorrect. This is especially explicit in systems using confusion-network confidence, LatticeRNN features, prompt designs that expose n-best costs, and alignment-aware multi-view models (Mallidi et al., 2018, Wagner et al., 2023, Ognjen et al., 2024, Wang et al., 2024).
Value of context and history: previous-turn text modestly improves lexical DDSD; first-query context substantially improves follow-up classification; short-horizon causal interaction history can dominate performance in multi-speaker edge settings. This suggests that a nontrivial portion of DDSD difficulty arises from temporally ambiguous utterances rather than purely local acoustics or lexical form (Gillespie et al., 2020, Ognjen et al., 2024, Kim et al., 9 Apr 2026).
Deployment tradeoffs: streaming and on-device systems often sacrifice some representational richness to meet latency, memory, or pre-ASR constraints. Yet several papers show that causal architectures, TCN-based summarizers, reference-conditioned detectors, and compact student models can remain competitive, particularly when distillation or history-aware routing is added (Tong et al., 2020, Rudovic et al., 2021, Cornell et al., 2021, Chi et al., 4 Aug 2025).
A concise comparison of representative milestones is given below.
| System | Main signals | Reported headline result |
|---|---|---|
| LSTM fusion DDSD (Mallidi et al., 2018) | acoustics + 1-best + decoder features | 5.2% EER |
| Lexical + token confidence (Gillespie et al., 2020) | current text + previous text + token confidences | 9.1% EER |
| Prosody + verbal intermediate fusion (Krishna et al., 2023) | acoustic + text + ASR + prosody | 5.7% FA@10% FR, 8.9% EER |
| Streaming ResLSTM (Tong et al., 2020) | acoustics only | 41.1% relative EER reduction over baseline |
| iAEC-M (Cornell et al., 2021) | mixture audio + playback reference | playback FRR 0.150 vs 0.348 baseline |
| GPT2 multimodal LLM (Wagner et al., 2024) | audio + text + decoder signals | 6.34% EER |
| FLoRA multimodal LLM (Palaskar et al., 2024) | audio + video + text | 6.8 EER, 2.6 FA@10 on B2B-1 |
| SELMA (Wagner et al., 31 Jan 2025) | audio + text in multitask LLM | 7.78% EER |
| SAS / SDAR (Kim et al., 9 Apr 2026) | audio-only or audio+video + causal history | F1=0.86 audio-only, 0.95 audio+video |
The literature also makes clear what DDSD is not. It is not equivalent to wake-word spotting, because many settings of interest are keyword-free. It is not solved by spatial localization alone, because a speech source in front of the device is only a proxy for device addressability. It is not reducible to text-only classification, because ASR ambiguity, overlap, playback, and missing modalities can materially alter the decision boundary. A plausible implication of the collected work is that future DDSD systems will continue to combine multimodal evidence, uncertainty modeling, and causal interaction context, while adapting the exact architecture to whether the deployment point is pre-ASR edge routing, post-ASR follow-up disambiguation, or a multitask assistant stack (Pertilä et al., 2021, Ognjen et al., 2024, Kim et al., 9 Apr 2026).