Keyword-Aware Loss Function

• Keyword-aware loss functions are training objectives that prioritize keyword-relevant units over uniform supervision, enhancing the model's focus on detection and retrieval tasks.
• They modify standard loss computations by selecting high-confidence frames, applying temporal smoothing, ranking mechanisms, or contrastive losses tailored to keyword events.
• Empirical studies show these methods can significantly reduce error rates and improve metrics in applications like keyword spotting, ASR biasing, and cross-modal retrieval.

A keyword-aware loss function is a training objective that places explicit emphasis on keyword-relevant units rather than supervising all positions uniformly. In the cited literature, the emphasized unit can be a single high-confidence frame inside a keyword segment, a smoothed temporal peak inside a broad pooling window, a positive–negative score ordering between keyword and non-keyword examples, the false-accept and false-reject count of a special <kw> token over an $N$-best list, a context-weighted subset of query words in cross-modal retrieval, or the positions and identities of biased rare-word tokens in contextual ASR. The common rationale is to align optimization with the downstream event of interest—detection, retrieval, or biasing—more directly than standard frame-wise or token-wise cross-entropy (Sun et al., 2017, Park et al., 2020, Xu et al., 2021, Labrador et al., 2022, Um et al., 5 Jan 2025, Kwok et al., 11 Sep 2025).

1. Problem formulation and recurring design patterns

The central design choice in a keyword-aware loss is the object that becomes keyword-conditioned. In small-footprint keyword spotting, one line of work treats the keyword as a temporal event and selects only one frame or one smoothed peak per keyword occurrence for positive supervision, while preserving ordinary background penalties elsewhere (Sun et al., 2017, Park et al., 2020). Another line of work treats the keyword as a ranking target and optimizes separability between true-keyword scores and competing or spurious scores by maximizing a differentiable surrogate of the area under the receiver-operating-characteristic curve (Xu et al., 2021). Sequence-to-sequence keyword spotting adapts Minimum Bayes-Risk training so that the risk counts only false accepts and false rejects of a special <kw> token, thereby making the loss explicitly depend on keyword-count errors rather than general transcription mismatch (Labrador et al., 2022).

Cross-modal and contextual-biasing settings instantiate the same principle differently. In video moment retrieval and highlight detection, keyword awareness is introduced by weighting text words according to their cosine similarity with clustered video-context features, and then using those weighted text representations inside clip-level and video-level contrastive objectives (Um et al., 5 Jan 2025). In contextual biasing for rare-word ASR, the loss is decomposed into a masked cross-entropy term that trains the pointer network only at biased-word positions and a binary classification term that learns when the model should bias at all (Kwok et al., 11 Sep 2025). Audio-text based keyword spotting adds another variant: Relational Proxy Loss focuses on structural relations within acoustic embeddings and within text embeddings, rather than only comparing acoustic and text embeddings on a point-to-point basis (Jung et al., 2024).

Taken together, these formulations indicate that “keyword-aware” is not a single canonical formula. It denotes a family of objectives in which keyword relevance modifies frame selection, window selection, score ranking, hypothesis risk, word weighting, or bias-gating behavior.

2. Segment-level pooling losses in keyword spotting

A foundational formulation appears in max-pooling loss training for Long Short-Term Memory networks in small-footprint keyword spotting. Let $X=(x_1,\dots,x_T)$ be the input sequence, $y_t$ the softmax output at frame $t$, $L=\{\ell_1,\dots,\ell_P\}$ the frame-index ranges for the $P$ keyword occurrences, $\hat{\ell}$ the background frames, and $k_p^\dagger$ the true keyword label for segment $\ell_p$. For each keyword segment,

$$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$

The max-pooling loss is

$X=(x_1,\dots,x_T)$0

This differs from frame-wise cross-entropy,

$X=(x_1,\dots,x_T)$1

by applying ordinary frame-wise cross-entropy to all background frames but supervising only the single “best” frame within each keyword segment. The stated motivation is that run-time keyword spotting needs one strong spike per segment, and max-pooling loss trains the model to concentrate confidence into one peak, matching the downstream detection rule based on smoothed-posterior thresholding (Sun et al., 2017).

The training and evaluation protocol reinforces that alignment. The model uses 20-dim log-Mel filterbank energies, a unidirectional LSTM with 64 memory cells and projection layer size 32, left context of 10 frames, and right context of 10 frames. At test time, the smoothed keyword posterior is

$X=(x_1,\dots,x_T)$2

with $X=(x_1,\dots,x_T)$3, a trigger threshold $X=(x_1,\dots,x_T)$4, lockout $X=(x_1,\dots,x_T)$5, and latency window $X=(x_1,\dots,x_T)$6. On far-field recordings of the single keyword “Alexa,” the max-pooling loss trained LSTM initialized with a cross-entropy pre-trained network yielded $X=(x_1,\dots,x_T)$7 relative reduction compared to the baseline feed-forward DNN in Area Under the Curve measure; the LSTM trained with max-pooling loss from random initialization yielded $X=(x_1,\dots,x_T)$8 AUC change, and the LSTM trained with cross-entropy yielded $X=(x_1,\dots,x_T)$9 (Sun et al., 2017).

Smoothed Max-Pooling generalizes this idea by replacing exact frame selection with smoothed temporal peaks inside positive pooling windows. With a temporal smoothing kernel $y_t$0, smoothed score $y_t$1, positive windows $y_t$2, and complement set $y_t$3, the loss is

$y_t$4

where

$y_t$5

and

$y_t$6

The encoder uses $y_t$7 consecutive windows to discover keyword parts, while the decoder uses one window that straddles the keyword end. The total loss is

$y_t$8

The paper states that this allows semi-supervised joint training of encoder and decoder without strictly depending on frame-level labeling from LVCSR; only approximate word-end information is used to position windows (Park et al., 2020).

Empirically, the best model, “Max4 SMP_SMP,” achieved false-reject rates of $y_t$9 on clean non-accented speech, $t$0 on accented speech, $t$1 on query logs, and $t$2 on noisy in-vehicle speech at $t$3 FA/hr, compared with the baseline “CE_CE” rates of $t$4, $t$5, $t$6, and $t$7, respectively. The summary states that SMP_SMP reduces false rejects by $t$8–$t$9 relative to baseline across all conditions, while omitting smoothing in “MP_MP” hurts, indicating that temporal smoothing is essential for stable peaks (Park et al., 2020).

3. Ranking and risk minimization for keyword detection

A different strand of keyword-aware loss design starts from ranking. For binary classification, the AUC objective is

$L=\{\ell_1,\dots,\ell_P\}$0

Because the indicator is non-differentiable, the cited work replaces it with a squared-hinge surrogate,

$L=\{\ell_1,\dots,\ell_P\}$1

leading to

$L=\{\ell_1,\dots,\ell_P\}$2

For multi-class keyword spotting, the network produces keyword-confidence scores $L=\{\ell_1,\dots,\ell_P\}$3, the positive-score set is

$L=\{\ell_1,\dots,\ell_P\}$4

and the negative-score set is

$L=\{\ell_1,\dots,\ell_P\}$5

with

$L=\{\ell_1,\dots,\ell_P\}$6

for keyword samples and

$L=\{\ell_1,\dots,\ell_P\}$7

for non-keyword samples. The minibatch loss becomes

$L=\{\ell_1,\dots,\ell_P\}$8

The paper reports that $L=\{\ell_1,\dots,\ell_P\}$9 gave the best trade-off, and on Google Speech Commands v1 the multi-class AUC model with fixed-proportion sampling achieved total accuracy $P$0, closed accuracy $P$1, and macro-averaged $P$2, compared with $P$3, $P$4, and $P$5 for cross-entropy. On v2, the fixed-sampler mAUC model achieved $P$6, $P$7, and $P$8, compared with $P$9, $\hat{\ell}$0, and $\hat{\ell}$1 for cross-entropy (Xu et al., 2021).

Sequence-discriminative keyword spotting applies keyword awareness at the level of hypothesis risk. Classical Minimum Bayes-Risk training minimizes

$\hat{\ell}$2

but the keyword-specific specialization counts only <kw> errors. With $\hat{\ell}$3 the number of <kw> tokens in hypothesis $\hat{\ell}$4 and $\hat{\ell}$5 the number in the reference,

$\hat{\ell}$6

and the risk is

$\hat{\ell}$7

The total fine-tuning objective adds an RNN-T regularizer: $\hat{\ell}$8 In practice, beam size $\hat{\ell}$9, top $k_p^\dagger$0 hypotheses, learning rate $k_p^\dagger$1, $k_p^\dagger$2, $k_p^\dagger$3, $k_p^\dagger$4, and $k_p^\dagger$5 are used. At inference time, the utterance-level confidence is

$k_p^\dagger$6

and thresholding this score yields the detection decision. This formulation makes the loss directly sensitive to the two operational errors of keyword spotting—false accepts and false rejects—rather than to generic sequence mismatch (Labrador et al., 2022).

4. Keyword-aware contrastive learning in cross-modal retrieval

In video moment retrieval and highlight detection, keyword-aware loss is introduced through context-dependent word weighting and two contrastive objectives. Let $k_p^\dagger$7 be word-level text features, $k_p^\dagger$8 clustered video-context features, and $k_p^\dagger$9 a temperature. The similarity matrix is

$\ell_p$0

A column-wise softmax followed by max-pooling over clusters yields the per-word weight vector

$\ell_p$1

and the keyword-weighted text features are

$\ell_p$2

This assigns larger weights to words that are more discriminative under the overall video context (Um et al., 5 Jan 2025).

The first contrastive term, the clip-keyword loss $\ell_p$3, aligns each clip feature with the aggregated keyword-weighted query representation. With clip-level video features $\ell_p$4, aggregated weighted text embedding $\ell_p$5, ground-truth moment indices $\ell_p$6, batch size $\ell_p$7, and cosine similarity $\ell_p$8,

$\ell_p$9

The second term, the video-keyword loss $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$0, uses a pooled positive video feature $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$1 and aligns it against the keyword-weighted query across the batch: $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$2 The combined keyword-aware contrastive loss is

$$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$3

and the full training objective is

$$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$4

with $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$5 (Um et al., 5 Jan 2025).

The quantitative evidence is an ablation on the QVHighlights validation split. The baseline without $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$6 and $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$7 obtains $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$8, $$l_p^\dagger = \arg\max_{t\in \ell_p} y_t^{k_p^\dagger}.$$9, HD-mAP $X=(x_1,\dots,x_T)$00, and HIT@1 $X=(x_1,\dots,x_T)$01. Adding only $X=(x_1,\dots,x_T)$02 gives $X=(x_1,\dots,x_T)$03, $X=(x_1,\dots,x_T)$04, $X=(x_1,\dots,x_T)$05, and $X=(x_1,\dots,x_T)$06; adding only $X=(x_1,\dots,x_T)$07 gives $X=(x_1,\dots,x_T)$08, $X=(x_1,\dots,x_T)$09, $X=(x_1,\dots,x_T)$10, and $X=(x_1,\dots,x_T)$11; adding both gives $X=(x_1,\dots,x_T)$12, $X=(x_1,\dots,x_T)$13, $X=(x_1,\dots,x_T)$14, and $X=(x_1,\dots,x_T)$15. The summary states that adding either contrastive term yields a solid gain over the baseline and that combining both yields the largest improvement (Um et al., 5 Jan 2025).

5. Contextual biasing and rare-word decoding

Keyword-aware loss also appears in contextual biasing, where the objective is not generic token prediction but targeted improvement on biased rare words. In the cited TCPGen-based formulation, $X=(x_1,\dots,x_T)$16 is the target sequence length, $X=(x_1,\dots,x_T)$17 the vocabulary size, and $X=(x_1,\dots,x_T)$18 the set of time-step indices at which a biased token occurs. At decoding step $X=(x_1,\dots,x_T)$19, the system has the original ASR model probability $X=(x_1,\dots,x_T)$20, the pointer-network probability $X=(x_1,\dots,x_T)$21 over a candidate set $X=(x_1,\dots,x_T)$22, and the interpolation probability $X=(x_1,\dots,x_T)$23 (Kwok et al., 11 Sep 2025).

The first keyword-aware term is a masked cross-entropy on the pointer output: $X=(x_1,\dots,x_T)$24 The mask ensures that the pointer network is trained only at positions where a biased word actually occurs. The second term is a binary classification loss on the generation probability: $X=(x_1,\dots,x_T)$25 The combined loss is

$X=(x_1,\dots,x_T)$26

No additional weighting between $X=(x_1,\dots,x_T)$27 and $X=(x_1,\dots,x_T)$28 is used, and the only hyperparameter introduced is $X=(x_1,\dots,x_T)$29, with $X=(x_1,\dots,x_T)$30 reported to work well. The probability masking strategy further sets

$X=(x_1,\dots,x_T)$31

for $X=(x_1,\dots,x_T)$32, removing the scaling factor $X=(x_1,\dots,x_T)$33 outside the pointer set in order to avoid over-suppression of non-biased tokens (Kwok et al., 11 Sep 2025).

The training pipeline freezes the encoder weights of the base ASR model and trains only TCPGen with $X=(x_1,\dots,x_T)$34, explicitly not using a standard ASR loss for TCPGen. On NSC-Part-2 with Whisper-small, the unadapted model has WER $X=(x_1,\dots,x_T)$35, B-WER $X=(x_1,\dots,x_T)$36, and U-WER $X=(x_1,\dots,x_T)$37. Fine-tuning on $X=(x_1,\dots,x_T)$38 hours of synthetic data yields WER $X=(x_1,\dots,x_T)$39, adding AGEM gives $X=(x_1,\dots,x_T)$40, standard TCPGen with ASR loss remains at $X=(x_1,\dots,x_T)$41, and TCPGen-2L with $X=(x_1,\dots,x_T)$42 and $X=(x_1,\dots,x_T)$43 achieves WER $X=(x_1,\dots,x_T)$44, B-WER $X=(x_1,\dots,x_T)$45, and U-WER $X=(x_1,\dots,x_T)$46. The abstract reports that adapting Whisper to $X=(x_1,\dots,x_T)$47 hours of synthetic data reduced the word error rate on the NSC Part 2 test set from $X=(x_1,\dots,x_T)$48 to $X=(x_1,\dots,x_T)$49 (Kwok et al., 11 Sep 2025).

The $X=(x_1,\dots,x_T)$50 ablation clarifies the operating trade-off. On the NSC development set, increasing $X=(x_1,\dots,x_T)$51 from $X=(x_1,\dots,x_T)$52 to $X=(x_1,\dots,x_T)$53 raises FAR from $X=(x_1,\dots,x_T)$54 to $X=(x_1,\dots,x_T)$55 and TAR from $X=(x_1,\dots,x_T)$56 to $X=(x_1,\dots,x_T)$57, while reducing B-WER from $X=(x_1,\dots,x_T)$58 to $X=(x_1,\dots,x_T)$59. The summary states that as $X=(x_1,\dots,x_T)$60 increases, the model fires more often, reducing B-WER but slightly increasing U-WER from over-biasing. This makes the loss interpretable in terms of “what to bias” and “when to bias,” rather than only end-task WER (Kwok et al., 11 Sep 2025).

6. Relational formulations, misconceptions, and open boundaries of