Shortlisting Model for Domain Classification

• Shortlisting Model (SLM) is a scalable, first-stage system that uses a shared BiLSTM network to score and shortlist candidate domains from a large set.
• It leverages character- and word-level embeddings to capture subword patterns and morphology, enabling effective classification over 1,500 overlapping domains.
• The model prioritizes high recall@k by returning a top-5 candidate list, providing a robust foundation for a subsequent, feature-rich reranking stage.

Searching arXiv for the referenced paper and closely related context. A Shortlisting Model (SLM), in the sense of "A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding," is the first-stage model in a two-stage domain classification pipeline for intelligent personal digital assistants (IPDAs) (Kim et al., 2018). In that work, the SLM is exactly the component called Shortlister: a fast, lightweight, shared neural network that takes a user utterance, scores all candidate domains, and returns a small top-$k$ list for downstream reranking. Its role is to make large-scale domain classification feasible when the system must choose among 1,500 domains, many of them overlapping in functionality, independently evolving, and heterogeneous in quality and popularity (Kim et al., 2018).

1. Problem setting and two-stage formulation

In the large-scale IPDA setting, an utterance such as “play michael jackson” or “how can I bake an apple pie?” must be mapped to a domain, where a domain represents an application or function such as music, recipes, taxi, weather, or smart home (Kim et al., 2018). The paper contrasts this with traditional spoken language understanding deployments containing roughly 20 carefully designed, non-overlapping domains with a shared schema. The large-scale setting instead contains 1,500 overlapping domains, including many user-developed skills, and the task is to choose the best domain among $n$ candidates with $n \approx 1500$ (Kim et al., 2018).

The need for shortlisting is driven by latency, memory, compute, and scalability constraints. A naive design would run each domain’s binary domain classifier, multi-class intent classifier, and sequence slot tagger, then rank the resulting hypotheses. The paper states that this is feasible for 8–9 domains, but not for thousands. Running full NLU for every domain is too slow for a production IPDA, requires prohibitive memory footprint and machine count, and does not scale gracefully as new domains appear (Kim et al., 2018).

The proposed solution is a two-stage pipeline. The first stage is the Shortlisting Model (Shortlister / SLM), which uses only character- and word-level features from the utterance, runs a single shared neural model, scores all domains, and emits a $k$-best candidate list. The second stage is HypRank, a list-wise reranker that uses richer hypothesis vectors including the Shortlister score, domain-intent-slot NLU scores, user preference signals, and domain popularity or quality signals (Kim et al., 2018). Formally, the system is described as:

$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$

This division of labor is central. The SLM is intentionally lightweight and context-free, optimized for high recall at low latency, while the reranker is feature-rich but invoked only on a small candidate set. A plausible implication is that the SLM should be judged less by 1-best accuracy than by whether it preserves the true domain in the shortlist.

2. Architecture of the Shortlisting Model

Shortlister is a shared neural classifier with three layers: an orthography-sensitive character+word embedding layer, a BiLSTM utterance encoder, and an output layer that scores domains (Kim et al., 2018).

Let $\mathcal{C}$ be the set of characters, $\mathcal{W}$ the set of words, and $\oplus$ denote vector concatenation. Character embeddings are

$e_c \in \mathbb{R}^{25}, \quad c \in \mathcal{C},$

and word embeddings are

$e_w \in \mathbb{R}^{100}, \quad w \in \mathcal{W}.$

For each word $n$0, the model runs forward and backward character LSTMs over its characters and forms an orthography-sensitive word embedding $n$1:

$n$2

This representation concatenates the last forward character-LSTM state, the first backward character-LSTM state, and the word embedding. The paper states that this enables the model to capture subword patterns such as misspellings, morphology, and brands, which are important for domain discrimination (Kim et al., 2018).

Given the word-vector sequence $n$3, a word-level BiLSTM encodes the utterance. The final forward and backward hidden states are concatenated into the utterance representation

$n$4

This single vector summarizes the utterance for domain scoring (Kim et al., 2018).

The output layer has two variants. In the global softmax variant, the model computes

$n$5

with

$n$6

In the per-domain in/out softmax variant, each domain $n$7 has its own binary classifier:

$n$8

where $n$9 is the in-domain probability and $n \approx 1500$0 is the out-of-domain probability. At inference time, $n \approx 1500$1 is used as the shortlisting score, and the top-$n \approx 1500$2 domains are selected accordingly (Kim et al., 2018).

The per-domain design is intended to avoid the extreme “winner-takes-all” behavior of a flat softmax in highly overlapping label spaces. This suggests that score independence matters when multiple domains are semantically similar and all deserve to survive into a reranking stage.

3. Training objective and optimization

The training objective depends on the output variant. For the global softmax model, the loss is standard cross-entropy:

$n \approx 1500$3

where $n \approx 1500$4 is the one-hot ground-truth domain vector and $n \approx 1500$5 is the softmax output (Kim et al., 2018).

For the per-domain in/out softmax model, the loss is

$n \approx 1500$6

The factor $n \approx 1500$7 balances the contribution of the many negative domains against the single positive domain, so that in-domain and out-of-domain probabilities are trained on similar scales (Kim et al., 2018).

Implementation details are shared with HypRank. The paper specifies the DyNet framework, Adam optimization with initial learning rate $n \approx 1500$8, and variational dropout for LSTM layers. Training is described as standard mini-batch SGD, with no special handling for class imbalance beyond the balancing factor in $n \approx 1500$9. The model uses neither hierarchical softmax nor sampled softmax nor negative sampling; with $k$0, full-domain training remains practical (Kim et al., 2018).

A notable architectural property is that almost all parameters are shared across domains. The character embeddings, word embeddings, and BiLSTM encoder are global; only the final output heads scale with the number of domains. The model therefore performs a single pass per utterance, followed by either one large matrix multiplication or $k$1 small binary heads, instead of running 1,500 separate domain-specific NLU stacks (Kim et al., 2018).

4. Interaction with reranking and the role of $k$2

The SLM is the first half of a deliberately asymmetric system. Shortlisting uses only raw-text lexical features from the utterance and excludes contextual or user-specific signals. Reranking, by contrast, consumes a hypothesis vector for each shortlisted domain that includes the Shortlister confidence $k$3, intent classification confidence, slot CRF Viterbi path probability, average and maximum slot-tag confidence, pretrained domain, intent, and slot embeddings, user enablement and recent usage indicators, and domain popularity and quality features (Kim et al., 2018).

The paper explicitly states: “For the same reason, this work only uses contextual information in the reranking stage, and the utility of including it in the shortlisting stage is left for future work” (Kim et al., 2018). This constraint is not incidental. The shortlister must be computationally cheap enough to run over all domains, so it excludes intent models, slot models, domain popularity statistics, and personalized context. HypRank can afford to use these richer signals because it runs only on the $k$4 candidates returned by the SLM.

The value of $k$5 is empirically important. The paper reports k-best classification accuracies for $k$6 and notes that performance “starts to level off at 5-best list” in the large-scale setting (Kim et al., 2018). Accordingly, the HypRank experiments use $k$7. This choice balances two considerations stated in the paper: a higher upper bound for reranking, since the true domain is in the top-5 about 96% of the time in the 1,500-domain setting, and modest reranking cost, since only five hypotheses need expensive downstream processing (Kim et al., 2018).

This division also clarifies an important misconception. The SLM is not intended to be a complete domain resolver. Its function is candidate generation with high recall@$k$8, not full contextual resolution.

5. Evaluation and empirical behavior

The paper evaluates Shortlister in two settings: a Traditional IPDA with 20 built-in, high-quality, non-overlapping domains and more than 4M labeled utterances, and a Large-Scale IPDA with 1,500 overlapping domains and more than 6M utterances (Kim et al., 2018). In the large-scale dataset, utterances were originally in strict invocation patterns such as “Ask $k$9 to $$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$0” and were preprocessed to remove the domain invocation string.

The train/dev/test partition is structured so that Shortlister and HypRank do not share training data. For the large-scale setting, the splits are: SL train 5M, SL dev 530K, HR train 830K, HR dev 20K, and test 530K. For the traditional setting, they are: SL train 3M, SL dev 415K, HR train 715K, HR dev 20K, and test 420K (Kim et al., 2018). The non-overlap between Shortlister and HypRank training sets is explicitly used to avoid overfitting HypRank to Shortlister outputs.

The evaluation metric for the SLM is k-best classification accuracy, described as essentially recall@k:

$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$1

Latency is discussed qualitatively rather than numerically; the paper does not publish exact per-utterance latency figures (Kim et al., 2018).

The reported results are as follows:

Setting	Variant	1-best	3-best	5-best
Traditional (20 domains)	$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$2	95.58%	98.45%	98.81%
Traditional (20 domains)	$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$3	95.56%	98.43%	98.77%
Large-Scale (1,500 domains)	$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$4	81.38%	92.53%	95.77%
Large-Scale (1,500 domains)	$$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$5	81.49%	92.81%	95.93%

In the 1,500-domain setting, Shortlister therefore reaches approximately 96% Recall@5, establishing a strong upper bound for the reranking stage (Kim et al., 2018). The per-domain in/out variant $$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$6 is slightly stronger than the global softmax, especially in the large-scale setting. The paper attributes this to better behavior with overlapping domains (Kim et al., 2018).

The paper also reports downstream context. With $$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$7 and $$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$8 in the large-scale IPDA setting, Shortlister alone achieves 81.49%, while the best reranker, $$\text{Utterance } x \xrightarrow{\text{Shortlister (SLM)}} \text{Top-}k \text{ domains } \{d_1,\dots,d_k\} \xrightarrow{\text{HypRank}} \hat{d}.$$9, reaches 93.83%, against an upper bound of 95.93% set by Shortlister’s 5-best recall (Kim et al., 2018). This shows that the SLM’s value lies not merely in standalone classification but in preserving enough candidate quality for a richer list-wise reranker to recover most of the remaining gap.

6. Scalability, design significance, and broader implications

The scalability of the SLM follows from several explicit design choices. First, feature extraction is shared across domains. Second, each utterance requires only a single encoder pass. Third, the shortlister uses only character and word features, with no domain embeddings at shortlisting, no user context, and no domain popularity or intent-slot machinery (Kim et al., 2018). Fourth, the model does not require hierarchical softmax or approximation tricks at 1,500 domains. The paper characterizes the shortlisting cost as essentially one medium-sized BiLSTM plus a $\mathcal{C}$0 linear transform, which is substantially cheaper than running domain-specific NLU for every domain (Kim et al., 2018).

Several broader design lessons are explicit or strongly suggested by the study. A two-stage architecture becomes necessary when the label space is very large and overlapping. Shared lexical encoders can provide high recall@$\mathcal{C}$1 even under severe scale. Character-level modeling is particularly useful when domain discrimination depends on noisy user text, morphology, or brand and skill names. Designing the first stage around Recall@k rather than 1-best accuracy is operationally more important because reranking can only repair errors if the true domain survives the shortlist (Kim et al., 2018).

The paper also frames the per-domain binary-softmax variant as a response to overlapping label spaces. A plausible implication is that once the candidate space becomes semantically entangled, independent in/out scoring may be a more natural front-end than a single normalized softmax. At the same time, the work deliberately excludes contextual features from the SLM, leaving open the question of whether context-aware shortlisting can improve recall without violating latency budgets.

Within the scope of (Kim et al., 2018), however, the definition is precise. A Shortlisting Model is not a general small LLM, a voting procedure, or a feature selector. It is a single, shared BiLSTM-based text classifier that encodes an utterance as

$\mathcal{C}$2

scores every domain either by

$\mathcal{C}$3

or by

$\mathcal{C}$4

and returns the top-$\mathcal{C}$5 domains for subsequent list-wise reranking (Kim et al., 2018). In the evaluated 1,500-domain IPDA setting, that design yields approximately 95.93% 5-best accuracy, making large-scale domain classification operationally tractable without running full NLU across the entire domain inventory (Kim et al., 2018).